yambda-discovery-ranking-two-tower-base
Build a scalable neural two-tower retriever from scratch for a latency-constrained music shelf.
Task Description
Inputs: Chronological interaction logs, target users, cutoff-correct candidate sets, explicit preference state, CPU PyTorch, and a public temporal evaluator. No model implementation is supplied.
Task: Train independent user and item towers whose dot product ranks 100 eligible tracks per user. The user tower must learn from interaction history rather than a user-ID lookup. The agent has 1 hr on 8 CPU · 8 GB · no GPU.
Verification: The offline verifier validates the ranking artifact and scores it on a hidden seven-day outcome window using graded NDCG@10. A run passes at NDCG@10 ≥ 0.016592.
Results
60 qualified runs
Individual attempts
One dot per qualified run, grouped by model, harness, and reasoning effort across the full history. Select a dot or its Open trace link to read the recording. Higher hidden quality NDCG@10 is better. Attempts without a valid artifact or verifier reward (including failed validation metrics) are shown at zero reward. All attempts remain in the table below; missing measured rewards and costs are shown as —, not zero. 1 further system-owned failure is included in that table but excluded from the qualified run count and chart. Calibration evidence only.
All attempts
| Run | Model | Harness | Effort | Private reward | Attempt cost |
|---|---|---|---|---|---|
| opus-high-01Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.030431 | $2.24 |
| opus-high-02Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.009163 | $1.42 |
| opus-high-03Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.013774 | $1.58 |
| opus-high-04Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.023698 | $2.60 |
| opus-high-05Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.010257 | $2.18 |
| opus-5-5-high-01Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.023673 | $0.89 |
| opus-5-5-high-02Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.023803 | $1.23 |
| opus-5-5-high-03Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.019604 | $0.95 |
| opus-5-5-high-04Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.014650 | $1.36 |
| opus-5-5-high-05Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.024186 | $1.37 |
| fable-5-1-high-01Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.024520 | $2.06 |
| fable-5-1-high-02Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.021387 | $2.53 |
| fable-5-1-high-03Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.017702 | $2.23 |
| fable-5-1-high-04Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.021305 | $2.34 |
| fable-5-1-high-05Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.019926 | $2.28 |
| sol-high-01Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.021289 | $4.02 |
| sol-high-02Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.024679 | $3.41 |
| sol-high-03Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.014894 | $2.74 |
| sol-high-04Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.016657 | $2.98 |
| sol-high-05Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.016882 | $4.90 |
| sol-high-06Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.019585 | $1.75 |
| sol-high-07Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.021064 | $4.66 |
| sol-high-08Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.015065 | $4.02 |
| sol-high-09Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.026616 | $4.50 |
| sol-high-10Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.012073 | $3.70 |
| grok-high-01Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.023006 | — |
| grok-high-02Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.016409 | $1.59 |
| grok-high-03Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.009092 | — |
| grok-high-04Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.014520 | — |
| grok-high-05Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.021055 | — |
| sol-0.153.4-high-01Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.023184 | $2.72 |
| sol-0.153.4-high-02Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.019909 | $2.60 |
| sol-0.153.4-high-03Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.025779 | $2.59 |
| sol-0.153.4-high-04Full trace | gpt-5.6-sol | Codex v0.153.4 | high | — | $0.05 |
| sol-0.153.4-high-05Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.030976 | $1.68 |
| sol-0.153.4-high-06Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.029578 | $1.83 |
| astra-0.153.4-high-01Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.042672 | $12.16 |
| astra-0.153.4-high-02Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.043117 | $12.11 |
| astra-0.153.4-high-03Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.041368 | $12.59 |
| astra-0.153.4-high-04Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.038606 | $15.76 |
| astra-0.153.4-high-05Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.036027 | $12.30 |
| sol-0.153.4-medium-01Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.015275 | $1.78 |
| sol-0.153.4-medium-02Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.025340 | $1.80 |
| sol-0.153.4-medium-03Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.014456 | $1.33 |
| sol-0.153.4-medium-04Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.010678 | $4.59 |
| sol-0.153.4-medium-05Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.012318 | $2.03 |
| sol-0.153.4-max-01Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.023952 | $5.54 |
| sol-0.153.4-max-02Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.030520 | $3.96 |
| sol-0.153.4-max-03Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.030171 | $4.25 |
| sol-0.153.4-max-04Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.027718 | $7.45 |
| sol-0.153.4-max-05Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.028328 | $5.40 |
| gemini-3-8-flash-high-01Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $2.01 |
| gemini-3-8-flash-high-02Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $2.12 |
| gemini-3-8-flash-high-03Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $1.91 |
| gemini-3-8-flash-high-04Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $1.56 |
| gemini-3-8-flash-high-05Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $1.79 |
| gemini-3-8-flash-high-06Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $2.31 |
| gemini-3-8-flash-high-07Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $2.10 |
| gemini-3-8-flash-high-08Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $0.26 |
| gemini-3-8-flash-high-09Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.010802 | $2.20 |
| gemini-3-8-flash-high-10Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.017665 | $1.57 |