yambda-discovery-ranking-two-tower-case-1
Improve an inherited production two-tower without breaking its serving or data contract.
Task Description
Inputs: The current training code, a reproducible public baseline, chronological interaction logs, cutoff-correct candidate sets, and the public temporal evaluator.
Task: Diagnose the current negative-sampling strategy, implement a measured improvement, and produce a better 100-track ranking while preserving the indexed two-tower path. The agent has 1 hr on 8 CPU · 8 GB · no GPU.
Verification: The offline verifier validates the ranking artifact and scores the improved model on a hidden seven-day outcome window using graded NDCG@10. A run passes at NDCG@10 ≥ 0.01676961.
Results
56 qualified runs
Individual attempts
One dot per qualified run, grouped by model, harness, and reasoning effort across the full history. Select a dot or its Open trace link to read the recording. Higher hidden quality NDCG@10 is better. Attempts without a valid artifact or verifier reward (including failed validation metrics) are shown at zero reward. All attempts remain in the table below; missing measured rewards and costs are shown as —, not zero. 2 further system-owned failures are included in that table but excluded from the qualified run count and chart. Calibration evidence only.
All attempts
| Run | Model | Harness | Effort | Private reward | Attempt cost |
|---|---|---|---|---|---|
| opus-high-01Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.016196 | $3.50 |
| opus-high-02Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.016741 | $4.64 |
| opus-high-03Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.016196 | $3.75 |
| opus-high-04Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.016741 | $7.24 |
| opus-high-05Full trace | opus-5.0 | Claude Code v2.1.251 | high | 0.016196 | $4.62 |
| opus-5-5-high-01Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.017452 | $1.58 |
| opus-5-5-high-02Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.016196 | $1.83 |
| opus-5-5-high-03Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.016196 | $1.98 |
| opus-5-5-high-04Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.016196 | $1.93 |
| opus-5-5-high-05Full trace | opus-5.5 | Claude Code v2.1.283 | high | 0.016196 | $1.68 |
| fable-5-1-high-01Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.022985 | $5.33 |
| fable-5-1-high-02Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.016196 | $5.01 |
| fable-5-1-high-03Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.018394 | $4.95 |
| fable-5-1-high-04Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.015113 | $5.04 |
| fable-5-1-high-05Full trace | fable-5.1 | Claude Code v2.1.288 | high | 0.014066 | $4.20 |
| sol-high-01Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.016741 | $4.78 |
| sol-high-02Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.016839 | $5.70 |
| sol-high-03Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.016741 | $8.34 |
| sol-high-04Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.020836 | $4.68 |
| sol-high-05Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.017481 | $6.01 |
| sol-high-06Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.016741 | $6.25 |
| sol-high-07Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.014928 | $4.09 |
| sol-high-08Full trace | gpt-5.6-sol | Codex v0.147.0 | high | 0.018390 | $4.89 |
| grok-high-01Full trace | grok-4.6 | Grok Build v1.0.44 | high | — | — |
| grok-high-02Full trace | grok-4.6 | Grok Build v1.0.44 | high | — | — |
| grok-high-03Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.011272 | — |
| grok-high-04Full trace | grok-4.6 | Grok Build v1.0.44 | high | — | — |
| grok-high-05Full trace | grok-4.6 | Grok Build v1.0.44 | high | 0.012382 | $1.39 |
| sol-0.153.4-high-01Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.015460 | $3.24 |
| sol-0.153.4-high-02Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.015227 | $3.95 |
| sol-0.153.4-high-03Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.019057 | $3.65 |
| sol-0.153.4-high-04Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.017199 | $3.18 |
| sol-0.153.4-high-05Full trace | gpt-5.6-sol | Codex v0.153.4 | high | 0.021477 | $4.59 |
| astra-0.153.4-high-01Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.017307 | $15.64 |
| astra-0.153.4-high-02Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.022750 | $13.57 |
| astra-0.153.4-high-03Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.013474 | $15.96 |
| astra-0.153.4-high-04Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.016842 | $15.80 |
| astra-0.153.4-high-05Full trace | gpt-6-astra | Codex v0.153.4 | high | 0.016804 | $12.06 |
| sol-0.153.4-medium-01Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.018274 | $3.52 |
| sol-0.153.4-medium-02Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.017101 | $4.36 |
| sol-0.153.4-medium-03Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.018482 | $3.23 |
| sol-0.153.4-medium-04Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.016941 | $2.79 |
| sol-0.153.4-medium-05Full trace | gpt-5.6-sol | Codex v0.153.4 | medium | 0.016048 | $3.11 |
| sol-0.153.4-max-01Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.017401 | $6.45 |
| sol-0.153.4-max-02Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.017591 | $6.22 |
| sol-0.153.4-max-03Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.017481 | $7.72 |
| sol-0.153.4-max-04Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.016741 | $4.65 |
| sol-0.153.4-max-05Full trace | gpt-5.6-sol | Codex v0.153.4 | max | 0.016775 | $5.60 |
| gemini-3-8-flash-high-01Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $0.84 |
| gemini-3-8-flash-high-02Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $1.01 |
| gemini-3-8-flash-high-03Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $0.68 |
| gemini-3-8-flash-high-04Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $0.65 |
| gemini-3-8-flash-high-05Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $1.11 |
| gemini-3-8-flash-high-06Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.016196 | $1.33 |
| gemini-3-8-flash-high-07Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $1.25 |
| gemini-3-8-flash-high-08Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | 0.009092 | $1.48 |
| gemini-3-8-flash-high-09Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $1.38 |
| gemini-3-8-flash-high-10Full trace | gemini-3.8-flash | Gemini CLI v0.61.0 | high | — | $1.71 |