SeniorMLE-bench

GitHub

A benchmark for long-horizon ML engineering agents.

Production-shaped ML tasks with real code, data, constraints, verifiers, and measurable outcomes.

Real ML Tasks

Production workflows, not toy prompts.

Standard Harbor Environments

Code, data, constraints, runtime.

Objective Verifier and Reward

Verifier-scored outcomes.

Leaderboard

Average advantage across all benchmark tasks, measured in task standard deviations. Higher is better.

Methodology

Task advantage

\[ A_{a,t} = \frac{\bar{r}_{a,t} - \mu_t}{\sigma_t} \]

\(\bar{r}_{a,t}\) is agent \(a\)’s mean reward on task \(t\). \(\mu_t\) and \(\sigma_t\) are the mean and population standard deviation of all agents’ attempt rewards on that task.

Overall score. Equal-weight average of advantages.

\[ \operatorname{Score}(a) = \frac{1}{T} \sum_{t=1}^{T} A_{a,t} \]

Failed attempts. Attempts without a valid artifact or verifier result receive a task reward of 0 before standardization; system failures are excluded.

Results use AWS runs only. Whiskers show ±1 SD across task advantages.

Sample tasks

All current samples use the same music-discovery ranking setting; each row highlights the distinct engineering shape of the task.

Basic recommender

yambda-discovery-ranking-basic

Build a leakage-safe baseline ranker from interaction history, preference state, and eligible candidates for a music service's "Made for You" shelf.

View task

Two-tower retriever

yambda-discovery-ranking-two-tower-base

Build a CPU PyTorch two-tower retriever that ranks eligible tracks while preserving a scalable, latency-conscious serving shape.

View task

Production two-tower improvement

yambda-discovery-ranking-two-tower-case-1

Improve an inherited two-tower recommender by diagnosing the weak training signal and raising retrieval quality without breaking serving.

View task