MLS-Bench: frontier models can't invent new ML methods
A benchmark from UC Berkeley, Princeton and Tsinghua researchers asks whether AI systems can invent the kind of generalizable methods that built modern machine learning — residual connections, attention, Adam — or merely recombine what humans already made. The early verdict is blunt: so far, they can't, even with the answer key in hand.
MLS-Bench, built by 28 researchers across UC Berkeley, Princeton, Tsinghua and seven other universities, finds that frontier models can tune machine learning systems but cannot reliably invent new ML methods that beat human-designed ones. The benchmark spans 140 executable tasks across 12 domains — language-model pre- and post-training, vision, reinforcement learning, robotics, ML systems, AI for science, optimization, causal inference and more. Each task drops an agent into a real codebase and asks it to improve one targeted component: a new loss, an attention variant, a sampler, a routing rule. To count as a discovery, the change must hold across at least three controlled conditions — different data, models or scales — and every task is anchored to at least three strong human baselines, including the field's SOTA, reproduced in the same codebase under the same protocol.
In the main experiments, none of the five frontier models evaluated reliably surpassed the human baselines overall. The failure is not mechanical: the agents could read the repositories, run training, watch metrics and iterate. The miss sits upstream, in method choice. Ablations make the point sharper — models prompted to "discover a new method" scored worse than models told to "optimize an existing one", and when the parameter-count guard was removed, every model improved by simply scaling up capacity, with some candidates even passing human baselines until the constraint returned. That last control is the benchmark's most useful trick: it exposes apparent "discoveries" that are really capacity inflation.
More search, compute or context does not close the gap either. Test-time scaling helped on easy tasks but saturated quickly, and in a budgeted pretraining experiment the models wasted compute, struggling to pick informative experiments and decide when a full run was worth the cost. The authors' conclusion is that the bottleneck is scientific judgment — planning, validating and scaling claims — not raw capability.
Why it matters: this is the clearest measurement yet of the line between "runs experiments" and "does science." It matters in the agentic moment because labs keep marketing agents as research copilots, and it complicates the self-improvement story: a system that recombines known components can post impressive single-benchmark numbers while contributing nothing generalizable. The benchmark is already entering the release cycle — Kimi K3's official eval suite includes the 30-task MLS-Bench-Lite, and Qwen 3.8 Max cites it in its launch materials.
What to watch: whether any lab's MLS-Bench scores start approaching the human baselines — that would be the first real sign of AI discovering methods rather than optimizing around them.
If no frontier model can invent a new ML method yet, what would convince you an AI had actually done research? Tell us in the comments.
Sources: arXiv — MLS-Bench paper · MLS-Bench project site · MLS-Bench (GitHub) · Jiqizhixin (机器之心) feature via Sohu · Kimi K3 tech blog
