How to — tell a real benchmark from a marketing one
Every model launch ships a benchmark table. Almost every one of them is, in some sense, true. None of them are telling you the same thing. The job is not to find a fake benchmark — most aren't fake — but to find the one whose number you can carry into a decision. Here is the routine.
Benchmarks are the scoreboard the field runs on, and the scoreboard is under strain. Terminal-Bench 4.0 spent recent releases removing saturated tasks and fixing broken ones; SWE-bench had to be split into a "Verified" subset because the original got solved; Google DeepMind just ran a double-blind pilot with the Singapore AI Safety Institute so test prompts can't be absorbed into training. The numbers you read today are more honest than the ones you read a year ago — and the ways they are still misleading have gotten subtler. (For a refresher on what these tests actually are, AI 101 — What is an AI benchmark? lays out the three parts — tasks, harness, grader — that this routine pokes at.)
1. Ask whether the test is still a test
The first question is the oldest: has this benchmark been solved? A benchmark is a fixed set of tasks, and once the field approaches the ceiling, scores stop telling you anything about which model is better — they start telling you which model is better at memorizing the answers. SWE-bench hit this in 2024; the team responded by filtering down to a 500-instance human-verified subset that became the score everyone quotes. Terminal-Bench is on the same arc; version 4.0 was, in effect, a recalibration, and the team now retires tasks that stop being discriminative.
The user-facing tell is a score that is suspiciously high, or one that hasn't moved in months while every other number on the leaderboard has. A benchmark whose top score is 95% is usually a benchmark whose bottom score is also 95%. Check whether the result page still updates — if the leaderboard froze, the eval froze with it.
2. Look for the evals that are missing
No benchmark table shows every eval the lab ran. The honest table shows the wins and a few of the losses; the marketing table shows only the wins. So the second move is not to read what is on the page — it is to imagine what is not.
Two or three evals are standard for a serious frontier launch. If your model launch doesn't report them, that's the story. Coding models without a SWE-bench score, long-context models without a needle-in-a-haystack number, agent models without Terminal-Bench — these are not oversights; they are signals. The labs know which evals would make their case harder, and they have decades of collective experience choosing not to run them. When a launch cites an obscure bespoke eval instead of the standard one, treat the obscure one as the test designed to flatter the new model.
3. Translate "best score" into a confidence interval
A point estimate on a benchmark is a mean without a margin. If the benchmark has 500 tasks, the standard error on a top score is large enough that a one-point gap between two models is, in most cases, noise. The honest reporting style is the one Terminal-Bench adopted: confidence intervals next to the number, so you can see whether two adjacent ranks actually overlap.
If the launch table has no error bars, no run count, and no mention of how many independent trials were averaged, treat the ranking as suggestive at best. The fix is to find a leaderboard that does report those — Stanford HELM and the Hugging Face Open LLM Leaderboard are the two most reliable for this — and read the same model's score there. If the in-house number and the leaderboard number disagree by more than a few points, something in the harness is doing the flattering.
4. Check the harness, not just the score
Two models can post different scores on the same benchmark because one of them was given better tools. The harness — the runtime, the agent scaffold, the prompt template, the number of tries, the allowed context — is part of what is being measured. When the launch claims the new model "tops" a benchmark, the question is what scaffold was used to top it.
A useful rule: if the harness is not described in the launch post, the score is not portable. The model cannot be expected to reproduce it on your workload, because you don't know what scaffold produced it. The benchmark kit that most public leaderboards run on is lm-evaluation-harness, and the closer a launch's setup is to that baseline, the more comparable the score is to anyone else's. When a launch numbers its win against a different scaffold than the public one, that is a separate story.
5. Ask whether the model has seen the questions
Contamination is the silent killer of benchmark scores. When test items leak into training data, the model scores well because it has memorized the answers, not because it can solve them. Detecting this is genuinely hard; a contamination audit that finds nothing can mean "clean" or just "the audit wasn't sensitive enough." When Is Benchmark Contamination Detectable? makes the second reading explicit: information-theoretic limits on detection mean even careful audits can miss things.
The tells are usually circumstantial. A score that improved much faster than the field's plausible learning rate. A benchmark the lab has not previously cited. A new eval that launched around the same time as the model. None of these is proof, but together they are reasons to weight the number down. Google DeepMind's double-blind pilot is the structural answer — test prompts sealed in a cryptographic box so the provider can't absorb them — and it is the direction the field will move in over the next year.
6. Ask what the benchmark is actually measuring
A benchmark measures what it measures, and not what you care about. SWE-bench tests whether a model can fix real GitHub issues; it does not test whether it can ship a feature in your codebase. MMLU tests broad knowledge recall; it does not test whether the model can write a contract you would sign. Terminal-Bench tests long-running computer tasks in a shell; it does not test whether the same model would make a competent colleague on Slack.
The practical move is to keep two lists in your head — what the benchmark tests, and what you need — and refuse to let them collapse. The closer those two lists are, the more weight the number deserves. The further apart they are, the more the number is a vibe dressed up as a measurement. (SWE-bench and its original paper, and Terminal-Bench with its 4.0 release notes, are the two best places to read what a benchmark actually tests, in the authors' own words.)
Don't do this
The single worst move is to cite a benchmark number without naming the harness, the run count, and the eval version. "Model X scored 82 on benchmark Y" is not a fact; it is a sentence missing its footnotes. The other trap is to read a saturated benchmark as a verdict — once the top of a leaderboard is flat, every additional percentage point is decoration. If you find yourself explaining a model's launch in a single benchmark number, you are still inside the marketing.
How you'll know it worked
When you've done this, you can summarize any benchmark claim in three lines without re-reading the source: what the eval actually tests, what the harness was, and whether the number is even discriminative at the score it posted. You can say which leaderboard to check the same number on, which evals were conspicuously absent, and how much weight the score deserves for the workload you actually have. And when a new launch lands tomorrow, you can read the table in five minutes instead of believing it in five seconds.
Which benchmark have you stopped trusting — and which one do you still believe? Tell us in the comments.
Sources: AI 101 — What is an AI benchmark? · Stanford HELM · Terminal-Bench 4.0 release notes · lm-evaluation-harness (GitHub) · Google DeepMind on double-blind evaluations · When Is Benchmark Contamination Detectable? (arXiv)
