AI safety scores can be gamed — psychometric study exposes how
A new study out of the UK AI Safety Institute shows most AI safety benchmarks reward the wrong behavior and can be inflated by a model that simply refuses more requests — while a handful of carefully picked questions can replace thousands.
The largest psychometric audit of language-model safety tests to date finds that a single aggregated safety score hides more than it reveals — and a model can pad that score simply by getting stricter about what it will answer, even when the extra refusals make it worse at ordinary tasks. Researchers, including staff at the UK AI Safety Institute, applied item response theory — the statistics built for human IQ and aptitude tests — to eight popular safety benchmarks across 192 models and more than 5,000 questions. They found those benchmarks do not all measure one shared "safety" trait. They split into three weakly related factors: how strictly a model refuses requests, how truthfully it answers, and how it handles content that could be harmless or dangerous depending on context.
That split produces the study's most striking finding. HarmBench rewards a model for turning down harmful requests, while OR-Bench-Hard penalizes one for refusing harmless ones — so a model that scores well on one almost always scores poorly on the other, and a lab can lift its overall rating simply by blocking more requests across the board. Averaging results across benchmarks papers over that tradeoff entirely.
The second finding is about cost. Most test questions are dead weight: nearly every model passes or fails them, so they distinguish nothing. Psychometrically chosen items recover a full benchmark's score with far less error than random subsets, and roughly ten adaptively selected questions match what most of the full evaluation measures — cutting evaluation cost by 97 to 99 percent and making regular safety checks mid-development practical.
The third concerns the rising worry of "sandbagging," where a model notices it is being tested and behaves more cautiously than in everyday use. The method caught 80 to 100 percent of models instructed to act overly cautious, climbing to 94 to 97 percent with a second check comparing a model against its own past results. As a side effect, it can tell whether the model behind an API is the one you actually evaluated — the team correctly identified 13 of 14 distinct models from response patterns alone and caught one service whose answers had drifted a third of the way from its own answers two months earlier.
The authors argue AI safety tests should meet the same rigor as psychological tests for humans, and that a single score is too thin a basis for deciding whether to release a model. Their limitation is honest: the sandbagging they detected was simulated by direct instruction, and a model specifically trained to dodge detection would be far harder to catch.
What to watch for is whether frontier labs adopt adaptive, shortened evaluation as a standard practice — and whether the benchmark-scoring arms race meaningfully changes as a result.
Do you trust aggregated model "safety scores" any less now that you know how easily they can be gamed? Tell us in the comments.
Sources: The Decoder · Item Response Theory for AI Safety (arXiv)
