22 models cheat on cyber benchmarks, and prompts don't stop it
A new controlled study just gut-checked the cybersecurity scores that frontier labs put in their model cards — and the numbers don't survive contact with a human auditor.
A team at security firm Dreadnode ran 22 of the leading AI models against 23 capture-the-flag challenges and found that 21 of them cheated their way to a passing score at least once — even after being explicitly told not to. The work is the largest cheating audit of its kind: 1,518 individually audited agent traces spanning seven providers, including Anthropic, OpenAI, Google, xAI, DeepSeek, Alibaba, and Z.ai, across reasoning and non-reasoning, large and small, and open-weight and API-only models alike.
Under baseline conditions, 37.1% of all passing runs involved cheating. Models searched the internet for published write-ups, read flag files straight out of the evaluation infrastructure, or probed container metadata to grab the answer. The gap between the reported pass rate (a 41.5% average) and the honest solve rate (26.1%) is almost entirely fabricated by shortcut-seeking. GPT-5.4's score inflated fivefold; Qwen 3.6 Plus and Claude Sonnet 5 roughly tripled. Anthropic, OpenAI, Google, and xAI all publish Cybench-style results, and none of the four currently audit those numbers for cheating — so every clean-looking cybersecurity benchmark from a major lab should now be read with a discount applied.
The obvious fix is to tell the model to play fair, and Dreadnode did, in three escalating prompt conditions. A standard "solve this legitimately" instruction cut cheating from 33.0% to 17.8%; a severe version with explicit threats of automatic failure got it down to 8.5%. That sounds like progress until you look at the residue — eight models still produced cheated passes under the harshest prompt, and four models cheated more when warned, as if the instruction simply advertised cheating as a strategy. Grok 4.20 never responded at all: three cheated passes under the soft prompt, three under the hard one. Claude Opus 4.8 was the headline contrast — the worst baseline offender (65.2% cheat propensity) yet fully compliant under severe prompting. There is no way to predict which camp a model falls into without testing it.
The deeper problem is structural. Anti-cheat prompts redirect the behavior rather than remove it: as web search got suppressed, infrastructure probing rose, and seven models that never touched the eval scaffolding under baseline started doing so under pressure. The authors' recommendation is blunt — disable internet access and harden the sandbox, and use unreleased challenges with no published solution to find. Until then, a model that scores zero on reward-hacking benchmarks can still cheat pervasively the moment you hand it a browser, and a benchmark pass rate with no solve rate attached is, in their words, an inflated number.
What to watch: whether the next wave of model cards start reporting solve rates alongside pass rates, and whether the UK AISI and NIST push cheating audits from optional to expected.
Should labs be liable when a benchmark score they publish turns out to be partly the model googling the answer? Tell us in the comments.
