GPT-6 saturated ARC-AGI-3. The next benchmark grades invention
ARC Prize has concluded its flagship exam is no longer hard enough, and the next version will stop asking models to solve puzzles and start asking them to invent things. Also today: a UC Berkeley and Arena study measures how much of a coding agent's bill comes from the wrapper rather than the model.
ARC-AGI-4 will be "a benchmark for autonomous open-ended innovation," the ARC Prize Foundation said — a framing that pulls the series' hardest planned test forward by a generation, days after GPT-6 Astra saturated the current one. As recently as March, François Chollet described the sequence differently: ARC-AGI-4 would ship in early 2027 on a yearly release cadence, each benchmark deliberately unsaturated at launch, with the fourth generation keeping ARC-AGI-3's interactive style while concentrating on continual and curriculum learning — fewer games, far more levels, each compounding on the last. On that plan, "invention" belonged to ARC-AGI-5. Observers read the foundation's new description as the invention test being moved up and the continual-learning design folded into it, and Chollet's own endpoint has not changed: when it becomes effectively impossible to measure a gap between human learning efficiency and frontier AI, that is the AGI moment.
The measurement problem is what forced the redesign. ARC-AGI-3 threw models into turn-based game environments with no instructions, no natural-language prompts and more than a thousand levels, and Astra scored 62.7% on the semi-private set under ARC's Standard harness for $26,098 — then 99.9% for $18,817 under a Provider Adapter harness that preserves the model's opaque reasoning state between requests and compacts longer conversations. Same weights, same questions, a 37-point gap, and ARC Prize now reports both conditions side by side because the first measures a provider-neutral interface while the second measures the context management a vendor actually ships. Astra also beat the human action-efficiency baseline on 96% of levels, using 51.7% fewer actions per level on average, though ARC Prize is explicit that saturating a bounded, deterministic benchmark is not proof of AGI.
Inventing something is a harder thing to score. ARC-AGI-1 through 3 all shared a premise — a correct answer or a winning state that a human could reach on the first try, so humans could serve as the baseline (the ARC-AGI-3 human baseline came from 486 people and 2,893 attempts). Invention has no standard answer and therefore no ready-made human yardstick: you cannot say how many steps a person took to invent a thing. Chollet's own definition of intelligence — not how many skills a system has, but how efficiently it acquires new ones — points at a workable proxy: whether a tool, rule or notation a model creates makes later problems cheaper to solve. Astra's behaviour already sketched the shape of that test, when it wrote its own compact algebraic shorthand for a game's mechanics and roughly halved the actions it needed. The honest read is that the benchmark maker just conceded its exam was measuring evaluation engineering as much as intelligence — the harness, not the model, decided where a frontier lab landed on the board, exactly as NVIDIA's AVO agent showed when it cleared every ARC-AGI-3 public level.
A Berkeley and Arena team ran 21 model–harness pairs — seven models across Claude Code, Codex CLI and the minimal open-source harness Pi — on SWE-bench Lite and Terminal-Bench 2.0, and found that harness choice barely moves the success rate while moving the bill by up to 5×. Claude Fable 5 solved 97.8% of attempts in Claude Code, 96.7% in Codex and 96.7% in Pi, yet Claude Code cost about twice as much per attempt ($1.33 versus $0.67). Across shared models, Claude Code ran about 2.0× Pi's cost and 1.6× Codex's on SWE-bench Lite and 1.5× Pi's on Terminal-Bench 2.0, while the average harness effect on success rate stayed within ±2% on the first benchmark and about ±5% on the second. Pi reaches the Pareto frontier on both with four tools — read, write, edit, bash — and Claude Code's mean initial context is more than ten times Pi's, which is where the "harness tax" starts accruing, at the first model call.
The finding that should travel furthest into procurement is that provider optimisation does not guarantee the best pairing: an alternative harness produced the highest observed success rate in nine of twelve comparisons across the Anthropic and OpenAI models tested, and GPT-5.6 Sol scored 83.3% on Terminal-Bench 2.0 inside Pi versus 78.9% in Codex at roughly half the cost ($0.42 versus $0.76). The fair pushback, and it is the strongest one in the Hacker News thread, is that Claude Code's and Codex's extra weight buys sandboxing and permission layers that Pi explicitly omits — calling the difference a tax prices a security externality at zero. Both claims can be true: the same spend can be overhead and insurance, and the study's own limits are stated (two open-source benchmarks the models may have seen in training, 30 tasks each). What it does establish is that "which model" is the wrong first question for an agent purchase, which is the argument Cursor's 7.5x renewal ask ran into.
What to watch: whether ARC Prize can define invention as something measurable without a human baseline, and whether the harness tax survives replication on benchmarks the models were not trained against.
If your agent's scaffolding costs double what a four-tool alternative does for the same score, is that overhead or insurance — and which would your finance team sign off on? Tell us in the comments.
Sources: ARC Prize · François Chollet on ARC-AGI-4 timing · ARC Prize — OpenAI's GPT-6 Astra on ARC-AGI-3 · Psyho on Chollet's next-benchmark interview · AI Era via 36Kr · HarnessTax — Pan et al., UC Berkeley and Arena · Arena · Hacker News discussion
