Gowers: LLMs solve math's biggest problems with counterexamples
A Fields Medalist's honest read on the LLM math boom, and a new benchmarking tool that tests models against your own data instead of someone else's.
Fields Medalist Timothy Gowers says the most famous mathematics problems LLMs have solved so far came almost all as counterexamples, not proofs. Writing days after OpenAI announced it had cracked ten major open problems — including the first construction of a non-sofic group and a superexponential lower bound for multicolor Ramsey numbers — Gowers points to a pattern: the headline results, from the non-sofic group to the Jacobian conjecture and the unit distance conjecture, are overwhelmingly existence statements that disprove something, rather than elegant proofs of deep theorems.
His deeper argument is that "finding a counterexample" isn't a crisp category at all. The logical form of Gluskin's theorem and Vinogradov's three-primes theorem is nearly identical, yet one is a counterexample and the other is a theorem — what matters is mathematical context and what experts believed. The useful observation is that the example-finding methods LLMs excel at — off-the-shelf constructions, just-do-it proofs, the probabilistic method, generic examples — play directly to their two obvious strengths: knowing a huge amount of mathematics and being able to brute-force search far more paths than a human ever would. Where humans may still hold an edge, Gowers suggests, is the "nose" for pruning the search tree — judging which directions are fruitful — and he isn't sure that quality emerges naturally from scaling. His test for when LLMs have truly reached human-level mathematics: a proof as surprising as the 2016 cap-set breakthrough, a genuinely new method that nobody saw coming.
Artificial Analysis launches Optima, letting anyone benchmark AI models against their own data. The independent evaluation shop's new platform builds custom benchmarks from your datasets, agent traces, or even just a written description of a use case, then runs frontier models against them and compares quality, cost per task, and time per task side by side. Early testers used it to find finance agents that cut costs tenfold without losing quality and to pick models that matched a law firm's writing style. It's a practical answer to the "benchmarks are broken" critique — Epoch AI found that scaffold choice alone can swing agentic benchmark scores by up to 15 percentage points — though a custom benchmark is only as sound as its design.
What to watch: whether labs take Gowers' hint and reward models for exploring fewer dead ends — a training tweak that might push them toward proofs instead of lucky examples.
Do you trust a model more when it finds a counterexample than when it claims a proof? Tell us in the comments.
Sources: Gowers's Weblog · Techmeme · Artificial Analysis · The Decoder
