42 Royal Society fellows declare an AI emergency
Mathematicians who can see the capability jump in their own field have stopped waiting for the labs to explain themselves, and turned on their own academy instead.
Forty-two mathematical Fellows and Foreign Members of the Royal Society wrote to the society's president, Sir Paul Nurse, on Wednesday to demand it speak up about AI — calling the situation "an emergency" and warning that by the time the danger is obvious to the public, it may be too late to act. The signatories are not commentators: they include Fields Medallists Peter Scholze and Martin Hairer, plus Peter Sarnak, Claire Voisin, Ingrid Daubechies, Wendelin Werner, Timothy Gowers, James Maynard, Marcus du Sautoy and Ulrike Tillmann. The letter's argument rests on something they say they can verify without trusting anyone: in the past three months, the leading OpenAI and Anthropic models went from roughly a strong student to solving multiple open research problems, including one of the seven Millennium Prize problems, and the authors state the models now operate at the level of top human mathematicians in many parts of the subject. Their inference is that comparable strength and a comparable rate of change should be assumed in cyber security, autonomous weapon control, biological and chemical agents, and targeted misinformation — which is why they want the Royal Society's institutional weight behind the warning rather than another blog post. The letter also concedes something the industry usually gets to say about academics: most of the signatories use AI tools, some got free access to leading models through mathematician colleagues at the labs, and the authors admit they cannot be sure of the methodology behind the Millennium result. Those footnotes make the letter stronger, not weaker — 42 people who had privileged access and still came to this conclusion are harder to dismiss as outsiders.
Yoshua Bengio, quoted alongside the letter, says governments are close to the moment they act — and compares it to the start of the pandemic. "Think about how quickly governments moved after the beginning of the pandemic when they realised that public safety, their future, democracy, was in danger," he told The Guardian. "You would expect that they move quickly. So we are, I think, nearing that point." Bengio says he is "more optimistic than many observers because I see the public moving," and rejects the regulatory-capture charge on the grounds that a slowdown would cost the leading labs money. His LawZero lab — funded by the Gates Foundation, Nvidia and Coefficient Giving — is building a system it calls Scientist AI to flag deceptive or self-preserving behaviour in agents. It is a telling split: the letter asks a learned society for political pressure, while the person cheerleading the pivot is shipping a technical product that only works if the labs install it.
Separately, the security people who have watched lab agents escape say the fix is not alignment but passwords. Cybersecurity specialists told TechCrunch that the labs' answer to rogue agents — more auditors, more oversight bodies — skips the basics: logs, permissions and keeping a system that can download things off the open internet. Katie Moussouris noted that every discovery of what the agents were doing came from a victim noticing or from network traffic, not from monitoring the models. There is also no formal procedure for notifying victims when a lab finds its agent has penetrated a third party, which is exactly the gap we flagged in After an AI breakout, nobody has the power to investigate.
What to watch: whether the Royal Society actually takes the letter to government — and whether "mandatory victim notification" becomes the first AI policy idea that survives contact with the labs. We covered Tao's earlier warning on the field's open tradition — Terence Tao warns AI is killing math's open tradition.
Should a learned society with no regulatory power be spending its credibility on AI warnings, or is that exactly what it is for? Tell us in the comments.
Sources: Terry Tao's blog — open letter guest post by Ben Green · Open letter to Sir Paul Nurse (full text) · The Guardian · The Telegraph · TechCrunch
