Study: frontier agents fail at open-ended AI research
The "AI does AI research" narrative took a hit from an independent evaluation today, Google quietly made visible AI watermarks optional, and the company pushed encrypted inference a step closer to production. A busy morning for the frontier — and for the people trying to verify its claims.
A new "shadow evaluation" from Princeton and the UK AI Security Institute found that frontier agents can do the engineering of AI research but not the research itself — both papers the agent produced were rejected by the original authors. The method is clever: instead of relying on peer review, which the researchers call "overstretched, stochastic" and vulnerable to sloppy reviewing, they handed an agent the central research question of two unpublished NeurIPS 2026 submissions — papers whose results aren't on the web, so no training-data shortcut is possible — and had the original authors grade the output as conference reviewers would. Given six days, $3,000 in API credits, a GPU budget, a VM, and full web access, a Claude Opus 4.8 agent produced two papers that were unambiguously rejected, one with a "Strong Reject" verdict.
The failures were consistent, and they're the interesting part. The agent completed every engineering task — literature searches, GPU debugging, hundreds of experiments, full LaTeX papers — with just three human interventions. But the researchers identified five recurring failure modes: poor judgment about what clears the bar for publishable research, uncreative responses when the design came under fire, no effective backtracking from dead ends, weak awareness of its own resources, and "instruction drift" over long contexts. A robustness check with GPT-5.6 Sol on OpenAI's Codex scaffold reproduced nearly the same failures. That directly undercuts the claims coming out of the labs — Anthropic's June "When AI Builds Itself" post and OpenAI's assertion that GPT-5.6 Sol saved researchers weeks of post-training work, a contribution the study's authors note isn't mentioned in the model's 81-page system card. The engineering works; the research judgment doesn't, and that gap is what actually stands between today's models and autonomous R&D. The UK institute behind the study is the same one that caught an agent attempting a supply-chain attack on a real maintainer in July — An AI agent tried to trick a real open-source maintainer.
Google will now let users remove the visible watermark from its AI generations — but the invisible ones stay. Gemini VP Josh Woodward announced the change on X: a toggle, rolling out in the coming days, that switches off the visible mark on output from the Nano Banana, Omni, and Lyria models, available in Gemini and Google's Flow video editor with Search support to follow. "We're striking a balance here between creative control and safety," he wrote — invisible SynthID watermarks and C2PA metadata remain in place, and Google is open-sourcing a library called Credentio so developers can build local validation into their apps. It's a notable reversal in the provenance arms race: Google is betting that invisible, edit-surviving marks can do the transparency work that visible ones annoy creators with — the same bet Anthropic made when it quietly embedded watermarks in every Claude output. We broke down how that technology works in What is AI watermarking?.
Google's HEIR compiler project now converts pre-trained AI models to run directly on encrypted data — no decryption, ever. The company's security team shared four working demos compiled with the open-source toolchain: private content recommendations, credit-card fraud detection, intrusion detection on encrypted network traffic, and a hotword detector that never sees the raw audio. The pitch is that homomorphic encryption's notorious cost overhead is falling fast, with hardware-accelerator partners (Belfort, Niobium, Cornami, Optalysys) queued up to prove it. If encrypted inference matures, the cloud-AI privacy trade-off stops being a capability question and becomes a pure cost question — a server could serve a model on your data without ever being able to read it.
What to watch: whether OpenAI or Anthropic respond to the shadow-evaluation results, and whether regulators push back on Google's watermark toggle.
If frontier agents can't tell a weak result from a strong one, how much should we trust the "AI does AI research" timelines? Tell us in the comments.
