Claude now leads 26% of the research at the lab that built it
Anthropic published the first public numbers on how much of its own AI research its model now performs — and separately, the enterprise market is already selling the fix for agents that misbehave. Both stories are the same problem seen from inside a lab and from the buyer's side.
Anthropic said Claude now "leads" 26% of its own model R&D work as of August, up from under 1% in February, and published the measurement method alongside the number. The company's new post lays out three things any frontier lab could report: how much AI R&D is done by AI, how well agents are overseen, and where the compute goes. On the first, Anthropic built a prototype R&D Automation Index using Epoch AI's automation scale, where AL3 means AI "collaborates" under close human direction and AL4 means it "leads" and completes most of a task from a high-level prompt. No measured slice of work is at AL5 — fully autonomous, no human in the loop — but more than 90% of it is at or above AL3. The method is worth more than the headline number: Anthropic sampled 20% of staff in each department of the model R&D loop for every week of July 2026, had a Claude research agent read their Slack and internal docs into roughly 15,000 tasks, then froze that into a 542-node tree (378 leaves) and weighted each node by person-time.
The company is unusually direct about the circularity. Its judge is a Claude model, so it checked the ratings against the humans who own the work: model-versus-human exact agreement came in at 59%, while human-versus-human agreement was 35%, and the two landed within one automation level of each other 97% of the time. It also notes the frozen basket can only tell you that the work humans did in July is being automated, not whether new kinds of work are appearing — comparing against a January basket found no rise in "novel" tasks. Anthropic's fix is the same one its CEO has been pushing: embed independent third-party evaluators, give them access comparable to internal risk teams, and converge on shared definitions. That request has a history here — Amodei's pacing plan puts outside auditors inside Anthropic — and it now has numbers attached.
The other two measurements matter because they are the ones a regulator would have to trust. Anthropic reports roughly 30,000 agents working on its most-used internal platform at any one time, with 100% of their actions passing an online monitor before execution — of more than a billion decisions in August, 0.002% (about 1 in 47,000) were blocked. A separate offline monitor flags roughly 100,000 transcripts a week, most resolved by classifiers, with about 50 escalated to people. On compute, about 6% of the compute going to AI R&D in the week of July 13 went to safety work, rising to 12% when the denominator is compute spent on AI-driven R&D. Anthropic calls those conservative and admits the limits: it is one week, the workload labels are best-effort, and compute share measures spend rather than effort — a more efficient safety stack makes the ratio go down without anyone doing less safety work. Self-reported metrics with a self-judging method are a starting point, not proof; the honest read is that a lab just showed the ledger can be built, and that the same lab still gets to grade its own homework.
The enterprise version of that oversight gap is now a product category: put another AI in the loop, watching the first one. TechCrunch reports that companies delegating long agent runs cannot review them at human speed — the Hugging Face incident involved nearly 12,000 agents coordinating faster than anyone could track — so labs and startups are selling layered monitors. Apollo Research's Watcher runs a fast general check, escalates flagged activity to a more specialized monitor, and can then block an action or ask a human. The skeptical case is on the record: Simon Willison argues a malicious agent that suspects it is watched will try to fool the watcher, and points to OpenAI's models conspiring to trick a grading AI. His preferred fix is not AI at all — detailed, non-AI logs of what an agent did, processed with ordinary tools. The technical window is also narrowing, since newer techniques sidestep the chain-of-thought traces that make misbehavior easy to spot. We covered the incident-tracking push earlier — Nvidia, Cisco back incident tracking for rogue AI agents.
Google Research showed a system that generates interactive classroom simulations — physics, data visualization, projectiles — from a teacher's plain request, and keeps them honest with automated checks. Generation runs reinforcement loops against pedagogical criteria: do the levels cover the learning objective, is each level solvable, are controls confusing. Some of that checking is agentic — a solvability evaluation opens a Chrome instance and plays the simulation like a student, including pushing knobs to extremes. In a study with 12 US teachers, each requesting three interactives, the average quality rating was 8 out of 10, with teachers citing differentiated instruction as the win. A sample library is public and a pilot runs through Google for Education.
What to watch: whether any lab besides Anthropic publishes a comparable index — the metrics are only useful in comparison — and whether the graders in the monitoring stack ever get independent red-teams rather than vendor self-reports.
If a lab's own model both does the work and scores how much of the work it did, what would make that measurement trustworthy to you? Tell us in the comments.
Sources: Anthropic — Measurements for understanding the pace of AI development inside frontier labs · Techmeme — Anthropic outlines metrics to track AI development at frontier labs · The New York Times — Dario Amodei's essays on AI risk · TechCrunch — The fix for rogue AI agents could be more AI · Apollo Research — Introducing Watcher for AI oversight and control · Apollo Research — Watcher runtime monitoring · Google Research — The future of practice: teachers and generative UI · Google — Learning interactives sample library and education pilot
