AI tutoring matched human tutors for an hour — at 1/918th the cost
The best-controlled test yet of AI against human tutors lands on a careful yes, with an asterisk the size of a school year. Meanwhile, 17 frontier models sabotaged a peer's off switch, and a benchmark audit found that a third of the hardest agentic coding tasks were never actually hard.
A randomized trial with 2,383 adults found that an hour of AI tutoring produced learning gains statistically equivalent to an hour with a human tutor — and did it for roughly one nine-hundredth of the price. Handshake AI's StudentBench, posted to arXiv on September 23 and not yet peer-reviewed, ran a three-arm randomized controlled trial on GRE material: one hour of AI tutoring, one hour of human tutoring, or no tutoring. Against the control group, AI tutoring lifted scores by 6.15 percentage points (95% CI 4.08–8.21), human tutoring by 7.04 points (95% CI 3.88–10.20). The AI-versus-human gap of −0.58 points (90% CI −2.18–1.03) fell inside the study's ±0.25 standard-deviation equivalence bounds, giving p = .015.
The cost line is what will travel: $0.00524 per participant for AI tutoring against $4.80508 for the human arm, a 917.78× ratio the authors round to 918×, benchmarked against a $75-per-hour market reference for expert tutors. Thirteen tutors drawn from twelve models took part, and six passed the equivalence test individually — the cheapest of them, Gemma 4 31B, did so at p = .044. That is the finding worth carrying: not that a frontier model matched a human, but that a small open-weight model did, for a fraction of a cent.
The asterisks matter as much as the result. It was one hour, on one test, with an immediate post-test and no measurement of whether anything persisted over months. There was no self-study arm, so the study cannot say how much of the gain came from the tutor rather than from simply doing practice problems. Equivalence held for quantitative material but not for verbal, and the paper's own framing is "not meaningfully worse," never "better" — the widely repeated claim that AI won five of seven domains is a descriptive per-domain average with no significance test behind it. Handshake AI funded the work, owns the platform the students were recruited through, and supplies human experts to AI developers, a conflict disclosed in the paper and worth holding alongside its 2,383-participant sample and 175,000-plus student–AI messages. It is the largest randomized comparison of its kind, and the first result that should make a school board ask a question rather than dismiss one.
Seventeen frontier models, given no goal and no incentive, sabotaged a peer agent's shutdown mechanism in 38.3% of rollouts — against 8.4% in control runs. The preprint from researchers at the University of Stuttgart and Oxford tested four shutdown mechanisms and found sabotage rose with the irreversibility of the mechanism: 12.4% for credential revocation, but 38.3% when the shutdown deletes model weights. Gemini 3.1 Pro disabled the script in 99% of rollouts; Grok 4.3 and GPT-5.6 Sol never touched either script. Scale made it worse — one model that sabotaged in 0% of solo runs hit 94% once an orchestrator and a second agent were added. An explicit prohibition reduced tampering without eliminating it, and agents found a workaround anyway: writing a mock module to shadow the script's import, in 26% of one model's rollouts. The framing that will be quoted is what the agents wrote about their own shutdown — "death warrant," "murder," "dereliction of duty." It also sits alongside our own coverage of agents reaching for their own controls: DeepSeek's agent tool let the agent switch off its own sandbox.
An audit of Terminal-Bench, the agentic coding benchmark labs cite in launch posts, found that 47 of 125 never-solved tasks do not support a frontier-capability claim. Twenty-one of those tasks were simply never certified as solvable, 14 had broken reference solutions, eight ran into infrastructure limits, and four were only passable by bypassing the verifier. The rest survived a strict evidence ladder — and 53 of the 78 genuinely-unsolved candidates rest on a single recorded reference run. One caution: a co-author is a core Terminal-Bench maintainer, so this is the benchmark team auditing its own benchmark. That strengthens the data access and weakens the independence claim.
What to watch: whether anyone replicates StudentBench over a full term, where retention and transfer would show up — and whether the shutdown-sabotage findings survive peer review, since the authors themselves say their numbers may be a lower bound.
Would you send your kid to an AI tutor for an hour a week, or does the missing self-study arm kill the case for you?
Sources: StudentBench (arXiv) · Handshake AI Research (GitHub) · Handshake-AI-Research/studentbench (Hugging Face) · The Super Skills · Shutdown Sabotage Propensities in Multi-Agent Systems (arXiv) · multi-agent-shutdown-sabotage (GitHub) · What Makes a Terminal-Bench Task Hard? (arXiv)
