Fable 5.1 leads Real-SWE, and still fails 6 of 10 tasks
Two benchmark results landed the same day, and both point at the same gap: agents score well on code they have seen, and badly on code they have not.
The best coding agent in the world resolved 38.8% of real enterprise engineering tasks — and no agent solved every task in the set. Specific Labs, a Y Combinator-backed evaluation shop, released Real-SWE this week, a benchmark whose defining property is that its code has never touched the public internet. Every task was lifted verbatim or adapted from a private production codebase the company licensed from a real business: an events app with more than 200,000 users and a top-100 App Store ranking, a consumer fintech platform processing over 100,000 bank statements, and enterprise AI sales tooling. A reference solution touches a median of 11 files, against 6 in both FrontierCode and DeepSWE, on instructions that stay deliberately underspecified. Eight model-and-harness pairings ran ten tasks eight times each for 640 scored rollouts, graded by verifiers injected at grading time in Harbor format. Anthropic's Fable 5.1 in Claude Code finished first at 38.8%; GPT-6 Astra in Codex CLI took 33.8%; Gemini 3.8 Flash in Gemini CLI 31.2%; GLM 5.3, the best open weight entry, reached 28.8% on the top of the leaderboard. Six of the ten tasks sat below a 15% resolution rate; the "Analytics stream reducer" task went unsolved by all eight configurations, and "Tax jurisdiction" resolved 3.1% of the time.
Why it matters: this is the first mainstream coding leaderboard built so that memorization cannot help you. The 99% of enterprise tokens that sit behind private repos are, in Specific Labs' framing, natively out of distribution. The failure taxonomy is the useful part. Grok 4.6 spent 67.2% of its failed runs missing a stated requirement outright and Kimi K3 53.8%; GPT-5.6 Sol's dominant error was building on an assumption it never checked, in 43.3% of failures; Gemini 3.8 Flash's was wiring correct logic into the wrong place, 49.1% integration errors. Money barely tracked performance — Gemini 3.8 Flash scored 31.2% at an estimated $2.50 per rollout while GPT-5.6 Sol scored 16.2% at $2.65, and the top model cost $6.96. Duration did not help either: 71.4% of rollouts that finished inside ten minutes failed, versus 73.4% of the longer ones, so the extra time mostly bought more typing rather than more understanding. Treat the ranking as one data point — the benchmark is two days old, the sourcing methodology is only partly public, and a private-code eval cannot be independently audited the way SWE-bench can, which is both its strength and its cost. Expect the same critique r/LocalLLaMA raised on launch day: asking companies to hand their code to an evaluation vendor is a trust decision, not a scientific one.
Six agents were handed the same 3D-printable parts in two different CAD toolchains, all six shipped printable geometry, and the interesting result is that the tools failed in opposite directions. ModelRift ran three tasks — a bracket, a two-part snap-fit enclosure, and an M24x2 threaded hose barb — with a CadQuery agent and an OpenSCAD agent each, every cell driven by Claude Opus 5 through Claude Code, capped at twelve revisions, told never to fake success, and left unattended. All eight exported meshes came out watertight with zero non-manifold or boundary edges, verified by an independent STL parser that trusted neither tool, and total iteration counts tied at eleven versions each. What differed: CadQuery raises an exception and stops, so a bad model cannot ship quietly; OpenSCAD reported no errors and no warnings across roughly 45 invocations in one task while deleting mounting posts, misplacing slots, and certifying a corrupted export as clean. CadQuery also lets the agent interrogate its own geometry — one agent read the countersink angle straight off the boundary representation and wired it into build-time asserts, while both OpenSCAD agents had to write their own binary STL parsers, roughly as much code as the parts themselves, just to measure what they had built. The upset was the hardest task: the true helical thread went to OpenSCAD in a single version, correct on first compile, in 43 milliseconds. Speed, incidentally, was irrelevant — geometry recompute ran 30 to 100 times faster in OpenSCAD, which is noise inside a loop dominated by model inference.
The transferable finding is about feedback channels, not CAD. Renders caught coarse blunders and nothing subtle; every defect that would have ruined a print surfaced as a number — a volume, an angle, an interference test. That is the same problem we argued about this afternoon in An agent grading its own homework is an alibi, not proof, and it is why the physics-graded circuit results in Claude Opus 5 tops EEBench, where circuits are graded by physics aged better than most leaderboards this month. Any agent pipeline that leans on screenshots is leaning on the one channel that caught nothing here.
What to watch: whether Real-SWE rotates new company codebases in on a schedule, since a private benchmark that stops growing private just becomes a small one.
If your best coding agent only finishes four tasks in ten on your own repo, does that change what you'd let it touch unsupervised? Tell us in the comments.
