SWE-bench agents lose up to 14 points when a repo's surface changes
Two research papers landed back to back on what an agent score actually certifies — one strips a benchmark repo of everything familiar and watches the score fall, the other asks models to choose a direction before they can see where it leads. A Chinese studio closed the loop on the commercial side.
Researchers at Shanghai Jiao Tong, Xi'an Jiaotong and East China Normal universities rewrote the SWE-bench repositories so that nothing about their behaviour changed, only their surface — and the coding agents working on them lost up to 14.4 points. The instrument, SchrodingerRepo, treats the test repository as a value generated at evaluation time: the problem statement is rewritten, repository-owned namespaces are remapped to semantically plausible alternatives, files are reordered, and local implementations are rewritten with behaviour preserved. On SWE-bench Verified, Gemini-3.1-Flash-Lite fell from 56.7% to 42.3%, GPT-5.4-mini from 46.8% to 35.6%, GPT-5.1 from 44.6% to 36.2%, and DeepSeek-v4-Flash from 72.8% to 66.8%. Of the four transformation levels, the namespace remapping — renaming the repo's own symbols — did the most damage on its own.
The cost side is uglier than the accuracy side. GPT-5.1's average actions per task rose 61% and its input tokens 161%; DeepSeek-v4-Flash went from 46 actions to 100 and from roughly 1.05 million input tokens to 3.7 million. Between 81.6% and 83.6% of the additional actions went to exploration and localization — the agent hunting for where things live, which is what a human does in an unfamiliar codebase and what a model that recognises a repo never has to do.
The control is what turns this from a stunt into a finding. On temporally held-out SWE-rebench instances — issues created after the evaluated models shipped — the same transformations preserved Pass@1 and only raised interaction cost. If the tasks had simply become harder, both benchmarks would have sagged. The authors' own leakage probe found clear evidence of memorised cues in more than 65% of SWE-bench Verified instances, with more than 18% reproducible at patch or test level. Read it alongside this month's private-code results — Fable 5.1 leads Real-SWE, and still fails 6 of 10 tasks — and the pattern holds: agents are strongest precisely where they have been before. It is the contamination problem we flagged in August, measured this time with an ablation — Detecting benchmark contamination that survives RL post-training.
City University of Hong Kong and Microsoft researchers built a benchmark out of 502 decision forks pulled from real engineering and research trajectories, and the best frontier model picked the better branch 59.7% of the time. Taste-Bench mines those forks automatically — from parallel attempts at the same task and from detours inside a single run — then asks the model to choose without seeing what happened next. GPT-5.5 lands 0.2 points behind GPT-5.6 Sol at 59.5%, forks whose deciding evidence only appears later in the trajectory are harder for every model, and a larger reasoning budget does not improve accuracy at all. What does work is training: distilling the judgement of a teacher that had seen the outcome into Qwen3.6-27B lifted held-out accuracy from 30.0% to 47.9%, and on 41 unseen SWE-bench Pro tasks an executor given that student's advice went from 14.6% to 33.7% success, against an oracle ceiling of 39.0%. Taste is the thing long-horizon agents are missing, and the paper's real contribution is showing it is teachable rather than merely purchasable with tokens.
China's first AI-made theatrical feature now has a release date: _Sanxingdui: Future Past_ opens nationwide on October 23. Bona Film Group dated the film on Sunday: 100 minutes, generated through its AIGMS production centre on the Boka Film Cloud Studio, with every character an original digital creation rather than a replica of a real actor. AI handled generation and execution while story, art and edit decisions stayed with the human production team. The public-screening licence from the National Film Administration — the approval that clears any film for Chinese cinemas — came first, and it is the gate the sector was actually waiting on, as we noted when China's AI video boom reached 221,900 shows for 1,055 hits; a licence means the pipeline cleared content review, not that anyone would pay to watch it. What is new on Sunday is the commercial claim: an AI-made film now sits on a cinema calendar five weeks out. The premise reads like a comment on the moment — Sanxingdui bronze inscriptions hide a binary code, and a future civilisation dependent on a superintelligence needs it to escape.
What to watch: whether any public coding leaderboard gets rerun under transformed repositories, and whether Sanxingdui's opening weekend settles the AI-film cost argument.
If part of a benchmark score is really familiarity with the repo, which number should an enterprise buyer trust — the leaderboard, or the private-code eval? Tell us in the comments.
Sources: Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It? (arXiv) · The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks (arXiv) · Taste-Bench code (GitHub) · The Sequence Radar Issue 940 · ITHome — Sanxingdui: Future Past set for October 23 · Bona Film Group via 10jqka
