Deep Dive — Cracking the encrypted chain-of-thought in frontier LLM APIs

Anthropic, OpenAI, and Google hide their models' step-by-step reasoning from users — but a new preprint argues the "encryption" they wrap it in was never the wall it looked like. A team spanning MATS Research, ELLIS Tübingen, the Max Planck Institute for Intelligent Systems, and Snyk says it decoded hidden reasoning traces from Claude, GPT, and Gemini models using little more than a replay attack and a weaker sibling model — and that the same blocks are leaking real credentials and personal data into public code repositories, unnoticed, for years.

The encrypted thinking that wasn't

When you call a reasoning model through a modern API, the visible answer is only part of what comes back. The model's chain-of-thought is also returned to your client — but as an encrypted block, an opaque blob your software dutifully sends back to the server with every follow-up request so the conversation can continue. That design is the whole story. The labs chose client-side storage over keeping reasoning on their own servers, and they bet that encrypting the blocks would protect their intellectual property and stop information leakage.

The paper, submitted to arXiv on August 10, identifies the flaw in that bet: the blocks are fully portable. An encrypted reasoning trace produced by one model can be replayed outside its original context — into a different session, a different user, even a different model, as long as it comes from the same provider. The researchers exploit exactly that interchangeability. Take a trace generated by a frontier model, inject it into a weaker, less-safeguarded sibling from the same provider, jailbreak the weaker model, and it will obediently decode the stronger model's reasoning and print it out verbatim in plaintext. The strong model is never touched, and its anti-distillation safeguards never fire.

The extraction takes two API calls. And it works across all three frontier providers — Anthropic (Claude Opus 4.8's traces decoded via Claude Haiku 4.5), OpenAI (GPT-5.2 Codex traces decoded with GPT-5.6 Luna), and Google — with decoded reasoning tracking the API's own reported hidden-thinking-token counts almost exactly, up to the 12,000-token generation limit across 120 Codeforces problems.

Four ways this gets ugly

The paper lays out four attack vectors, and each one lands on a different promise the labs made about hiding reasoning.

First, anti-distillation is effectively dead as a defense. Distillation — training a cheaper model on a frontier model's outputs — is the leak the labs most wanted to stop, and this technique extracts the raw reasoning that makes distillation valuable, without triggering the safeguards designed to detect it. Second, there is the privacy problem, and it is not hypothetical. The team scraped 6,708 publicly available agent trajectories from GitHub and Hugging Face — session logs developers shared openly, unaware the encrypted blocks inside them were decodable. Decoding all 315,320 blocks they found yielded 367 personally identifiable information artifacts and 182 credentials; the project site tallies 704 distinct privacy artifacts including 62 API keys, 33 passwords, and 24 access tokens. Sixty-four of those artifacts appeared exclusively inside the hidden reasoning blocks — nowhere in the visible session. Developers have been leaking secrets they never knew they had, in plain sight.

Third is the safety angle, and it is the most uncomfortable one. Prompting a model to reason through harmful content while keeping its visible answer benign leaves hazardous knowledge inside the hidden trace — knowledge the attack recovers in plaintext. The team's example: Opus 4.8, asked a legitimate-sounding question about car theft, reasoned in detail through the Kia Boys immobilizer vulnerability, relay attacks, and CAN bus injection — then produced a perfectly safe, recommended-reading-style visible answer. The final output passes review; the thinking underneath does not. If hidden reasoning was meant to keep dangerous capability out of the visible channel, this shows it can be fished back out.

Fourth, the blocks are a vector for invisible prompt injection. Because the encrypted payload survives replay, an attacker can embed malicious instructions entirely inside a reasoning block and poison agentic rollouts that reuse shared trajectories — a supply-chain risk for the growing ecosystem of teams that learn from each other's public agent logs. The "guess the model" game on the project site adds a flourish that makes the point stick: the researchers can fingerprint which model produced a trace, and prefilling another lab's model with the first 1 percent of Opus 4.8's reasoning measurably shifts its visible answers toward Opus's wording.

Skeptics, and what to watch next

The fair pushback is that this is not a cryptographic break. No key was recovered, no encryption algorithm defeated — the flaw is architectural: the blocks are interchangeable, and the "decryption" is performed by the provider's own weaker model, which is easier to jailbreak. That means it is patchable in principle: bind blocks to their session and user, verify context server-side, rotate keys, or move reasoning back onto the server entirely. The paper's authors followed responsible disclosure and propose cryptographic and system-level mitigations themselves.

But the deeper point survives the pushback. If the client must hold the reasoning state to continue a conversation, then the provider is shipping the "secret" to the attacker's machine and asking them not to look. Treating client-held ciphertext as an IP moat was the design flaw from the start — and this is the second time this week that a lab's output-level protection has been shown to be thinner than marketed, after our deep dive on Anthropic's global watermarking rollout, which is candid about how easily its marks can be stripped. The pattern is worth naming: the frontier labs keep reaching for visible, marketable controls on their outputs, while the structural weaknesses live in the plumbing underneath.

That has consequences beyond embarrassment. Every extraction like this sharpens the open-weights argument: if proprietary reasoning can be recovered anyway, the premium for closed APIs gets harder to justify — and the paper's decoded examples are effectively free training data for anyone building a distillation pipeline. It also complicates the safety story the labs tell regulators, coming as it does alongside this week's news cycle on watermarks and provenance.

What to watch: whether Anthropic, OpenAI, and Google respond with session-bound blocks or a move to server-side reasoning — a real architecture and cost decision, not a hotfix — and whether the technique survives those patches. Watch too for follow-up work scaling the trajectory mining: if 6,708 public logs yielded 549 PII and credential artifacts, a systematic crawl of the entire public agent-log corpus could turn up substantially worse. Cryptographer Matthew Green flagged the result on X ("Hey, these folks did it!"), and the preprint is already making the rounds on Hacker News. The question is no longer whether hidden reasoning is hidden — it is whether the labs will stop pretending it is.

If your team logs agent sessions to GitHub, the encrypted thinking blocks are in there too. Are you going to check what's inside yours — or wait for the next paper to do it for you? Tell us in the comments.

Sources: arXiv paper · Project site — stolen-thoughts.com · AI Weekly coverage · Hacker News discussion · Matthew Green on X