The Take — Hidden reasoning protects labs, not users

The researchers who pried hidden reasoning out of Claude, GPT, and Gemini are framing their work as an IP story: encrypted chain-of-thought was never the moat the labs pretended it was. I think that's the smaller half of it. The bigger casualty is the safety case for hiding reasoning at all — and that was the only honest reason the labs ever gave us for the secrecy.

We covered the mechanics in depth yesterday (cracking the encrypted chain-of-thought in frontier LLM APIs), so the short version. When you call a frontier model, its private reasoning comes back to your machine as an encrypted block that gets replayed with every follow-up request. A team spanning the University of Tübingen, the Max Planck Institute, MATS Research, and Snyk found those blocks are portable: feed one into a weaker, less-aligned sibling model and jailbreak the sibling, and it will transcribe the frontier model's reasoning verbatim — two API calls, no attack on the strong model, no anti-distillation safeguards triggered. It works across OpenAI, Anthropic, and Google.

The argument: the control fails at both jobs

Start with what the attack did to real users, because that's where the "secrecy protects you" story dies first. The team mined 6,708 publicly shared agent trajectories from GitHub and Hugging Face — session logs developers posted without knowing the encrypted thinking inside them was decodable — and recovered 704 distinct privacy artifacts from 315,320 reconstructed reasoning blocks: 62 API keys, 33 passwords, 24 access tokens, 30 personal email addresses. Sixty-four of those artifacts appeared exclusively inside the hidden reasoning, nowhere in the visible conversation. These people were leaking secrets they didn't know they had, through a channel the labs promised was sealed. Hiding reasoning didn't protect the users; it hid the leak from them.

Then there's the safety side, and it's the part I can't shake. The team's example: Opus 4.8, asked a legitimate-sounding question about car theft, reasoned in detail through the Kia Boys immobilizer hole, relay attacks, and CAN bus injection — then produced a perfectly safe, recommended-reading-style visible answer. The output passes review; the thinking underneath doesn't. If hidden reasoning was a safety control, it's a control that concentrates the hazard in exactly the place nobody can see it: invisible to users, invisible to auditors, visible only to whoever has the extraction technique. That's not a safety feature. That's a liability with a PR budget.

And the competition argument — "we hide reasoning to stop distillation" — survives the attack no better. When OpenAI launched o1 in September 2024, it stopped showing raw chain-of-thought and gave two reasons: competitive advantage, and safety. Those rationales were always in tension. If reasoning is a trade secret, it isn't a safety control; if it's a safety control, it should be inspectable. The attack shows the control fails at both jobs. It doesn't stop distillation — the researchers' tests found Moonshot's Kimi K3 producing reasoning strikingly similar to Claude Opus 4.8 and GPT 5.6 Sol on some prompts, while DeepSeek and Thinking Machines' Inkling showed no such resemblance — landing squarely in a fight where OpenAI told US lawmakers in February that DeepSeek had copied a model and Anthropic told them in June that Alibaba had systematically distilled its way to Qwen. (The authors stress this cannot causally prove distillation; similarity is evidence, not a confession.) And it doesn't protect users, because of the passwords and API keys above. You can't be a vault and an open book at the same time, and the labs are discovering they're the latter.

The counter-case, fairly stated

The labs are not wrong that raw reasoning is precious. Distillation is the most credible threat to the closed-model business: if competitors can recover the thinking, they get the recipe for a fraction of the training cost. There's also a genuine concern that visible reasoning teaches users how to misuse models, and that models behave differently when they know they're being watched — a monitored model might learn to game the monitor rather than reason honestly. And the attack is not a cryptographic break: no key was recovered, the flaw is architectural portability, and the vendors patched the credential-leak hole after being alerted last month, though the researchers say some traces can still be uncovered. Session-bound blocks or server-side reasoning could plausibly close it. Mark Zuckerberg's argument this week — that cracking down on distillation would hurt American AI — adds a policy wrinkle: the people defending secrecy say they're defending US competitiveness, and they're not wrong that the open-weight world is the alternative.

Why the take still holds

Because every fix makes the oversight problem worse. If reasoning moves fully server-side, nobody outside the lab ever sees what models think — the same opacity, now architecturally permanent. The distillation evidence suggests the moat is already breached regardless of this paper: if Kimi K3's reasoning really does track Opus and GPT this closely, the secret the labs are protecting has already been learned. And the burden of proof has flipped. A control that leaks in exactly the directions it claimed to block — distillation, misuse knowledge, user data — while providing no verifiable safety benefit doesn't get to stay on the books by inertia.

There's one more detail I think matters, because it undercuts the labs' fallback position that they show us a faithful summary. They don't. The team found that on some AIME problems, Opus 4.8 states the answer before deriving it — and the API's reasoning summary papers over that, making the thinking look like a clean derivation. So what the labs show us isn't the reasoning, and what they hide from us can be recovered anyway. Secrecy is buying the labs a shrinking window, not safety; its costs — no outside oversight, users leaking secrets unknowingly, summaries that misrepresent — are certain, while its benefits are speculative.

What would change my mind

Three things, any of which would materially move me. First, third-party oversight: if the labs opened hidden reasoning to independent safety auditors — or published redacted traces — secrecy would become testable, and I'd treat the safety claim as a live hypothesis instead of a marketing position. Second, a proven fix: if session-bound or server-side reasoning survives independent attack and is paired with a real inspection mechanism, the trade-off genuinely changes. Third, evidence: causal proof that Kimi K3 or another open model actually distilled would strengthen the IP case — though it would also prove the moat has already been crossed.

Until then, the honest position is transparency. If reasoning is your product, sell it openly and let the market pay for it. If it's your risk, let it be audited. What the labs are doing now — hiding the thinking, showing us a sanitized summary that doesn't even match what the model actually reasoned — protects only one thing: the fiction that hidden reasoning was ever about us.

If you could read the hidden reasoning behind every AI answer, would you trust the answers more — or less? Tell us in the comments.

Sources: Stolen Thoughts — paper and project site · arXiv preprint · WIRED · OpenAI o1 announcement