Sparse weight decomposition — LLM circuits from under 1% of the data
Mechanistic interpretability has always had a cost problem: explaining a model meant training a second model. A new preprint from IQuest Research and collaborators at Oxford, Stanford, Tsinghua, and the Safe AI Forum argues the circuits are already sitting in the weights — you just have to pull them out.
A research team led by IQuest Research (至知创新研究院), with collaborators from Oxford, Stanford, Tsinghua, and the Safe AI Forum, is proposing Sparse Weight Decomposition (SWD), a method that extracts interpretable circuits directly from a pretrained model's weights — using under 1% of the data that training-based approaches like Transcoder need.
The core move is to factorize each dense weight matrix into two sparse matrices whose shared intermediate coordinates become "bottleneck units": individually addressable read/write paths that can be scored, selected, and ablated exactly like the features that sparse autoencoders produce — but without training a surrogate network on millions of activation samples first. The preprint reports that SWD matches the held-out fidelity of Transcoder and other baselines while consuming less than 1% of their data, and reaches the same circuit sufficiency and necessity targets with fewer active connections on GPT-2, Qwen2.5, and Qwen3.5-27B. It also scales to full-model replacement — all 48 attention and MLP projections of GPT-2 Small — and ships a zero-data variant that needs no calibration text at all.
Why this matters: mechanistic interpretability has been bottlenecked by cost. Sparse autoencoders require enormous activation-collection pipelines and GPU-hours per model, per layer, per checkpoint — which is why interpretability stays a one-off research exercise instead of a routine part of model development. SWD's pitch is that the model has already encoded its structure in its weights, so re-learning it from scratch is wasteful; a few thousand calibration tokens and a sparse factorization get you an intervenable surface. That could turn interpretability into something like a unit test — run after fine-tuning, before release, when auditing LoRA merges.
The honest caveats, which the authors flag themselves: this is a cheaper screwdriver, not a decompiler. Decomposed components still need semantic labeling, nonlinear layers (gated MLPs, attention softmax) don't vanish because a matrix was factored, and the method only works on open-weight models — you can't decompose a model you can only reach through an API. Dense control experiments in the paper show the gains come from sparse read/write structure, not from decomposition per se, which is the right way to test a claim like this. The reference implementation is out under Apache-2.0, so the community can check the work.
What to watch: whether SWD holds up on larger gated architectures and whether labs start folding weight-level checks into their safety and regression pipelines.
If interpretability gets cheap enough to run on every checkpoint, does that change how much you trust open models? Tell us in the comments.
Sources: arXiv paper · QbitAI · OpenAI Hub · SWD repo (GitHub)
