Kimi Delta Attention gains 2D rotations from a signed gate
A two-number change to one of the open field's most widely used attention modules buys a provable class of expressivity at no extra cost — and the authors are unusually clear about what it does not buy.
Allowing Kimi Delta Attention's channel-wise gate to go negative — into [−1, 1] — and its delta-rule coefficient up to 2 lets the module realise 2D rotations, giving it the state-tracking expressivity of DeltaProduct-2 while its transitions stay diagonal-plus-rank-one and non-expansive. The paper is from the OpenEuroLLM project and the AutoML group around Frank Hutter, and the resulting model is called Complex KDA (CKDA). The motivation is a known ceiling: delta-rule linear RNNs update their state with a low-rank correction, and a low-rank correction cannot represent a rotation directly, so anything requiring orientation or group state needs more layers or a more expensive update. Earlier work composed two delta-rule transitions inside one update to get the rotation, which raises the rank and the cost of every step. CKDA gets there by letting the gate supply the second transformation as a reflection.
The theory is the part worth reading twice. The authors prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix, and that a single CKDA layer can track every finite group isomorphic to a subgroup of SO(3). They also show that many existing state-tracking results need one fewer layer once CKDA replaces other diagonal-plus-rank-one linear RNNs — the expressivity is bought back inside the recurrence rather than in depth. Attention research has been moving faster than the models adopting it: we covered the efficiency end of that in Declarative Attention cuts long-context KV reads by up to 52%.
The empirical work is deliberately unglamorous. On finite-group word problems — S₃, S₄ and A₅ — a single layer trains on words of length up to 32 and is then evaluated for length generalisation out to 512, where CKDA gives the strongest extrapolation of the KDA range settings tested. On a phase-preserving audio task it holds the waveform's periodic structure where the other variants drift. In language modelling, a 1.3-billion-parameter run over 100 billion FineWeb-Edu tokens plus a six-rung scaling ladder through Megatron-LM, with 180 measured cells and RULER evaluations, it beats Transformers and other linear RNNs and lands "similar" to the KDA baseline with promising scaling behaviour.
That last sentence is the honest headline for anyone deploying this: on ordinary language modelling, CKDA does not beat KDA — it matches it. The gain is in the class of problems the recurrence can represent at all, which is precisely where long-context state tracking has been short. The code is MIT-licensed, the models are on Hugging Face, and the artifact is unusually reproducible for a paper like this: every table regenerates from committed measurements with no GPU and no cluster. The caveats are the authors' own — a GRU still wins their audio toy task, and the reference metrics are single-run or best-of-three.
What to watch: whether Moonshot, whose Kimi Linear introduced KDA, or the hybrid-attention crowd widen these ranges in a production training run, and whether the extra expressivity shows up where it should — in long-context state-tracking evaluations rather than leaderboards.
If a signed gate is all it takes, how many linear-attention variants are one parameter range away from tracking state they currently cannot? Tell us in the comments.
Sources: Complex KDA (arXiv) · ComplexKDA (GitHub) · OpenEuroLLM models (Hugging Face) · Hacker News discussion · r/LocalLLaMA
