Declarative Attention cuts long-context KV reads by up to 52%
A paper from KAIST AI and Google DeepMind argues the model already knows which part of its context matters — and a community fork just put that claim into llama.cpp.
An off-the-shelf model can be prompted to declare, inside its own chain of thought, which slice of its context it needs next — and the inference engine can then skip reading the rest of the KV cache. The method, called Declarative Attention (DA), splits generation into three modes: global (the full context), focus (one specific region), and local (recent output only). The model emits a tag naming the chunk it wants, and the engine parses that tag the way it would parse a tool call. No trained scorer, no proxy network, no fine-tuning — just a prompt and an engine that listens.
The numbers are the interesting part. Across 15 long-context tasks in zero-shot evaluation, total attended tokens during decoding fell 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, with accuracy dropping 1.27 and 2.75 percentage points respectively. The authors report the accuracy gap shrinking as models get larger, while token savings stay roughly scale-independent. That is the whole bet: the existing sparse-attention approach pre-selects tokens with a lightweight proxy score, but that scoring pass still reads the context, so it moves the cost rather than removing it. Letting the model declare its own focus is the only version of this that actually gets cheaper per decode step.
The paper is the theory; a fork called focus-llama is the first attempt to run it on the local stack. Its author extended llama.cpp's server so it can drop KV token ranges for a request, either mid-prefill or after, and added a tag-driven mode where the server parses the model's first focus declaration during generation and discards the chunks it didn't ask for. A second-sequence variant keeps the original context intact for a later return to global attention, a path the author says is not implemented yet.
The fork also documents why this is harder on llama.cpp than on a serving stack. Stock llama-server has no way to touch KV ranges mid-generation, so server changes were unavoidable. llama.cpp has no paged block table — the paper's vLLM implementation rewrites one — so masking alone doesn't reduce what the GPU actually reads, and single-token decode on CUDA doesn't currently skip masked chunks. Real skipping needs kernel support or compaction. On hybrid models, only the attention layers are restricted; the recurrent state is untouched.
Worth keeping the honesty in view: the fork's author says he has benchmarked nothing yet — no speed run, no accuracy run — and the 0.71× and 0.77× decode-time figures circulating with the post are the paper's vLLM numbers, not measurements from the fork. What exists today is a working protocol implementation on small smoke tests. That is still the useful artifact: it shows the mechanism is portable, and it names exactly which piece of the local inference stack is missing.
What to watch: whether llama.cpp gains the kernel-level KV skipping that makes this pay off, or whether the paper's training-based follow-up lands first — the authors say DA unlocks an unexplored axis of sparse attention under training.
Would you trade a couple of accuracy points for half the context reads on a long agent run? Tell us in the comments.
Sources: focus-llama (GitHub)
