llama.cpp's CPU prompt processing gets 3–7x faster on k-quants
Three local-AI items for this hour: a kernel change that makes the wait before the first token 3 to 7 times shorter on a plain CPU, two small Chinese models that answer questions instead of writing text, and a 2017 laptop finishing real agent work.
Local models just got dramatically faster to query on a processor most people already own — a kernel merged into llama.cpp today makes prompt processing 3 to 7 times quicker for the quantised formats the local community actually runs. The change reworks how the project multiplies matrices: instead of unpacking the same quantised weights over and over, once per row of input, it unpacks them once into a 256×256 block and sweeps an optimised inner loop across that block, using the integer vector instructions on recent Intel and AMD chips. The pull request's own benchmarks — an 8192×8192 matrix multiply, best of five timings, eight threads on an AMD 9950x3D — land at 7.04× stock for q3_K, 6.94× for q5_K, 5.05× for q6_K and 4.67× for q4_K.
The reason this is a bigger deal than a benchmark number is prefill. Prompt processing is the wait before the first token, and it scales with every token of context you hand the model — which is exactly the cost agentic coding setups re-pay on every turn, because they resend the whole transcript rather than a fresh question. Local AI has spent two years staring at tokens per second during generation and quietly tolerating a prefill figure that made long prompts unpleasant on anything short of a serious GPU. The number that decides whether the machine you already own is usable just moved.
The caveats come from the pull request itself, and they matter. The gain appears above 64 rows of input, fades to break-even at 32, and falls to roughly 80% of stock performance in pure one-token-at-a-time generation — so this is a prefill and batch win, not a decode win. It covers k-quants only, though the author writes that most of the code is portable C++ with a microkernel and a few unpacking routines to fill in per architecture, leaving an obvious path to the IQ quants and to ARM. One more detail worth noticing on a site that covers AI writing code: the author discloses that he wrote the original idea and first draft, then used AI to write the low-level unpackers and the microkernel, after spending time understanding the result well enough to rewrite its comments in his own words.
InternLM's new decision models answer every question in a single forward pass and never generate a single token. Intern-Decision ships at 4B, 2B and 0.8B on Hugging Face, fine-tuned from Qwen3.5-4B with the vision tower frozen: you hand it a state, a schema of typed questions — multiple choice, a score, a yes/no — and optional images, and it returns a calibrated probability distribution for each field by reading the logits sitting in front of each answer slot. On the project's own seven-suite benchmark, the 4B model averages 90.02 against the hosted Jev system it is API-compatible with at 88.74, winning five suites, tying two and losing WildJailBreak — at 44 milliseconds per query on an RTX 4090, against Jev's 110. The sharper claim is calibration: a lower Brier score (0.347 versus 0.358) and a much lower expected calibration error (0.065 versus 0.095), with the 0.8B variant answering in 34 milliseconds and averaging 79.4.
It is the same argument we have been tracking all month, now from a lab with serious infrastructure behind it — Fastino's 340M decision model beats its own 1B on routing made the case at a tenth of the size. The engineering point underneath both: an agent's gate does not need a chat model, it needs a scorer, and a scorer costs one prefill instead of an open-ended generation. Community GGUF conversions of all three sizes appeared within hours of the release.
And in the local-model forums, the number that landed hardest was an old one: a 2017 seventh-generation i5 laptop with 8 GB of RAM and no GPU ran a coding agent for 20 minutes and finished the job at about 10 tokens per second. The model was Ling Tiny 3.0, inclusionAI's 8B mixture-of-experts with roughly a billion parameters active per token — released in August and now sitting at about 19,000 downloads and 488 likes on Hugging Face — driven through llama.cpp in a write-run-iterate loop until it produced a working script to find models served elsewhere on the network. The replies are as useful as the post: one user describes running the same model on an old Vega 8 integrated GPU under llama.cpp's Vulkan backend, at 25 to 30 tokens per second of decode, for Home Assistant voice control; others report looping at higher-precision quants and call the model immature. Both are true. What is new is that CPU-only agent work has stopped being a stunt — and the kernel above is what stops it being a dare.
What to watch: whether the tiled kernel reaches the IQ quants and ARM, and whether the prefill gain holds up on the shorter contexts a chat turn actually adds.
Would you trade 20 minutes of a laptop's time for a script you didn't write — or does the prefill cost still keep you renting tokens from an API? Tell us in the comments.
Sources: llama.cpp PR #27851 — tiled mul_mat for k-quants · LocalLLaMA discussion of the CPU prefill PR · Intern-Decision-4B (Hugging Face) · Intern-Decision (GitHub) · Intern-Decision model family · LocalLLaMA — Ling Tiny 3.0 is a glimpse of the future · Ling-3.0-tiny (Hugging Face)
