AI 101 — What is speculative decoding?

If you've ever wondered why an LLM feels slow to type out long answers — or why Nvidia's engineers spent a September 2026 blog post telling model designers how many attention groups to use — the answer is a technique called speculative decoding.

Speculative decoding is an inference speedup where a small, fast "draft" model guesses several upcoming tokens, and the big target model verifies them all in a single pass instead of generating one token at a time. The output is bit-identical to what the big model would have produced on its own; it's just reached the end of the prompt faster.

That's the whole idea. Two models, one extra step, no quality loss. The math that decides whether the trick pays off comes from the GPU underneath, which is why a chipmaker is now telling model designers how to build for it.

Why it matters right now

LLM inference is bottlenecked by memory bandwidth, not raw FLOPS — the model spends most of its time loading weights from high-bandwidth memory into the compute units rather than crunching numbers. Autoregressive generation amplifies this: the GPU generates one token per forward pass, then waits. Speculative decoding amortizes that wait by batching several guesses into a single verification pass.

The technique moved from a research curiosity in late 2022 (the original paper from Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google) to a production technique used by every major serving stack — vLLM, TensorRT-LLM, llama.cpp — within three years. Today, DFlash, a diffusion-style draft from Jian Chen, Yesheng Liang, and Zhijian Liu, reaches roughly 6× lossless speedup, while EAGLE-3 from researchers including Yuhui Li at Peking University and Hongyang Zhang at Waterloo is the current favourite for large models on GPUs.

Nvidia's September 2, 2026 developer guide turned the speedup into an architecture rule: the optimal draft length is bounded by the GPU's attention tile size of 128, so a model with 8 attention groups should target 15 draft tokens, while one with 32 should target three. Get the math wrong and you pay for a half-filled tile. Inference optimization is no longer bolted on after the fact — it's a design constraint baked in before training starts.

The mental model

Think of a slow, careful editor reading a manuscript aloud. Normally, she reads one word at a time, pronounces it, breathes, reads the next. Speculative decoding hires a fast intern to read ahead — "the cat sat on the mat, then on the —" — and the editor only has to listen and say "yes, agreed" for each word the intern got right. When the intern guesses wrong ("rug" instead of "sofa"), the editor corrects and the intern starts again from there. The editor's final manuscript is identical to what she'd have produced alone, but she finished in roughly a third of the time because most of the easy words were pre-vetted.

The intern is the draft model — a smaller, cheaper transformer that runs fast. The editor is the big target model. The "yes, agreed" check is one parallel verification pass that scores every drafted token at once. The system only emits tokens the big model would have emitted itself, which is why the output is lossless rather than approximately faithful.

A concrete analogy

A barista at a busy café takes orders one at a time, makes each drink, hands it over, takes the next order. The bottleneck isn't the espresso machine — it's the human loop. A second barista, faster but less experienced, calls the next few likely orders into a queue while the first barista works. When the order is right, the first barista just confirms and pours. When it's wrong, the second barista stops guessing and the queue is rebuilt from that point.

Most guesses are right (especially in routine orders), so the first barista's per-drink time drops sharply. If every guess were wrong, the system would just add overhead — which is why speculative decoding only pays off when draft and target models share enough vocabulary, style, and distribution that the draft is usually close.

Common misconceptions

"It changes the output." It does not. The verification step rejects any draft token the target model wouldn't have produced, so the final sequence is identical bit-for-bit. This is unlike quantization or distillation, which change the model's weights and can shift answers subtly.

"It needs an external second model." Not always. Self-speculative decoding uses parts of the same model — skipping intermediate layers, or reusing a mixture of experts routing decisions — to draft and verify within one network.

"It's a training trick." It's purely an inference technique. The big model isn't retrained. Training a separate draft model is the expensive option — Nvidia's guide puts it at one to ten trillion tokens of additional compute — but training-free variants (EAGLE-3, Medusa, the Lookahead family) avoid that cost.

"It always speeds things up." Only when acceptance rate is high. If the draft model guesses wrong constantly, the verification pass wastes time. The geometry of the GPU — tile sizes, KV-cache layout — sets a hard ceiling on how long each draft chunk can be, which is why hardware now drives model design rather than the other way around.

"It's only for text." The same trick works wherever an autoregressive decoder exists — image generation with diffusion-style drafts, speech recognition (Whisper variants reach roughly 2× speedup), and increasingly tool-using agents that generate many short steps.

Where to learn more

The original 2022 paper, "Fast Inference from Transformers via Speculative Decoding," is the cleanest formal treatment — short, well-cited, and freely available on arXiv. For the engineering view, Nvidia's September 2026 developer blog walks through six leading variants with training costs and use cases. The vLLM documentation explains how speculative decoding is wired into production serving stacks. For context on the units and architecture the trick rides on, see our AI 101 — What is a token in AI? explainer.

Do you think labs should publish the speedup number for speculative decoding the way they publish benchmark scores, or treat it as plumbing? Tell us in the comments.

Sources: Leviathan, Kalman & Matias — Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192) · Nvidia Developer Blog — Mastering LLM Techniques: Inference Optimization · Hugging Face — Speculative Decoding for 2x Faster Whisper Inference · vLLM documentation — Speculative Decoding