Speculative decoding
Also known as: speculative sampling, assisted generation, draft-model decoding
Speculative decoding is a technique for making large AI models generate text faster.
A small, fast 'draft' model guesses the next several tokens, and the large model checks all of them in a single step, keeping the ones it agrees with. Because the large model has the final say, the output is statistically identical to what it would produce alone. While speculative decoding can make responses two to three times faster, the speed-up depends on how often the draft model guesses correctly.
Speculative decoding, explained at three levels
At home
Imagine a junior writer quickly drafting the next few words of a letter, and a senior editor checking the whole draft at once, keeping everything up to the first word they'd change. When the junior guesses well, several words get approved in the time it takes the editor to write one. Speculative decoding pairs a small, fast AI model with a large one in exactly this way.
At work
Speculative decoding is a technique that makes large AI models generate text faster, often two to three times, without changing the quality of their answers. A small draft model guesses ahead and the large model verifies those guesses in bulk. For a business, it means quicker responses and more users per GPU, usually through a setting in the serving software rather than new hardware.
At home
A big language model normally writes one token at a time, and each step requires reading all its billions of weights from memory, which is the slow part. Checking five tokens at once costs about the same as writing one. So a small helper model guesses five tokens, the big model checks them together, keeps the ones it agrees with, and supplies its own token at the first mismatch.
At work
The speed-up depends on the acceptance rate, meaning how often the large model agrees with draft tokens, plus the draft model's own cost and how many tokens are proposed per round. Draft sources vary: a separate small model from the same family, extra prediction heads as in Medusa, feature-level drafting as in EAGLE, or n-gram lookup that copies text from the prompt. Predictable output such as code, structured formats, and document edits gets the most drafts accepted.
At home
The clever part is the acceptance rule. The large model compares its own probability for each drafted token with the draft model's, accepting or rejecting in a way that is proven to produce the same output distribution as the large model working alone. So it isn't an approximation; you get the large model's quality, just faster. In my opinion, that guarantee is what makes it such a clever idea.
At work
Speculative decoding exploits idle compute: generating for a single request is limited by memory bandwidth, not arithmetic. At high batch sizes the GPU is already busy, so verification work competes with other requests and gains shrink or vanish. Implementations in vLLM, TensorRT-LLM, and SGLang support draft models, Medusa or EAGLE heads, n-gram lookup, and tree-based verification of several candidate branches. I'd measure latency and throughput under realistic load before enabling it, since the best setting varies with traffic.
Real-world examples
At home
Jerome noticed his AI coding assistant became much faster at filling in repetitive code. The provider had added speculative decoding, so predictable lines were confirmed in batches.
At work
Fatima runs AI infrastructure at a Montreal fintech startup. Enabling speculative decoding on their self-hosted model cut average response times noticeably without any change in output.
How it compares
| feature | Speculative decoding | Model Distillation |
|---|---|---|
| Output quality | Identical to large model | Smaller model, some quality loss |
| Models needed | Large model plus a draft | Only the small model at runtime |
| Speed gain | Often 2 to 3 times | Larger, but with trade-offs |
| Cost | Still runs the large model | Cheaper to serve |
Speculative decoding speeds up the same model. Distillation replaces it with a smaller one.
For analysts
If AI response times are hurting user experience, I'd ask whether speculative decoding is enabled before paying for bigger hardware. It's often a configuration change.
History & origin
Generating text one token at a time leaves GPUs underused, since each step waits for the last. In November 2022, Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google published 'Fast Inference from Transformers via Speculative Decoding,' and a DeepMind team led by Charlie Chen independently described 'speculative sampling' in early 2023. Variants followed, including Medusa in 2024, which added extra prediction heads to the main model instead of a separate draft model, and EAGLE, which predicted at the feature level. Serving engines such as vLLM and TensorRT-LLM added support, and major AI providers use forms of it in production. In my view, speculative decoding is a rare free lunch in AI: faster answers with no loss in quality.