Speculative decoding
Also known as: speculative sampling, assisted generation, draft-model decoding
Speculative decoding is a technique for making large AI models generate text faster.
A small, fast 'draft' model guesses the next several tokens, and the large model checks all of them in a single step, keeping the ones it agrees with. Because the large model has the final say, the output matches what it would have produced alone. While speculative decoding can make responses two to three times faster, the speed-up depends on how often the draft model guesses correctly.
Speculative decoding, explained at three levels
At home
Imagine a junior writer quickly drafting the next few words and a senior editor approving them all at once. If the draft is good, the work goes much faster. That's speculative decoding.
At work
Speculative decoding makes AI products feel faster and lets servers handle more requests, without changing answer quality.
At home
The big model would normally write one word at a time. With a small helper guessing ahead, it can confirm several words in one step whenever the guesses are right.
At work
Speed-up depends on the acceptance rate of draft tokens, the cost of the draft model, and how many tokens are proposed. It works especially well for predictable text such as code, structured formats, and repetitive edits.
At home
It might seem like cheating, but the result is mathematically the same as the big model working alone. In my opinion, that's what makes it such a clever idea.
At work
Implementation choices include separate draft models, self-speculation with extra heads, n-gram lookup from the prompt, and tree-based verification. Gains shrink at high batch sizes where GPUs are already busy, so benefits should be measured under real load.
Real-world examples
At home
Jerome noticed his AI coding assistant became much faster at filling in repetitive code. The provider had added speculative decoding, so predictable lines were confirmed in batches.
At work
Fatima runs AI infrastructure at a Montreal fintech startup. Enabling speculative decoding on their self-hosted model cut average response times noticeably without any change in output.
How it compares
| feature | Speculative decoding | Model Distillation |
|---|---|---|
| Output quality | Identical to large model | Smaller model, some quality loss |
| Models needed | Large model plus a draft | Only the small model at runtime |
| Speed gain | Often 2 to 3 times | Larger, but with trade-offs |
| Cost | Still runs the large model | Cheaper to serve |
Speculative decoding speeds up the same model. Distillation replaces it with a smaller one.
For analysts
If AI response times are hurting user experience, I'd ask whether speculative decoding is enabled before paying for bigger hardware. It's often a configuration change.
History & origin
Generating text one token at a time leaves GPUs underused, since each step waits for the last. In November 2022, Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google published 'Fast Inference from Transformers via Speculative Decoding,' and a DeepMind team led by Charlie Chen independently described 'speculative sampling' in early 2023. Variants followed, including Medusa in 2024, which added extra prediction heads to the main model instead of a separate draft model, and EAGLE, which predicted at the feature level. Serving engines such as vLLM and TensorRT-LLM added support, and major AI providers use forms of it in production. In my view, speculative decoding is a rare free lunch in AI: faster answers with no loss in quality.