Modern AI Engineering

Lesson 13.8 · 28 min · video lesson

Speculative Decoding: Draft Fast, Verify in Parallel

How can a big LLM write two or three tokens per forward pass without changing a single word of what it would have written anyway?

In short: LLM generation is slow because each token needs a full forward pass of a big model that is limited by memory bandwidth, not math. Speculative decoding lets a small, fast draft model guess several tokens ahead; the big target model then checks all the guesses in one parallel pass and keeps the longest correct prefix. With the right acceptance rule the output distribution is exactly the target model's, and real systems typically see around 2–3× faster decoding.

What you will learn

  • What problem does Speculative Decoding solve?
  • The Big Picture
  • Why is LLM generation slow?
  • The core idea behind Speculative Decoding
  • Step-by-step walkthrough
  • The verification step
  • Real numbers and speedup
  • Where it is used
  • Trade-offs
  • Quick Summary

In this lesson

  1. What problem speculative decoding solves
  2. Why LLM generation is slow
  3. The core idea and the big picture
  4. Step-by-step walkthrough
  5. The verification step: why output stays exact
  6. Real numbers and speedup
  7. Where it is used, and the trade-offs
  8. Worked example, step by step
  9. Practice: try it yourself
  10. Quick summary

Key terms in this lesson

Speculative decoding
Generating several tokens per target-model pass by drafting guesses cheaply and verifying them in parallel.
Target model
The large model whose output distribution we want to reproduce.
Draft model
A small, fast model (or cheap mechanism) that proposes candidate tokens.
Acceptance rate (α)
The probability a draft token is accepted; equal to ∑ min(p, q) for one position.
Speculative sampling
The accept-or-resample rule that keeps the output distribution identical to the target's.
Bonus token
The extra token sampled from the target's last position when all drafts are accepted.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.7 Continuous Batching: Keeping GPUs Busy Between Requests · 13.9 N-gram Speculation: Draft Tokens Without a Draft Model →