Modern AI Engineering

Lesson 13.9 · 25 min · video lesson

N-gram Speculation: Draft Tokens Without a Draft Model

When an LLM is rewriting your code and most of the answer is copied from the prompt, why should it generate every copied token the slow way?

In short: Speculative decoding speeds up generation by letting cheap guesses be verified by the big model in one pass, but a separate draft model costs memory, engineering and compute. N-gram speculation drops the draft model: it finds the last few generated tokens earlier in the text (usually the prompt) and proposes whatever followed them there. The big model verifies the guesses as usual, so the output is unchanged, and on copy-heavy tasks like code editing, summarisation with quotes or RAG answers, many tokens arrive per pass almost for free.

What you will learn

  • How an LLM generates text
  • Why generating text is slow
  • What is Speculative Decoding
  • The cost of a draft model
  • What is an N-gram
  • What is N-gram Speculation
  • N-gram Speculation step by step
  • Why the output stays exactly the same
  • Where it works well and where it fails
  • N-gram Speculation vs Draft Model Speculative Decoding

In this lesson

  1. How an LLM generates text, and why it is slow
  2. Speculative decoding and the cost of a draft model
  3. What an n-gram is
  4. What n-gram speculation is, step by step
  5. Code: prompt lookup on a bug fix
  6. Why the output stays exactly the same
  7. Where it works well and where it fails
  8. N-gram speculation vs draft-model speculative decoding
  9. Going one level deeper
  10. Practice: try it yourself

Key terms in this lesson

N-gram
A sequence of n consecutive tokens.
N-gram speculation
Speculative decoding where drafts are copied from earlier text that follows a matching n-gram.
Prompt lookup decoding
Another name for n-gram speculation that searches mainly in the prompt.
Draft
The k candidate tokens proposed before the target model verifies them.
Target model
The main LLM whose output must be preserved.
Acceptance rate
The fraction of draft tokens the target model confirms.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.8 Speculative Decoding: Draft Fast, Verify in Parallel · 13.10 Medusa: Parallel Decoding via Multiple Prediction Heads →