Modern AI Engineering

Lesson 13.11 · 23 min · video lesson

EAGLE: Feature-Level Drafting for Faster Inference

Why would guessing a model's hidden thoughts be easier than guessing its next word, and how does that make generation three times faster?

In short: EAGLE is a speculative decoding method whose tiny draft network predicts the target model's next internal feature vector (its top-layer hidden state) instead of the next token, and feeds back the token that was actually sampled to remove ambiguity. Its guesses are accepted far more often than a small separate model's, so the big model can confirm several tokens per pass with no change in output. EAGLE-2 adds dynamic draft trees and EAGLE-3 scales the idea further; all three are now built into major serving engines.

What you will learn

  • What is EAGLE
  • A quick recap of speculative decoding
  • The problem with token-level drafting
  • The big idea: draft at the feature level
  • Resolving the uncertainty by feeding back the token
  • The math behind the speedup with small numbers
  • EAGLE-2 and dynamic draft trees
  • How EAGLE lives on today
  • Quick Summary

In this lesson

  1. What is EAGLE?
  2. A quick recap of speculative decoding
  3. The problem with token-level drafting
  4. The big idea: draft at the feature level
  5. Resolving the uncertainty by feeding back the token
  6. The math behind the speedup with small numbers
  7. EAGLE-2 and dynamic draft trees
  8. How EAGLE lives on today
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Quick summary

Key terms in this lesson

Feature (hidden state)
The vector a transformer produces at a position just before the LM head turns it into token probabilities.
Feature uncertainty
The ambiguity that the next feature depends on which token was randomly sampled from the current one.
Draft head
EAGLE's small trained network (about one decoder layer) that proposes future features and tokens.
Acceptance rate (α)
The probability that a drafted token is accepted by the target model given earlier drafts were accepted.
Dynamic draft tree
EAGLE-2's context-dependent tree of candidate continuations, grown where the drafter is confident.
Lossless
Producing exactly the output distribution of the target model alone, despite drafting.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.10 Medusa: Parallel Decoding via Multiple Prediction Heads · 13.12 Model Quantization: Shrinking Weights Without Breaking Outputs →