Modern AI Engineering

Lesson 13.3 · 26 min · video lesson

Prefill-Decode Disaggregation: Splitting the Two Phases

What if the GPU that reads your prompt and the GPU that writes your answer were two different machines, and that made both faster?

In short: Prefill is compute-heavy and decode is memory-heavy, so when both share one GPU they get in each other's way: a long prompt arriving can freeze everyone else's token stream. Prefill-decode disaggregation runs the two phases on separate GPU pools and ships the KV cache between them. It gives steadier TTFT and TPOT and lets each pool be tuned on its own, at the cost of KV transfer, a fast network and more operational complexity.

What you will learn

  • How an LLM answers a request
  • What is the KV Cache?
  • Prefill is compute-heavy, Decode is memory-heavy
  • The problem when both run on the same GPU
  • TTFT vs TPOT
  • The naive approaches and their issues
  • What is Prefill-Decode Disaggregation?
  • How Prefill-Decode Disaggregation works
  • Walkthrough of one request
  • Advantages of Prefill-Decode Disaggregation
  • Disadvantages of Prefill-Decode Disaggregation
  • Where it works well and where it is overkill
  • Co-located vs Disaggregated serving

In this lesson

  1. How an LLM answers a request, and what the KV cache is
  2. Prefill is compute-heavy, decode is memory-heavy
  3. The problem when both run on the same GPU
  4. TTFT vs TPOT: two promises that pull apart
  5. The naive approaches and their issues
  6. What prefill-decode disaggregation is and how it works
  7. Advantages, disadvantages and where it fits
  8. Going one level deeper
  9. Practice: try it yourself

Key terms in this lesson

Co-located serving
Running both prefill and decode on the same GPUs.
Prefill-decode disaggregation
Serving prefill and decode on separate GPU pools, moving the KV cache from one to the other.
Interference
Slowdown one phase causes the other when they share a GPU, such as a prefill stalling decode steps.
Chunked prefill
Splitting a long prompt's prefill into smaller pieces that are mixed into decode batches.
Goodput
Throughput counting only requests that meet their latency targets.
KV transfer
Copying a request's key/value cache from a prefill worker to a decode worker over a fast interconnect.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.2 Prefill vs Decode: Two Distinct Phases of LLM Inference · 13.4 The KV Cache: Avoiding Redundant Attention Computation →