Modern AI Engineering

Lesson 13.2 · 27 min · video lesson

Prefill vs Decode: Two Distinct Phases of LLM Inference

Why can a GPU read your 4,000-token prompt in a fraction of a second, yet take several seconds to write a 300-token answer?

In short: Every LLM request runs in two phases. Prefill processes the whole prompt in one parallel pass and builds the KV cache; it is limited by raw compute. Decode then produces one token per step, reading all the weights and the cache each time; it is limited by memory bandwidth. Because the phases stress different hardware limits, they have different metrics (TTFT vs TPOT) and different optimizations.

What you will learn

  • What is LLM inference
  • The two phases: Prefill and Decode
  • Prefill explained in simple words
  • Decode explained in simple words
  • A diagram of the two phases and the KV cache flow
  • The KV cache as the bridge between the two phases
  • A step-by-step walkthrough of a few decode steps
  • Prefill vs Decode comparison table
  • Why this split matters: compute-bound vs memory-bound
  • The key metrics: TTFT, TPOT, throughput, and end-to-end latency
  • Optimization techniques mapped to each phase
  • Conclusion

In this lesson

  1. What LLM inference is
  2. The two phases: prefill and decode
  3. A few decode steps, slowly
  4. Prefill vs decode side by side
  5. Why the split matters: compute-bound vs memory-bound
  6. The key metrics: TTFT, TPOT, throughput, latency
  7. Optimization techniques mapped to each phase
  8. Worked example, step by step
  9. Practice: try it yourself
  10. Conclusion and common mistakes

Key terms in this lesson

Prefill
The phase that processes all prompt tokens in one parallel pass, building the KV cache and producing the first token.
Decode
The phase that generates output tokens one per forward pass, reusing and extending the KV cache.
Arithmetic intensity
FLOPs performed per byte moved from memory; low values mean a memory-bound workload.
TTFT
Time to first token: delay from request arrival to the first output token.
TPOT
Time per output token: average gap between consecutive generated tokens.
Throughput
Total tokens or requests a server produces per second across all users.
SLO
Service-level objective: a target such as “TTFT under 1 s for 99% of requests”.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.1 LLM Inference Optimization: The Full Landscape · 13.3 Prefill-Decode Disaggregation: Splitting the Two Phases →