Modern AI Engineering

Lesson 13.4 · 25 min · video lesson

The KV Cache: Avoiding Redundant Attention Computation

If an LLM re-read its entire conversation before writing every single word, how slow would it be, and how does it avoid that?

In short: When an LLM generates text one token at a time, the keys and values of earlier tokens never change, so recomputing them at every step wastes enormous work. The KV cache stores them once and reuses them, turning quadratic recomputation into linear work per step. The price is memory: the cache grows with every token, layer and user, which is why so much of inference engineering is about managing it.

What you will learn

  • How LLMs Generate Text
  • What Happens Inside the Model
  • The Problem: Repeated Computation
  • The Solution: KV Cache
  • Why Only Key and Value Are Cached, Not Query
  • How Much Faster Does It Get
  • The Trade-Off: Speed vs Memory

In this lesson

  1. How LLMs generate text
  2. What happens inside the model
  3. The problem: repeated computation
  4. The solution: the KV cache
  5. Why only keys and values are cached, not queries
  6. How much faster does it get?
  7. The trade-off: speed vs memory
  8. Going one level deeper
  9. Practice: try it yourself

Key terms in this lesson

Autoregressive generation
Generating text one token at a time, each conditioned on everything before it.
Query, key, value
Three vectors per token in attention: what it seeks, what it can be matched on, and what it passes on.
Causal mask
The rule that a token may attend only to itself and earlier tokens.
KV cache
Stored keys and values of past tokens for every layer and head, reused at each generation step.
Prefill
Processing all prompt tokens in one pass to fill the KV cache before generation starts.
Decode step
One forward pass that processes a single new token using the cache.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.3 Prefill-Decode Disaggregation: Splitting the Two Phases · 13.5 KV Cache Compression: Trading Some Accuracy for Speed →