Modern AI Engineering

Lesson 13.5 · 29 min · video lesson

KV Cache Compression: Trading Some Accuracy for Speed

Can we throw away three quarters of an LLM's memory of a conversation and still get nearly the same answer? Sometimes yes, and it depends on which quarter we keep.

In short: The KV cache stores a key and a value vector for every past token in every layer, and for long contexts it outgrows the model itself. KV cache compression shrinks it in four main ways: fewer bits per number (quantization), fewer tokens (eviction), fewer key/value heads (sharing across heads), or fewer dimensions per token (low-rank). This lesson measures the quality cost of each on real attention math and gives a decision guide.

What you will learn

  • What is an LLM and how it writes text
  • What is Attention
  • What is the KV Cache
  • Why the KV Cache becomes huge
  • What is KV Cache Compression
  • Approach 1: Quantization
  • Approach 2: Token Eviction
  • Approach 3: Sharing Keys and Values across Heads
  • Approach 4: Low-Rank Compression
  • Comparison of the approaches
  • When to use which one

In this lesson

  1. How an LLM writes text, and what attention needs
  2. The KV cache and why it becomes huge
  3. What KV cache compression is
  4. Approach 1: Quantization
  5. Approach 2: Token eviction
  6. Approach 3: Sharing keys and values across heads
  7. Approach 4: Low-rank compression
  8. Code: measuring the quality cost of each approach
  9. Comparison, and when to use which
  10. Common mistakes and how to spot them
  11. Practice: try it yourself

Key terms in this lesson

KV cache compression
Storing cached keys and values in fewer bytes while keeping attention outputs close to exact.
Scale (quantization)
The number that maps stored integers back to real values, often max|x| divided by the largest integer level.
Per-channel quantization
Using a separate scale for each vector dimension across tokens, which handles outlier channels in keys.
Attention sink
Early tokens that receive large attention regardless of content; keeping them stabilises eviction-based caches.
Heavy hitter
A token that has received a large share of attention, kept by score-based eviction policies.
Low-rank approximation
Representing a matrix with fewer basis directions, storing short coefficient vectors per row.
Multi-head Latent Attention
DeepSeek's design that caches a compact learned latent per token and reconstructs keys and values from it.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.4 The KV Cache: Avoiding Redundant Attention Computation · 13.6 Paged Attention: OS-Inspired Memory Management for KV Caches →