Modern AI Engineering

Lesson 13.1 · 27 min · video lesson

LLM Inference Optimization: The Full Landscape

Why does a chatbot that fits easily on one GPU suddenly run out of memory when eight people paste in long documents at the same time?

In short: An LLM writes one token at a time, and to avoid redoing work it stores a Key and a Value vector for every past token in every layer: the KV cache. That cache grows with context length and with the number of users, and it quickly becomes the main memory cost of serving. This lesson shows where the memory goes and walks through the four big families of KV cache compression (quantization, token eviction, sharing keys and values across heads, and low-rank compression) and when to pick each.

What you will learn

  • What is an LLM and how it writes text
  • What is Attention
  • What is the KV Cache
  • Why the KV Cache becomes huge
  • What is KV Cache Compression
  • Approach 1: Quantization
  • Approach 2: Token Eviction
  • Approach 3: Sharing Keys and Values across Heads
  • Approach 4: Low-Rank Compression
  • Comparison of the approaches
  • When to use which one

In this lesson

  1. What an LLM is and how it writes text
  2. What attention does
  3. What the KV cache is
  4. Why the KV cache becomes huge
  5. What KV cache compression is
  6. The four approaches
  7. Code: a KV cache budget for our chatbot
  8. Comparison of the approaches
  9. When to use which one
  10. Worked example, step by step
  11. Practice: try it yourself

Key terms in this lesson

Autoregressive generation
Producing text one token at a time, each token conditioned on all previous ones.
KV cache
Stored key and value vectors of all processed tokens, for every layer and head, reused at each decoding step.
Attention head
One of several parallel attention computations in a layer, each with its own query, key and value projections.
Quantization
Representing numbers with fewer bits plus a scale, trading a little accuracy for memory.
Token eviction
Dropping cached keys and values of selected tokens to keep the cache within a budget.
Grouped-query attention (GQA)
An attention design where groups of query heads share a single key/value head.
Multi-head Latent Attention (MLA)
A low-rank attention design that caches a compact latent vector per token and expands it into keys and values.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 12.8 Cursor: Inside an AI-Native Code Editor · 13.2 Prefill vs Decode: Two Distinct Phases of LLM Inference →