Modern AI Engineering

Lesson 13.6 · 26 min · video lesson

Paged Attention: OS-Inspired Memory Management for KV Caches

Why could an LLM server with plenty of free GPU memory still refuse new users, and how did borrowing an idea from 1960s operating systems fix it?

In short: Early LLM servers reserved one big contiguous chunk of GPU memory per request for its KV cache, sized for the longest possible answer, and wasted most of it. PagedAttention splits each request's KV cache into small fixed-size blocks that can live anywhere in memory, tracked by a block table, just like virtual memory pages in an operating system. Waste drops to a fraction of one block per request, more requests fit in a batch, and blocks can even be shared between requests.

What you will learn

  • Quick Recap: KV Cache
  • The Problem: Memory Waste in KV Cache
  • What is Paged Attention?
  • How Paged Attention Works
  • Why Paged Attention Is So Effective
  • Memory Sharing Across Requests

In this lesson

  1. Quick recap: the KV cache
  2. The problem: memory waste in the KV cache
  3. What PagedAttention is
  4. How PagedAttention works
  5. Why PagedAttention is so effective
  6. Memory sharing across requests
  7. Worked example, step by step
  8. Practice: try it yourself

Key terms in this lesson

PagedAttention
Storing and reading the KV cache in fixed-size non-contiguous blocks mapped by a block table.
Block (page)
A fixed-size chunk of KV cache memory holding a set number of tokens, e.g. 16.
Block table
Per-request list mapping logical block numbers to physical block locations.
Internal fragmentation
Memory reserved for a request that it never uses.
External fragmentation
Free memory split into gaps too small to satisfy a new allocation.
Copy-on-write
Sharing a block until someone writes to it, then giving the writer a private copy.
Reference count
The number of sequences currently pointing to a shared block.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.5 KV Cache Compression: Trading Some Accuracy for Speed · 13.7 Continuous Batching: Keeping GPUs Busy Between Requests →