Modern AI Engineering

Lesson 13.15 · 27 min · video lesson

vLLM: High-Throughput Serving with PagedAttention

Why could the same GPU and the same model serve several times more users simply by changing how memory for past tokens is organised?

In short: vLLM is an open-source engine for serving LLMs to many users at once. Its core idea, PagedAttention, stores each request's KV cache in small fixed-size blocks that can live anywhere in GPU memory, like pages in an operating system, so almost no memory is wasted and identical prefixes can be shared. Combined with continuous batching and an OpenAI-compatible API server, this lets one GPU run far bigger batches and deliver much higher throughput.

What you will learn

  • What is serving an LLM
  • A quick recap of prefill, decode, and the KV cache
  • The problem: the KV cache eats GPU memory
  • Why naive serving wastes memory
  • What is vLLM
  • PagedAttention, the core idea
  • How PagedAttention shares memory
  • Continuous batching
  • The OpenAI-compatible API server
  • The benefits of vLLM
  • vLLM in the real world

In this lesson

  1. What is serving an LLM
  2. A quick recap of prefill, decode and the KV cache
  3. The problem: the KV cache eats GPU memory
  4. Why naive serving wastes memory
  5. What is vLLM
  6. PagedAttention, the core idea
  7. How PagedAttention shares memory
  8. Continuous batching
  9. The OpenAI-compatible API server
  10. The benefits of vLLM, and vLLM in the real world
  11. Worked example, step by step
  12. Practice: try it yourself

Key terms in this lesson

Serving
Running a model as a service that answers many users' requests concurrently.
KV cache
Stored keys and values of earlier tokens, reused at every decoding step to avoid recomputation.
PagedAttention
vLLM's method of storing the KV cache in fixed-size blocks anywhere in memory, mapped by a block table.
Block table
A per-request list mapping logical KV blocks to physical blocks in GPU memory.
Copy-on-write
Sharing a block until someone needs to change it, then giving the writer its own copy.
Continuous batching
Re-forming the batch at every decoding step so requests join and leave without waiting for each other.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.14 llama.cpp: Running Large Models on Consumer Hardware · 13.16 SGLang: Structured LLM Programs for Efficient Inference →