Modern AI Engineering

Lesson 13.7 · 26 min · video lesson

Continuous Batching: Keeping GPUs Busy Between Requests

Why should a user asking “What is 2+2?” have to wait for someone else's 1,000-word essay to finish before getting a GPU slot?

In short: GPUs serve LLMs efficiently only when many requests share each forward pass, so servers batch requests. Static batching keeps a batch together until its longest request finishes, leaving slots idle and newcomers waiting. Continuous batching (iteration-level scheduling) makes batch decisions at every decode step: finished requests leave immediately and waiting ones join, keeping the GPU full and raising throughput several-fold on realistic traffic.

What you will learn

  • The Big Picture
  • Quick Recap: How an LLM Generates Tokens
  • Why Batching Matters for LLMs
  • The Old Way: Static Batching
  • The Problem with Static Batching
  • What is Continuous Batching?
  • The Ride-Share Analogy
  • How Continuous Batching Works Step by Step
  • A Numeric Example
  • Real Numbers and Speedup
  • Benefits of Continuous Batching
  • A Few Important Notes
  • Quick Summary

In this lesson

  1. The big picture
  2. Quick recap: how an LLM generates tokens
  3. Why batching matters for LLMs
  4. The old way: static batching, and its problem
  5. What continuous batching is
  6. A numeric example
  7. Real numbers and speedup
  8. Benefits and important notes
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Quick summary

Key terms in this lesson

Batching
Processing several requests in the same forward pass to share the cost of reading weights.
Static batching
Forming a batch once and running it until every request in it finishes.
Continuous batching
Updating batch membership at every decode iteration; also called iteration-level or in-flight batching.
Iteration
One forward pass that produces one new token for every decoding request in the batch.
Slot utilisation
The share of batch slot-steps that produce useful tokens.
Head-of-line blocking
New work waiting behind long-running work that it does not depend on.
Preemption
Pausing a running request, freeing its memory, and resuming it later.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.6 Paged Attention: OS-Inspired Memory Management for KV Caches · 13.8 Speculative Decoding: Draft Fast, Verify in Parallel →