Modern AI Engineering

Lesson 9.5 · 21 min · video lesson

Context Compaction: Fitting More Into a Finite Window

How can an AI agent keep working for hours when its context window fills up in minutes?

In short: Every LLM has a context window: a fixed maximum number of tokens it can see at once. Long conversations and agent sessions eventually overflow it. Context compaction solves this by replacing older parts of the history with a compact summary that keeps the important facts, decisions and open tasks, while keeping the most recent turns word for word.

What you will learn

  • What is a Large Language Model
  • What is the context window
  • What is context
  • The problem of long conversations
  • The naive fix and why it fails
  • What is context compaction
  • How summarization powers compaction
  • A step-by-step walkthrough
  • Context compaction in code
  • Compaction in real AI agents
  • Why context compaction is important

In this lesson

  1. LLMs, context and the context window
  2. The problem of long conversations
  3. The naive fix, and why it fails
  4. What is context compaction?
  5. How summarization powers compaction
  6. A step-by-step walkthrough
  7. Context compaction in code
  8. Compaction in real AI agents, and why it matters
  9. Common mistakes and how to spot them
  10. Practice: try it yourself

Key terms in this lesson

Context
Everything sent to the model in one call: instructions, history, tool data and the new message.
Context window
The maximum number of tokens a model can process in one call, input plus output.
Truncation
Cutting old messages to make the context fit, without regard to their importance.
Sliding window
Keeping only the last N messages of a conversation.
Context compaction
Replacing older context with a compact summary so the session can continue within the window.
Compaction trigger
The condition, such as a token threshold, that starts a compaction.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 9.4 Context Engineering: Curating the Model's Working Memory · 10.1 Vector Databases: Storing and Searching Embeddings at Scale →