Modern AI Engineering

Lesson 6.5 · 24 min · video lesson

Attention Sinks: The Hidden Cost of Extended Context

Why does a chatbot suddenly start writing gibberish the moment we delete the first few, seemingly useless, tokens of a long conversation?

In short: Trained LLMs dump a large share of their attention onto the first few tokens, even when those tokens mean nothing; these tokens are called attention sinks. If a streaming system evicts them to save memory, the attention distribution shifts and the model breaks down. Keeping a handful of sink tokens plus a sliding window of recent tokens (the StreamingLLM recipe), or giving the model a dedicated learned sink, lets it run stably over very long streams.

What you will learn

  • What is a Large Language Model
  • What is attention
  • The problem of streaming with long conversations
  • The naive fix and why it fails
  • What is an attention sink
  • Why the first tokens become a sink
  • A step-by-step numeric walkthrough
  • The fix in code
  • StreamingLLM and modern attention sinks
  • Importance of attention sinks

In this lesson

  1. What is a Large Language Model?
  2. What is attention?
  3. The problem of streaming with long conversations
  4. The naive fix and why it fails
  5. What is an attention sink?
  6. Why the first tokens become a sink
  7. A step-by-step numeric walkthrough
  8. The fix: StreamingLLM
  9. StreamingLLM and modern attention sinks
  10. Importance of attention sinks
  11. Going one level deeper
  12. Practice: try it yourself

Key terms in this lesson

Attention sink
A token, usually the first, that absorbs a large share of attention without adding much information.
KV cache
Stored keys and values of past tokens used during generation.
Window attention
Keeping only the most recent W tokens in the cache.
StreamingLLM
A method that keeps a few initial sink tokens plus a rolling window to stream text indefinitely.
Perplexity
A measure of how surprised a model is by text; lower is better.
Learned sink
An extra learnable attention slot or logit built into each head to absorb unneeded weight.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 6.4 Sliding Window Attention: Taming Very Long Contexts · 6.6 Flash Attention: Memory-Efficient Attention at Scale →