Modern AI Engineering

Lesson 6.4 · 23 min · video lesson

Sliding Window Attention: Taming Very Long Contexts

If each token may only look at its last few thousand neighbours, how can a model still connect ideas that are a hundred thousand tokens apart?

In short: Sliding window attention lets each token attend only to the previous W tokens instead of the whole sequence. That turns attention cost from quadratic to linear in sequence length and caps the KV cache at W entries per layer. Because layers are stacked, information can still hop W tokens per layer, so distant tokens are reachable indirectly, and many modern models mix sliding-window layers with a few full-attention layers.

What you will learn

  • What is attention?
  • The problem with normal attention
  • What is sliding window attention?
  • A simple step-by-step walkthrough
  • How information still travels far away
  • Comparing normal attention and sliding window attention
  • Where sliding window attention is used
  • Advantages and trade-offs

In this lesson

  1. What is attention?
  2. The problem with normal attention
  3. What is sliding window attention?
  4. A simple step-by-step walkthrough
  5. How information still travels far away
  6. Comparing normal attention and sliding window attention
  7. Where sliding window attention is used
  8. Advantages and trade-offs
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

Sliding window attention
Attention where each token sees only the most recent W tokens.
Window size (W)
How many recent tokens, including itself, a token may attend to.
Attention mask
A grid marking which query-key pairs are allowed; blocked pairs get weight 0.
Receptive field
How far back information can reach a token after stacking layers.
Rolling buffer cache
A fixed-size KV cache where new entries overwrite the oldest, using slot i mod W.
Global attention layer
A full-attention layer mixed into a mostly local model to restore long-range access.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 6.3 Grouped Query Attention: Fewer KV Heads, Same Quality · 6.5 Attention Sinks: The Hidden Cost of Extended Context →