Modern AI Engineering

Lesson 4.12 · 25 min · video lesson

Multi-Head Attention: Many Perspectives at Once

One attention pattern per token is a single opinion about what matters. What if each layer could hold a dozen different opinions at once, for the same cost?

In short: Multi-head attention runs several smaller attention operations ("heads") side by side. Each head has its own query, key and value projections, so it can learn its own pattern, such as "look at the previous word" or "look at the subject". The head outputs are concatenated and mixed by one more matrix, W_o. Because each head works in d_model / h dimensions, the total cost is about the same as one big head.

What you will learn

  • What is Multi-Head Attention?
  • A quick recap of Self Attention
  • Why do we need Multi-Head Attention?
  • Step-by-step working of Multi-Head Attention
  • A simple example walk-through
  • Where Multi-Head Attention is used
  • Advantages of Multi-Head Attention

In this lesson

  1. What is multi-head attention?
  2. A quick recap of self-attention
  3. Why do we need multi-head attention?
  4. Step-by-step working of multi-head attention
  5. Same cost as one big head
  6. A simple example walk-through
  7. Single-head vs multi-head
  8. Where multi-head attention is used and why it helps
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

Head
One independent attention computation with its own query, key and value projections.
d_head
The width of each head, usually d_model divided by the number of heads.
Concatenation
Placing the head outputs side by side to rebuild a d_model-wide vector.
Output projection (W_o)
The learned matrix that mixes the concatenated head outputs.
Self-attention
Attention where queries, keys and values all come from the same sequence.
Induction head
A kind of head found in trained models that helps continue patterns seen earlier in the context.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.11 Causal Masking: Preventing the Model from Seeing the Future · 4.13 Cross-Attention: Connecting Encoder Output to the Decoder →