Modern AI Engineering

Lesson 4.13 · 23 min · video lesson

Cross-Attention: Connecting Encoder Output to the Decoder

When a translation model writes the French word "chats", how does it know to look back at the English word "cats" in a completely different sentence?

In short: Cross-attention is attention between two different sequences. The queries come from the sequence being generated (for example the French decoder), while the keys and values come from another source (the English encoder output, an audio clip, or a text prompt for an image model). The math is the same scaled dot-product attention; only the origin of Q versus K and V changes.

What you will learn

  • What is Cross Attention?
  • Why do we need Cross Attention?
  • Query, Key, and Value in Cross Attention
  • Self Attention vs Cross Attention
  • Step-by-step working of Cross Attention
  • A simple example walk-through
  • Where Cross Attention is used
  • Importance of Cross Attention

In this lesson

  1. What is cross-attention?
  2. Why do we need cross-attention?
  3. Query, key and value in cross-attention
  4. Self-attention vs cross-attention
  5. Step-by-step working of cross-attention
  6. A simple example walk-through
  7. Where cross-attention is used
  8. Why cross-attention matters, and common mistakes
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

Cross-attention
Attention where queries come from one sequence and keys and values from another.
Encoder
The part of a model that reads the source input and produces one vector per input token.
Decoder
The part of a model that generates the output sequence one token at a time.
Encoder-decoder model
A Transformer with an encoder for the input and a decoder that cross-attends to it.
Alignment
Which source positions a target position draws from; cross-attention learns it softly.
Conditioning
Feeding extra information (a prompt, audio, an image) that guides what a model generates.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.12 Multi-Head Attention: Many Perspectives at Once · 4.14 Rotary Position Encoding: Position Without Fixed Lookup Tables →