Modern AI Engineering

Lesson 4.8 · 29 min · video lesson

Self-Attention: How Tokens See One Another

In “The animal didn't cross the street because it was too tired”, how does a model work out that “it” means the animal and not the street?

In short: Self-attention lets every token in a sequence look at every other token and build a new, context-aware version of itself. Each token produces a query, a key and a value; queries are compared with keys by dot product, scaled by √dₖ, turned into weights with softmax, and used to average the values. Running several of these in parallel (multi-head attention) lets a Transformer capture many kinds of relationships at once.

What you will learn

  • What is Self Attention?
  • Why do we need Self Attention?
  • Query, Key, and Value vectors
  • Step-by-step working of Self Attention
  • A simple example walk-through
  • Why Self Attention works so well
  • Multi-Head Self Attention
  • Where Self Attention is used

In this lesson

  1. What is Self Attention?
  2. Why do we need Self Attention?
  3. Query, Key, and Value vectors
  4. Step-by-step working of Self Attention
  5. A simple example walk-through
  6. Why Self Attention works so well
  7. Multi-Head Self Attention
  8. Where Self Attention is used
  9. Common mistakes and how to spot them
  10. Practice: try it yourself

Key terms in this lesson

Self-attention
An operation where every token in a sequence attends to every token of the same sequence to update its representation.
Query
A vector describing what a token is looking for, compared against keys.
Key
A vector describing what a token offers, matched against queries.
Value
A vector holding the information a token contributes to others' outputs.
Scaled dot-product attention
softmax(Q·Kᵀ / √dₖ) · V, the standard attention formula.
Attention head
One independent attention operation with its own Q, K, V projections.
Softmax
A function that turns a list of scores into positive weights summing to 1.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.7 Encoder vs Decoder: Two Sides of the Transformer · 4.9 Attention Math: Queries, Keys, and Values Unpacked →