Modern AI Engineering

Lesson 4.9 · 27 min · video lesson

Attention Math: Queries, Keys, and Values Unpacked

When a Transformer reads "I love cats", how does the word "love" decide how much to listen to "I" and how much to listen to "cats"? It is just four matrix steps, and we can do them by hand.

In short: Attention turns each token into three vectors: a query (what I am looking for), a key (what I offer) and a value (what I will pass on). Dot products between queries and keys give scores, we scale them by √dₖ, softmax turns each row into weights that sum to 1, and the output for each token is the weighted average of the value vectors. We will compute every number for a three-word sentence.

What you will learn

  • The Attention Formula
  • Setting Up: From Words to Vectors
  • Creating Q, K, and V Matrices
  • Computing Attention Scores (Q x K^T)
  • Scaling the Scores
  • Applying Softmax
  • Computing the Final Output (Attention Weights x V)
  • Putting It All Together

In this lesson

  1. The attention formula
  2. Setting up: from words to vectors
  3. Creating the Q, K and V matrices
  4. Computing attention scores (Q × Kᵀ)
  5. Scaling the scores
  6. Applying softmax
  7. Computing the final output (attention weights × V)
  8. Putting it all together in code
  9. How it compares, where it is used and common mistakes
  10. Worked example, step by step
  11. Practice: try it yourself

Key terms in this lesson

Query (Q)
A projection of a token that represents what information it is looking for.
Key (K)
A projection of a token that is matched against queries to decide relevance.
Value (V)
A projection of a token that carries the information passed on when it is attended to.
Dot product
Multiply two vectors position by position and add the results; large when they point the same way.
Softmax
A function that turns a list of numbers into positive weights that sum to 1.
dₖ
The length of each query and key vector; scores are divided by its square root.
Attention weights
The softmax output: how much each token listens to every other token.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.8 Self-Attention: How Tokens See One Another · 4.10 Scaled Dot-Product Attention: Why We Divide by √dₖ →