Modern AI Engineering

Lesson 4.10 · 25 min · video lesson

Scaled Dot-Product Attention: Why We Divide by √dₖ

Why does the attention formula divide by √dₖ, and not by dₖ, or 2, or nothing at all? A short piece of probability gives the exact answer.

In short: A dot product of two dₖ-long vectors with random, unit-variance entries has variance dₖ, so its typical size grows like √dₖ. Large scores push softmax into a near one-hot output where gradients almost vanish and learning stalls. Dividing by √dₖ brings the variance back to 1 for any vector width, which keeps softmax soft and trainable.

What you will learn

  • The Attention Formula (Quick Recap)
  • What Happens Without Scaling?
  • Why Do Dot Products Grow with dₖ?
  • Understanding Variance of the Dot Product
  • Proving It Step by Step: Variance of the Dot Product is dₖ
  • What Large Dot Products Do to Softmax
  • Why √dₖ is the Right Scaling Factor
  • Seeing It with Real Numbers
  • Putting It All Together

In this lesson

  1. The attention formula (quick recap)
  2. What happens without scaling?
  3. Why do dot products grow with dₖ?
  4. Understanding variance of the dot product
  5. Proving it step by step: the variance is dₖ
  6. What large dot products do to softmax
  7. Why √dₖ is the right scaling factor
  8. Seeing it with real numbers
  9. Common mistakes and how to spot them
  10. Practice: try it yourself
  11. Putting it all together

Key terms in this lesson

Variance
The average squared distance of a random value from its mean; it measures spread.
Standard deviation
The square root of the variance: the typical size of the spread, in the original units.
Saturation
When softmax inputs are so far apart that the output is nearly one-hot and its slope is nearly zero.
Vanishing gradient
A training signal so small that the affected weights effectively stop updating.
Scaled dot-product attention
Attention where query-key dot products are divided by √dₖ before softmax.
Jacobian of softmax
The table of slopes ∂pᵢ/∂zⱼ = pᵢ(δᵢⱼ − pⱼ) describing how outputs respond to inputs.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.9 Attention Math: Queries, Keys, and Values Unpacked · 4.11 Causal Masking: Preventing the Model from Seeing the Future →