Modern AI Engineering

Lesson 3.7 · 22 min · video lesson

RMSNorm: Simpler Normalization for Transformers

Most modern open LLMs quietly deleted half of LayerNorm, the mean subtraction and the bias, and trained just as well. Why does the simpler version work?

In short: RMSNorm normalizes a vector by dividing it by its root mean square, x / √(mean(x²) + ε), then multiplies by a learned scale γ. Unlike LayerNorm it does not subtract the mean and has no shift β. It keeps the property that matters most, control over the scale of activations, while being simpler and somewhat cheaper, which is why many modern LLMs such as the Llama family use it.

What you will learn

  • Why normalization is needed in deep networks
  • A quick recap of Layer Normalization (LayerNorm)
  • What RMSNorm is and how it works
  • The math behind RMSNorm with a concrete numeric example
  • LayerNorm vs RMSNorm - the key differences
  • Why modern LLMs prefer RMSNorm
  • A code example
  • Where RMSNorm fits in a Transformer
  • Quick Summary

In this lesson

  1. Why normalization is needed in deep networks
  2. A quick recap of Layer Normalization (LayerNorm)
  3. What RMSNorm is and how it works
  4. The math behind RMSNorm with a concrete numeric example
  5. LayerNorm vs RMSNorm: the key differences
  6. Why modern LLMs prefer RMSNorm
  7. A code example
  8. Where RMSNorm fits in a Transformer
  9. Common mistakes and how to spot them
  10. Practice: try it yourself
  11. Quick summary

Key terms in this lesson

RMSNorm
A normalization layer that divides a vector by its root mean square and multiplies by a learned scale.
Root mean square (RMS)
The square root of the average of squared values; a measure of a vector's typical magnitude.
Re-centring
Subtracting the mean so values average to zero; done by LayerNorm, skipped by RMSNorm.
Scale invariance
The output stays the same when the input is multiplied by a positive constant.
Pre-norm
A Transformer layout that normalizes the input of each sub-layer while leaving the residual stream untouched.
Residual stream
The running per-token vector that each Transformer block reads from and adds its output to.

This lesson is free. Sign in or create a free account to open it with its video and quiz.

← 3.6 Batch Norm vs Layer Norm: When to Use Each · 3.8 Recurrent Neural Networks: Processing Sequences in Order →