Lesson 3.6 · 23 min · video lesson
Batch Norm vs Layer Norm: When to Use Each
Two layers do almost the same arithmetic, subtract a mean and divide by a standard deviation, so why does one rule image networks and the other rule every Transformer?
In short: Normalization layers rescale activations so they have a mean of about 0 and a standard deviation of about 1, then apply a learned scale and shift. Batch Normalization computes those statistics for each feature across the examples in a mini-batch; Layer Normalization computes them for each example across its features. That single difference in direction decides where each one works best.
What you will learn
- What is Normalization?
- Why do we need Normalization?
- What is Batch Normalization?
- What is Layer Normalization?
- Batch Normalization vs Layer Normalization
- When to use which one?
In this lesson
- What is normalization?
- Why do we need normalization?
- What is Batch Normalization?
- What is Layer Normalization?
- Both, side by side in code
- Batch Normalization vs Layer Normalization
- When to use which one?
- Worked example, step by step
- Practice: try it yourself
Key terms in this lesson
- Normalization
- Rescaling a group of values to mean 0 and standard deviation 1, then applying a learned scale and shift.
- Batch Normalization
- Normalizing each feature using the mean and variance across the examples of a mini-batch.
- Layer Normalization
- Normalizing each example using the mean and variance across its own features.
- Running statistics
- Moving averages of BatchNorm's mean and variance collected during training and used at inference.
- γ and β
- Learned per-feature scale and shift applied after normalization.
- GroupNorm
- A batch-independent variant that normalizes groups of channels within each example.
This lesson is free. Sign in or create a free account to open it with its video and quiz.
← 3.5 Dropout: Controlled Forgetting as Regularization · 3.7 RMSNorm: Simpler Normalization for Transformers →