Lesson 2.7 · 23 min · video lesson
Regularization: Stopping Overfitting with L1 and L2
Our model fits the training data almost perfectly, then does far worse on new data. Can a tiny penalty on the size of its weights fix that?
In short: Overfitting happens when a model learns the noise in its training data instead of the real pattern. Regularisation fights it by adding a penalty on the model's weights to the loss, controlled by a strength λ. L1 (Lasso) adds λ·∑|w|, which drives many weights to exactly zero and so selects features; L2 (Ridge) adds λ·∑w², which shrinks all weights smoothly towards zero. We choose λ using validation data.
What you will learn
- What is overfitting?
- L1 Regularization or Lasso Regularization
- L2 Regularization or Ridge Regularization
In this lesson
- What is overfitting?
- The core idea: penalise complexity
- L1 regularisation (Lasso)
- L2 regularisation (Ridge)
- Code: taming a degree-9 polynomial
- L1 vs L2 regularisation side by side
- Regularisation in practice
- Worked example, step by step
- Practice: try it yourself
Key terms in this lesson
- Overfitting
- When a model fits training data, including noise, so closely that it generalises poorly.
- Regularisation
- Adding a penalty to the training objective that discourages overly complex models.
- λ (lambda)
- The regularisation strength that balances fitting the data against keeping weights small.
- Lasso (L1)
- Regularisation using the sum of absolute weights; produces sparse models.
- Ridge (L2)
- Regularisation using the sum of squared weights; shrinks weights smoothly.
- Weight decay
- The name for L2-style regularisation in neural network training.
- Sparsity
- Having many weights exactly equal to zero.
This lesson is free. Sign in or create a free account to open it with its video and quiz.
← 2.6 L1 vs L2 Loss: Choosing Your Error Penalty · 2.8 Reinforcement Learning: Teaching Agents Through Reward →