Lesson 3.5 · 23 min · video lesson
Dropout: Controlled Forgetting as Regularization
Why would switching off random neurons on purpose, every single training step, make a neural network smarter?
In short: Dropout is a regularisation technique: during training, each neuron's output is set to zero with probability p, and the survivors are scaled up by 1/(1 − p). This stops neurons from relying on specific partners and acts like training many thinned networks at once, which reduces overfitting. At test time dropout is switched off and the full network is used.
What you will learn
- What is Dropout?
- The problem of Overfitting
- Why do we need Dropout?
- How does Dropout work?
- A step-by-step example
- Dropout during training vs testing
- Dropout in code
- Variants of Dropout
- Advantages of Dropout
- Where Dropout is used
In this lesson
- What is dropout?
- The problem of overfitting
- Why do we need dropout?
- How does dropout work?
- A step-by-step example
- Dropout during training vs testing
- Dropout in code with a framework
- Variants of dropout
- Advantages, where dropout is used, and when not to use it
- Going one level deeper
- Practice: try it yourself
Key terms in this lesson
- Dropout
- A regularisation method that randomly sets a fraction of activations to zero during training.
- Dropout rate (p)
- The probability that any given unit is dropped on a training step.
- Overfitting
- When a model fits the training data, including noise, so well that it performs worse on new data.
- Inverted dropout
- Dropout that scales kept activations by 1/(1 − p) during training so nothing changes at inference.
- Co-adaptation
- Neurons relying on specific other neurons to correct them, forming fragile features.
- Monte Carlo dropout
- Keeping dropout on at inference and averaging many runs to estimate uncertainty.
This lesson is free. Sign in or create a free account to open it with its video and quiz.
← 3.4 Cross-Entropy Loss: Scoring Probability Predictions · 3.6 Batch Norm vs Layer Norm: When to Use Each →