Lesson 3.3 · 27 min · video lesson
Backpropagation: How Neural Networks Learn from Mistakes
Gradient descent needs the slope of the loss for every single weight, so how does a network with millions of weights get all of them from one backward sweep?
In short: Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a neural network. It runs a forward pass to get the prediction and loss, then walks backwards layer by layer, using the chain rule to pass an error signal from the output towards the input. Gradient descent then uses those gradients to update the weights.
What you will learn
- What is Backpropagation?
- The Chain Rule of Calculus
- Forward Pass
- Loss Calculation
- Backward Pass (Backpropagation)
- Step-by-Step Numeric Example
- Weight Update Using Gradient Descent
- Backpropagation in Python
In this lesson
- What is backpropagation?
- The chain rule of calculus
- Forward pass
- Loss calculation
- Backward pass (backpropagation)
- Step-by-step numeric example
- Weight update using gradient descent
- Backpropagation in Python
- Alternatives, real-world use and pitfalls
- Worked example, step by step
- Practice: try it yourself
Key terms in this lesson
- Backpropagation
- An algorithm that computes the gradient of the loss for every parameter by applying the chain rule backwards through the network.
- Chain rule
- The calculus rule that the derivative of a composed function is the product of the derivatives of its parts.
- Forward pass
- Running inputs through the network to compute and store activations, the prediction and the loss.
- Error signal (δ)
- The gradient of the loss with respect to a node's pre-activation, passed backwards layer by layer.
- Vanishing gradient
- When gradients become tiny in early layers because many small derivatives are multiplied together.
- Gradient check
- Comparing backprop gradients against finite-difference estimates to detect bugs.
This lesson is free. Sign in or create a free account to open it with its video and quiz.
← 3.2 Gradient Descent: Rolling Downhill to the Optimum · 3.4 Cross-Entropy Loss: Scoring Probability Predictions →