Modern AI Engineering

Lesson 3.2 · 25 min · video lesson

Gradient Descent: Rolling Downhill to the Optimum

A model with billions of numbers has no map of the best settings, so how does it find them by only ever looking at the ground under its feet?

In short: Gradient descent is the algorithm that trains almost every neural network. It measures how wrong the model is with a loss function, computes the gradient (the direction in which the loss rises fastest), and nudges every parameter a small step the opposite way: w ← w − η · ∂L/∂w. Repeating this many times walks the parameters downhill to a low loss.

What you will learn

  • The Big Picture
  • What is a Loss Function
  • What is Gradient Descent
  • The Intuition Behind Gradient Descent
  • The Math Behind Gradient Descent
  • Step-by-Step Numeric Example
  • Gradient Descent with Multiple Parameters
  • The Role of Learning Rate
  • Types of Gradient Descent
  • Gradient Descent in Python
  • Putting It All Together

In this lesson

  1. The big picture
  2. What is a loss function?
  3. What gradient descent is, and the intuition
  4. The math: the update rule
  5. Step-by-step numeric example
  6. Gradient descent with multiple parameters
  7. The role of the learning rate
  8. Types of gradient descent
  9. Gradient descent in Python
  10. Going one level deeper
  11. Practice: try it yourself
  12. Putting it all together

Key terms in this lesson

Loss function
A function that turns the model's predictions and the true answers into one number measuring error; lower is better.
Gradient
The vector of partial derivatives of the loss with respect to every parameter; it points in the direction of steepest increase.
Learning rate (η)
A hyperparameter setting how big a step gradient descent takes each update.
Epoch
One full pass through the training dataset.
Mini-batch
A small random subset of training examples used to compute one gradient update.
Divergence
When updates overshoot so badly that the loss grows instead of shrinking, often ending in NaN.

This lesson is free. Sign in or create a free account to open it with its video and quiz.

← 3.1 Neural Network Bias: What It Is and Why It Matters · 3.3 Backpropagation: How Neural Networks Learn from Mistakes →