Modern AI Engineering

Lesson 3.9 · 25 min · video lesson

PyTorch Internals: Dynamic Graphs and Autograd

We just derived backpropagation by hand for a network with four parameters. How does a library do it automatically for a model with billions, on a GPU, while you write ordinary Python?

In short: PyTorch is a Python library for building and training neural networks. Its core is the tensor, a multi-dimensional array that can live on a GPU, plus autograd, which records every operation you run into a computation graph and then walks that graph backwards to compute gradients. A training loop is just: forward pass, loss, loss.backward(), optimizer.step().

What you will learn

  • What is PyTorch?
  • What is a Tensor?
  • The problem PyTorch solves
  • What is a Computation Graph?
  • What is Autograd?
  • A complete training example
  • What is the GPU and why does PyTorch use it?
  • Why is PyTorch so popular?

In this lesson

  1. What is PyTorch?
  2. What is a tensor?
  3. The problem PyTorch solves
  4. What is a computation graph?
  5. What is autograd?
  6. A complete training example
  7. What is the GPU and why does PyTorch use it?
  8. Why is PyTorch so popular?
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

PyTorch
An open-source Python deep-learning library built around GPU tensors and automatic differentiation.
Tensor
A multi-dimensional array of numbers with a shape, a data type and a device.
Computation graph
A record of the operations that produced a result, used to compute gradients.
Autograd
PyTorch's reverse-mode automatic differentiation engine that fills each parameter's .grad.
Eager execution
Running each operation immediately as the Python code executes, building the graph on the fly.
Optimizer
An object such as SGD or Adam that updates parameters using their gradients.
GPU
A processor with thousands of simple cores that excels at parallel maths such as matrix multiplication.

This lesson is free. Sign in or create a free account to open it with its video and quiz.

← 3.8 Recurrent Neural Networks: Processing Sequences in Order · 3.10 TensorFlow Explained: Static Graphs and Production ML →