Lesson 3.8 · 25 min · video lesson

Recurrent Neural Networks: Processing Sequences in Order

How can a network read "the movie was not good" one word at a time and still remember the "not" by the time it reaches "good"?

In short: A recurrent neural network (RNN) processes a sequence one step at a time and carries a hidden state, a small vector of memory, from each step to the next. The same weights are reused at every step: hₜ = tanh(Wₓₕxₜ + Wₕₕhₜ₋₁ + b). RNNs are trained with backpropagation through time, struggle with long-range memory because gradients vanish or explode, and led to LSTMs, GRUs and eventually Transformers.

In this lesson

Why ordinary networks struggle with sequences
The core idea: a loop with memory
How it works, step by step with numbers
Input and output shapes: what RNNs can do
Training: backpropagation through time
LSTM and GRU: RNNs with gates
RNNs vs Transformers
Real-world use, pitfalls and when not to use an RNN
Worked example, step by step
Practice: try it yourself

Key terms in this lesson

Recurrent neural network (RNN): A network that processes a sequence one step at a time, passing a hidden state from each step to the next.
Hidden state: The vector an RNN carries between steps that summarises what it has seen so far.
Unrolling: Drawing an RNN as a chain of identical cells, one per time step, sharing the same weights.
Backpropagation through time (BPTT): Backpropagation applied to the unrolled RNN, summing gradients for the shared weights over all steps.
Vanishing / exploding gradients: Gradients that shrink towards zero or grow huge because they are products of many factors.
LSTM: A gated RNN with a cell state and forget, input and output gates that preserve information over long spans.
Gradient clipping: Rescaling gradients whose norm exceeds a threshold to prevent unstable updates.

This lesson is free. Sign in or create a free account to open it with its video and quiz.

← 3.7 RMSNorm: Simpler Normalization for Transformers · 3.9 PyTorch Internals: Dynamic Graphs and Autograd →