Modern AI Engineering

Lesson 4.5 · 25 min · video lesson

RNNs vs Transformers: A Fundamental Architecture Shift

Why did almost the whole field switch from RNNs to Transformers within a few years of 2017, and are RNN ideas really gone?

In short: Both RNNs and Transformers process sequences such as text. An RNN reads one token at a time and squeezes everything it has seen into a single memory vector, which makes it slow to train and forgetful over long distances. A Transformer uses self-attention to let every token look directly at every other token in parallel, which trains much faster on GPUs and handles long-range context well, at the cost of compute that grows with the square of the sequence length.

What you will learn

  • What both of them are for
  • What is an RNN?
  • The problem with an RNN
  • What is a Transformer?
  • The key difference in one line
  • RNN vs Transformer side by side
  • Let's tabulate the difference
  • When to use which one?
  • Summary

In this lesson

  1. What both of them are for
  2. What is an RNN?
  3. The problem with an RNN
  4. What is a Transformer?
  5. The key difference in one line
  6. RNN vs Transformer side by side (code)
  7. Let's tabulate the difference
  8. When to use which one?
  9. Going one level deeper
  10. Practice: try it yourself
  11. Summary

Key terms in this lesson

Sequence
An ordered list of items, such as tokens, where position carries meaning.
RNN
A network that processes a sequence one step at a time, updating a hidden state with shared weights.
Hidden state
The RNN's running memory vector, updated after each token.
LSTM
An RNN variant with gates that control what to keep, forget and output.
Vanishing gradient
When the learning signal shrinks towards zero as it travels back through many steps.
Self-attention
An operation where each token computes weights over all tokens and mixes their information.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.4 Embeddings: Encoding Meaning as Vectors · 4.6 The Transformer Architecture: Built on Attention →