Modern AI Engineering

Lesson 4.2 · 27 min · video lesson

Autoregressive Models: Predicting One Token at a Time

Why does ChatGPT type its answer one word at a time instead of showing the whole reply at once?

In short: An autoregressive model generates a sequence one piece at a time, and each new piece is predicted from all the pieces before it. The chain rule of probability says this is a complete way to describe any sequence, which is why GPT-style language models use it. It needs a causal mask during training, benefits hugely from a KV cache during generation, and is accurate but inherently sequential and therefore slow for long outputs.

What you will learn

  • What is an Autoregressive Model?
  • The Chain Rule of Probability
  • The Generation Loop
  • Step-by-Step Numeric Example
  • Why GPT-style Models are Autoregressive
  • Why Autoregressive Models Need Causal Masking
  • The Connection with KV Cache
  • Autoregressive vs Non-Autoregressive Generation
  • Popular Autoregressive Models we should know
  • Pros and Cons of Autoregressive Models
  • Quick Summary

In this lesson

  1. What is an Autoregressive Model?
  2. The Chain Rule of Probability
  3. The Generation Loop
  4. Step-by-Step Numeric Example in Code
  5. Why GPT-style Models are Autoregressive
  6. Why Autoregressive Models Need Causal Masking
  7. The Connection with KV Cache
  8. Autoregressive vs Non-Autoregressive Generation
  9. Popular Autoregressive Models we should know
  10. Worked example, step by step
  11. Practice: try it yourself
  12. Pros, Cons and Quick Summary

Key terms in this lesson

Autoregressive model
A model that generates a sequence one element at a time, each conditioned on the elements before it.
Chain rule of probability
The identity that a joint probability equals the product of each element's probability given the previous ones.
Teacher forcing
Training by feeding the true previous tokens as input and predicting every next token in parallel.
Causal mask
A mask that blocks attention to future positions so each token only uses itself and the past.
KV cache
Stored keys and values of already-processed tokens, reused at each generation step.
Greedy decoding
Always picking the most probable next token.
Exposure bias
The gap between training on true prefixes and generating from the model's own, possibly flawed, prefixes.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.1 Generative AI: Creating Instead of Classifying · 4.3 BPE Tokenization: How LLMs Split Text into Tokens →