Modern AI Engineering

Lesson 6.1 · 27 min · video lesson

A Timeline of LLM Architecture Improvements

Why did the field throw away a decade of recurrent networks for one idea called attention, and what has changed since?

In short: LLM architecture went through clear stages: recurrent networks that read one word at a time, attention that lets every word look at every other word, the Transformer built only from attention, a scaling era where bigger models and more data kept paying off, and Mixture of Experts that grows knowledge without growing cost per token. Today the frontier mixes these with new directions such as efficient attention, reasoning models and diffusion-style generation.

What you will learn

  • What is an LLM Architecture?
  • Stage 1: Reading one word at a time (RNN)
  • Stage 2: Attention
  • Stage 3: The Transformer
  • Stage 4: Scaling
  • Stage 5: Mixture of Experts (MoE)
  • Stage 6: New Directions
  • Summary of the evolution

In this lesson

  1. What is an LLM architecture?
  2. Stage 1: Reading one word at a time (RNN)
  3. Stage 2: Attention
  4. Stage 3: The Transformer
  5. Code: an RNN step loop vs one attention pass
  6. Stage 4: Scaling
  7. Stage 5: Mixture of Experts (MoE)
  8. Stage 6: New directions
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Summary of the evolution

Key terms in this lesson

Architecture
The blueprint of a neural network: its layers, their order and how information flows.
RNN
A network that reads tokens one by one and carries a hidden state forward.
Attention
A learned weighted lookup where each token mixes information from other tokens based on query-key similarity.
Transformer
An architecture built from self-attention and feed-forward layers with residual connections, without recurrence.
Scaling laws
Empirical rules showing loss falls predictably as model size, data and compute grow.
Mixture of Experts
A design where a router sends each token to a few of many expert sub-networks.
Decoder-only
A Transformer that uses a causal mask and predicts the next token; the design of most chat LLMs.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 5.4 Lost in the Middle: Why LLMs Miss Central Context · 6.2 Mixture of Experts: Routing Tokens to Specialists →