Modern AI Engineering

Lesson 4.7 · 25 min · video lesson

Encoder vs Decoder: Two Sides of the Transformer

BERT and GPT are both Transformers, yet one is great at understanding and the other at writing. What single design choice makes the difference?

In short: An encoder reads the whole input at once, letting every token look both left and right, and outputs a context-rich vector per token: ideal for understanding. A decoder generates text one token at a time and uses a causal mask so each token sees only earlier tokens: ideal for writing. Transformers come in three types, encoder-only, decoder-only and encoder–decoder, and we choose among them based on whether the task is understanding, open-ended generation or mapping an input to an output.

What you will learn

  • What is a Transformer?
  • A word about tokens
  • What is an Encoder?
  • What is a Decoder?
  • The one big difference
  • The three types of Transformers
  • Let's tabulate the difference
  • When to use which one?
  • Summary

In this lesson

  1. What is a Transformer?
  2. A word about tokens
  3. What is an Encoder?
  4. What is a Decoder?
  5. The one big difference
  6. The three types of Transformers
  7. Let's tabulate the difference
  8. When to use which one?
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Summary

Key terms in this lesson

Encoder
A Transformer stack that reads the whole input bidirectionally and outputs a contextual vector per token.
Decoder
A Transformer stack that generates tokens one at a time using causal self-attention.
Bidirectional attention
Attention where each token can attend to tokens on both its left and right.
Causal mask
A mask that blocks attention to future tokens by setting their scores to −∞ before softmax.
Cross-attention
Attention where decoder queries look at encoder keys and values.
Masked language modelling
Pre-training by hiding some tokens and predicting them from the surrounding context.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.6 The Transformer Architecture: Built on Attention · 4.8 Self-Attention: How Tokens See One Another →