Lesson 4.11 · 23 min · video lesson

Causal Masking: Preventing the Model from Seeing the Future

If a model is trained to predict the next word, but its attention can see the whole sentence, what stops it from simply peeking at the answer?

In short: A causal mask blocks every token from attending to tokens that come after it. We set the scores above the diagonal of the attention matrix to −∞ before softmax, so those weights become exactly 0. This lets a decoder-only LLM train on every position of a sentence in parallel without cheating, and makes training match how text is generated at inference time: one token at a time, left to right.

What you will learn

• Without Causal Masking
• With Causal Masking
• Implementation of Causal Masking
• The Causal Mask Matrix

In this lesson

1. The problem: a next-word predictor that can see the future
2. Without causal masking
3. With causal masking
4. The causal mask matrix
5. Implementation of causal masking
6. Causal vs bidirectional attention
7. Real-world use, pitfalls and limits
8. Going one level deeper
9. Practice: try it yourself

Key terms in this lesson

Causal mask

A triangular mask that blocks attention from any token to tokens after it.

Decoder-only model

A Transformer that generates text left to right using causally masked self-attention.

Bidirectional attention

Attention where every token can see tokens on both sides.

Lower-triangular matrix

A matrix whose entries above the diagonal are zero (or, here, blocked).

Padding mask

A mask that hides filler tokens added to make sequences in a batch the same length.

Prefill

The inference step where the whole prompt is processed in parallel before generating new tokens.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.10 Scaled Dot-Product Attention: Why We Divide by √dₖ · 4.12 Multi-Head Attention: Many Perspectives at Once →