Modern AI Engineering

Lesson 6.6 · 29 min · video lesson

Flash Attention: Memory-Efficient Attention at Scale

How can an algorithm that does the exact same math as standard attention, and even some extra math, run several times faster?

In short: Standard attention writes a huge n × n score matrix to slow GPU memory and reads it back, so it spends most of its time moving data rather than computing. Flash Attention computes exactly the same result in small tiles that stay in fast on-chip memory, using an online softmax so the full matrix never exists, and recomputes it during the backward pass instead of storing it. Flash Attention 2 and 3 improve how the work is split across the GPU and exploit newer hardware features.

What you will learn

  • A quick recap of standard attention
  • Why standard attention is slow
  • How GPU memory actually works (HBM vs SRAM)
  • The core idea behind Flash Attention
  • Tiling: breaking the work into small blocks
  • Online softmax: computing softmax without the full matrix
  • Recomputation in the backward pass
  • Flash Attention 2
  • Flash Attention 3
  • Advantages and impact of Flash Attention

In this lesson

  1. A quick recap of standard attention
  2. Why standard attention is slow
  3. How GPU memory actually works (HBM vs SRAM)
  4. The core idea behind Flash Attention
  5. Tiling: breaking the work into small blocks
  6. Online softmax: computing softmax without the full matrix
  7. Recomputation in the backward pass
  8. Flash Attention 2
  9. Flash Attention 3
  10. Advantages and impact of Flash Attention
  11. Worked example, step by step
  12. Practice: try it yourself

Key terms in this lesson

HBM
High Bandwidth Memory: the GPU's large but comparatively slow main memory.
SRAM
Small, very fast on-chip memory inside each GPU streaming multiprocessor.
IO-aware
Designed to minimise data movement between memory levels, not just arithmetic.
Kernel fusion
Combining several GPU operations into one kernel so intermediates never leave fast memory.
Tiling
Splitting matrices into small blocks that fit in fast memory and processing them one by one.
Online softmax
Computing softmax incrementally with a running max and sum, rescaling earlier partial results.
Recomputation
Recomputing intermediates in the backward pass instead of storing them.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 6.5 Attention Sinks: The Hidden Cost of Extended Context · 6.7 DeepSeek-V4: Anatomy of an Open-Source Frontier Model →