Modern AI Engineering

Lesson 6.3 · 25 min · video lesson

Grouped Query Attention: Fewer KV Heads, Same Quality

Why does a 70B model's memory for one long chat shrink from about 10 GB to about 1.25 GB just by letting attention heads share notes?

In short: During generation an LLM stores the keys and values of every past token (the KV cache), and with standard multi-head attention that cache becomes the main memory and speed bottleneck. Multi-Query Attention shares one key-value head across all query heads, which is fast but can hurt quality. Grouped-Query Attention sits in between: query heads are split into groups and each group shares one key-value head, keeping almost all the quality at a fraction of the memory.

What you will learn

  • The Big Picture
  • Quick Recap: Multi-Head Attention (MHA)
  • The Problem with Multi-Head Attention
  • What is Multi-Query Attention (MQA)?
  • What is Grouped-Query Attention (GQA)?
  • How Grouped-Query Attention Works
  • GQA is a Generalization of MHA and MQA
  • GQA vs MHA vs MQA
  • Real-World Use Cases
  • A Note on Terminology
  • Uptraining: Converting MHA to GQA
  • Quick Summary

In this lesson

  1. The big picture
  2. Quick recap: Multi-Head Attention (MHA)
  3. The problem with Multi-Head Attention
  4. What is Multi-Query Attention (MQA)?
  5. What is Grouped-Query Attention (GQA)?
  6. How Grouped-Query Attention works
  7. GQA is a generalization of MHA and MQA
  8. GQA vs MHA vs MQA
  9. Real-world use and a note on terminology
  10. Uptraining: converting MHA to GQA
  11. Going one level deeper
  12. Practice: try it yourself
  13. Quick summary

Key terms in this lesson

KV cache
Stored keys and values of past tokens, reused so each new token does not recompute them.
Multi-Head Attention (MHA)
Attention with several heads, each with its own query, key and value projections.
Multi-Query Attention (MQA)
All query heads share a single key head and value head.
Grouped-Query Attention (GQA)
Query heads are split into groups; each group shares one key head and one value head.
KV head
One key projection plus one value projection whose outputs are cached.
Uptraining
Converting an MHA checkpoint to GQA by pooling heads and training briefly.
Memory-bandwidth bound
Limited by how fast data can be read from memory rather than by arithmetic speed.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 6.2 Mixture of Experts: Routing Tokens to Specialists · 6.4 Sliding Window Attention: Taming Very Long Contexts →