Modern AI Engineering

Lesson 17.4 · 28 min · video lesson

Language Processing Units: A New Approach to LLM Inference

If a GPU can do a thousand trillion operations per second, why does a chatbot running on it still type at a few dozen words per second — and how did one chip design make that dramatically faster?

In short: An LPU (Language Processing Unit) is Groq's chip design for fast LLM inference. Token-by-token generation is limited by how fast weights can be read from memory, not by arithmetic, so the LPU keeps model weights in fast on-chip SRAM spread across many chips, and has a compiler plan every operation and every data transfer in advance so nothing ever waits or guesses. The result is very low latency per user, paid for with many chips per model and less flexibility than a GPU.

What you will learn

  • What is an LPU?
  • How an LLM writes text, one token at a time
  • The real bottleneck is memory, not math
  • Why a GPU struggles here
  • Idea 1: Keep the model on the chip
  • The problem with on-chip memory
  • Idea 2: Remove all the guesswork
  • Idea 3: A network that never waits
  • The assembly line
  • What happens when we send a prompt
  • Why an LPU is fast, all in one place
  • Where an LPU works well
  • Where an LPU does not work well
  • LPU vs GPU
  • When to use which one

In this lesson

  1. What is an LPU?
  2. How an LLM writes text, one token at a time
  3. The real bottleneck is memory, not math
  4. Why a GPU struggles here
  5. Idea 1: keep the model on the chip
  6. The problem with on-chip memory
  7. Idea 2: remove all the guesswork
  8. Idea 3: a network that never waits
  9. The assembly line
  10. What happens when we send a prompt
  11. Why an LPU is fast, all in one place
  12. Where an LPU works well, and where it does not
  13. LPU vs GPU, and when to use which
  14. Worked example, step by step
  15. Practice: try it yourself

Key terms in this lesson

LPU
Language Processing Unit: Groq's inference chip design for fast LLM token generation.
Decode phase
The token-by-token part of LLM generation, where each step depends on the previous token.
SRAM
Fast static memory built on the processor chip; high bandwidth but small capacity.
HBM
High Bandwidth Memory: stacked DRAM next to a GPU or TPU, large but slower than on-chip SRAM.
Deterministic execution
Running a program whose timing is fully planned in advance, so it takes the same cycles every time.
Static scheduling
A compiler deciding ahead of time which unit does what on which cycle, instead of hardware deciding at run time.
Tensor Streaming Processor
Groq's architecture in which data streams across slices of specialised functional units.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 17.3 Google TPUs: Purpose-Built Hardware for Neural Networks · 17.5 Cloud vs Edge: Where Should Your Model Run? →