Modern AI Engineering

Lesson 6.2 · 27 min · video lesson

Mixture of Experts: Routing Tokens to Specialists

How can a model have 671 billion parameters but only use 37 billion of them for each word it writes?

In short: A Mixture of Experts (MoE) layer replaces one big feed-forward network with many smaller "experts" and a small router that picks a few experts for each token. Total parameters (knowledge capacity) grow while compute per token stays close to that of a much smaller dense model. The price is memory for all experts, load-balancing work during training, and more complex serving.

What you will learn

  • Why Mixture of Experts was needed
  • What an "expert" really means
  • The router and how it picks experts
  • Where MoE sits inside a Transformer
  • Sparse activation and why it saves compute
  • Load balancing across experts
  • Advantages and challenges of MoE
  • Why MoE powers many modern LLMs

In this lesson

  1. Why Mixture of Experts was needed
  2. What an "expert" really means
  3. The router and how it picks experts
  4. Where MoE sits inside a Transformer
  5. Sparse activation and why it saves compute
  6. Load balancing across experts
  7. Code: a tiny top-2 MoE layer in numpy
  8. Advantages and challenges of MoE
  9. Why MoE powers many modern LLMs
  10. Common mistakes and how to spot them
  11. Practice: try it yourself

Key terms in this lesson

Expert
One of several feed-forward sub-networks inside an MoE layer.
Router (gate)
A small layer that scores experts for each token and picks the top-k.
Top-k routing
Keeping only the k highest-scoring experts for a token and mixing their outputs.
Active parameters
The parameters actually used to process one token.
Load balancing
Techniques that spread tokens evenly across experts so none is overloaded or idle.
Capacity factor
A cap on how many tokens each expert accepts per batch; overflow tokens are dropped or rerouted.
Shared expert
An expert that every token uses, holding common knowledge alongside routed experts.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 6.1 A Timeline of LLM Architecture Improvements · 6.3 Grouped Query Attention: Fewer KV Heads, Same Quality →