Modern AI Engineering

Lesson 17.7 · 24 min · video lesson

LLM Routing: Directing Each Query to the Best Model

Why pay a top-tier model to answer "How do I reset my password?" a million times a day?

In short: LLM routing puts a small decision layer, the router, in front of several language models and sends each query to the cheapest model that can answer it well. Routers can use rules, a trained classifier, embeddings, or a cascade that tries a small model first and escalates when its answer looks weak. Done well, routing cuts cost and latency sharply while keeping quality close to always using the biggest model.

What you will learn

  • The Big Picture
  • What is LLM Routing
  • Why we need LLM Routing
  • Anatomy of an LLM Router
  • Routing Strategies
  • A Full Trace Example
  • LLM Routing vs Mixture of Experts
  • When LLM Routing is Worth It
  • Common Mistakes and How to Fix Them
  • Quick Summary

In this lesson

  1. The big picture: one size does not fit all queries
  2. What LLM routing is and why we need it
  3. Anatomy of an LLM router
  4. Routing strategies
  5. A full trace example
  6. LLM routing vs Mixture of Experts
  7. When LLM routing is worth it
  8. Common mistakes and how to fix them
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

LLM router
A component that chooses which language model handles each request.
Model pool
The set of candidate models a router can choose from.
Cascade
Trying a cheaper model first and escalating to a stronger one only if the answer fails a check.
Semantic routing
Choosing a route by comparing the query's embedding with example queries for each route.
Routing threshold
The score above which a request is sent to the stronger, more expensive model.
Mixture of Experts (MoE)
A model architecture where a gate sends each token to a few expert sub-networks inside the model.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 17.6 On-Device ML: A TensorFlow Lite Android Walkthrough · 17.8 Building a Real-Time Voice AI Agent from Scratch →