Modern AI Engineering

Lesson 5.2 · 23 min · video lesson

Nucleus Sampling: Top-k and Top-p Demystified

Always picking the most likely word makes a chatbot dull and repetitive, but picking from all 100,000 words lets in nonsense. How do we keep the good choices and cut the junk?

In short: Top-k and top-p are filters applied to the next-token probabilities before sampling. Top-k keeps a fixed number of the most likely tokens; top-p (nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to at least p. Both throw away the long tail of unlikely tokens and renormalize the rest. Top-p adapts to how confident the model is, which is why it is the more common default; both are usually combined with temperature.

What you will learn

  • How an LLM picks the next token
  • The problem with always picking the best token
  • The problem with picking from every token
  • What is Top-k Sampling?
  • Step-by-step example of Top-k Sampling
  • The problem with Top-k Sampling
  • What is Top-p Sampling?
  • Step-by-step example of Top-p Sampling
  • Top-k vs Top-p Sampling
  • How Top-k and Top-p work with Temperature
  • When to use which one

In this lesson

  1. How an LLM picks the next token
  2. The problem with always picking the best token
  3. The problem with picking from every token
  4. What is top-k sampling? A step-by-step example
  5. The problem with top-k sampling
  6. What is top-p sampling? A step-by-step example
  7. Top-k vs top-p sampling
  8. How top-k and top-p work with temperature
  9. When to use which one, and common mistakes
  10. Worked example, step by step
  11. Practice: try it yourself

Key terms in this lesson

Greedy decoding
Always choosing the single most likely next token.
Top-k sampling
Sampling only from the k most likely tokens after renormalizing.
Top-p (nucleus) sampling
Sampling from the smallest set of top tokens whose probabilities sum to at least p.
Long tail
The many low-probability tokens that together can hold noticeable probability mass.
Renormalize
Rescale the kept probabilities so they sum to 1 again.
Cumulative probability
The running total of probabilities when tokens are sorted from most to least likely.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 5.1 Temperature Sampling: Dialing Up or Down Creativity · 5.3 Token Streaming: Rendering Outputs as They Arrive →