Modern AI Engineering

Lesson 5.1 · 23 min · video lesson

Temperature Sampling: Dialing Up or Down Creativity

Ask the same chatbot the same question twice and you may get two different answers. One small number, the temperature, decides how much that happens.

In short: An LLM produces a score (logit) for every possible next token; softmax turns those scores into probabilities and one token is sampled. Temperature divides the logits before softmax: below 1 it sharpens the distribution toward the top token (focused, repeatable output), above 1 it flattens it (varied, riskier output). Temperature 0 is treated as always picking the top token. Choosing it well depends on the task.

What you will learn

  • What is Temperature in LLMs?
  • How does an LLM pick the next token?
  • From scores to probabilities
  • Where does Temperature come into the picture?
  • Step-by-step example with numbers
  • Low Temperature
  • High Temperature
  • Temperature = 1 and Temperature = 0
  • Why is it called Temperature?
  • When to use which Temperature?
  • Common mistakes while using Temperature

In this lesson

  1. What is temperature in LLMs?
  2. How does an LLM pick the next token?
  3. From scores to probabilities
  4. Where does temperature come into the picture?
  5. Step-by-step example with numbers
  6. Low temperature vs high temperature
  7. Temperature = 1 and temperature = 0
  8. Why is it called temperature?
  9. When to use which temperature, and common mistakes
  10. Going one level deeper
  11. Practice: try it yourself

Key terms in this lesson

Logit
The raw, unnormalized score the model gives each vocabulary token.
Softmax
The function that turns logits into probabilities that are positive and sum to 1.
Temperature
A number that divides the logits before softmax to control randomness.
Greedy decoding
Always choosing the single most likely next token; what T = 0 means.
Sampling
Choosing the next token at random according to its probability.
Boltzmann distribution
The physics distribution e^(−E/kT) that softmax with temperature mirrors.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.15 Feed-Forward Networks: The Transformer's Memory Layer · 5.2 Nucleus Sampling: Top-k and Top-p Demystified →