Modern AI Engineering

Lesson 3.4 · 25 min · video lesson

Cross-Entropy Loss: Scoring Probability Predictions

When a model says "70% cat" and the picture is a dog, how do we turn that into one number that tells training exactly how wrong it was, and in which direction to fix it?

In short: Cross-entropy loss measures how much probability a model gave to the correct answer: the loss is −log(p_correct). Confident right answers cost almost nothing, and confident wrong answers cost a lot. It is the standard loss for classification and for training language models, and its gradient with respect to the logits is beautifully simple: p − y.

What you will learn

  • The Big Picture
  • What is Cross-Entropy
  • The Cross-Entropy Loss Formula
  • Why We Take the Negative Log
  • Binary Cross-Entropy Loss
  • Categorical Cross-Entropy Loss
  • Step-by-Step Numeric Example
  • Cross-Entropy Loss for Language Models
  • The Gradient of Cross-Entropy Loss
  • Quick Summary

In this lesson

  1. The big picture
  2. What is cross-entropy?
  3. The cross-entropy loss formula, and why the negative log
  4. Binary cross-entropy loss
  5. Categorical cross-entropy loss
  6. Step-by-step numeric example
  7. Cross-entropy loss for language models
  8. The gradient of cross-entropy loss
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Comparison, pitfalls and quick summary

Key terms in this lesson

Cross-entropy
The average surprise, −∑ p(x) log q(x), of a true distribution p measured with a predicted distribution q.
Logits
The raw, unnormalised scores a network outputs before softmax or sigmoid.
Softmax
A function that turns logits into probabilities by exponentiating and dividing by their sum.
One-hot vector
A label vector with 1 for the correct class and 0 for every other class.
KL divergence
A non-negative measure of how much one distribution differs from another; cross-entropy minus entropy.
Perplexity
exp of the average per-token cross-entropy; roughly how many tokens the model is "choosing between".

This lesson is free. Sign in or create a free account to open it with its video and quiz.

← 3.3 Backpropagation: How Neural Networks Learn from Mistakes · 3.5 Dropout: Controlled Forgetting as Regularization →