Modern AI Engineering

Lesson 14.1 · 25 min · video lesson

Evaluating LLMs: Metrics, Benchmarks, and Methods

How do we know a new prompt or a new model is actually better, and not just better on the three examples we happened to try?

In short: LLM evaluation is the practice of measuring how well a language model or LLM application performs, using test sets, metrics, benchmarks, human reviewers and other models as judges. Because LLM outputs are open-ended and non-deterministic, no single number is enough. We combine cheap automatic checks, task-specific tests, human judgment and safety testing, and we treat our own evaluation set as a core asset.

What you will learn

  • What is LLM Evaluation?
  • Why do we need LLM Evaluation?
  • Types of LLM Evaluation
  • Automatic Metrics
  • Benchmarks
  • Human Evaluation
  • LLM as a Judge
  • Task-Specific Evaluation
  • Safety and Red-Teaming Evaluation
  • Challenges in LLM Evaluation
  • Best Practices
  • When to use which method

In this lesson

  1. What is LLM evaluation, and why do we need it?
  2. Types of LLM evaluation
  3. Automatic metrics
  4. Benchmarks
  5. Human evaluation and LLM as a judge
  6. Task-specific evaluation
  7. Safety and red-teaming evaluation
  8. Code you can run: metrics and a confidence interval
  9. Challenges, best practices and when to use which method
  10. Going one level deeper
  11. Practice: try it yourself

Key terms in this lesson

LLM evaluation
Systematically measuring how well an LLM or LLM application performs on defined criteria.
Exact match (EM)
A metric that scores 1 only if the normalized output equals the reference.
Token F1
Harmonic mean of word-level precision and recall between output and reference.
Benchmark
A public standardized test set with fixed scoring used to compare models.
Golden test set
A curated set of inputs with expected answers or criteria for our own task.
Data contamination
When test questions appear in a model's training data, inflating its scores.
Red-teaming
Deliberately attacking a system to find unsafe or unwanted behaviour.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.17 TensorRT-LLM: NVIDIA's Optimized Inference Engine · 14.2 LLM-as-Judge: Automating Evaluation with Another Model →