Modern AI Engineering

Lesson 12.4 · 23 min · video lesson

Defining Done: Why Exit Criteria Shape Agent Quality

Why can an AI agent fix a tricky failing test in minutes, yet struggle to write a “good” product description that we are happy with?

In short: A definition of done is the clear line between a finished task and an unfinished one. When that line can be checked by a machine (tests pass, the number matches, the JSON validates), AI systems can try, check and retry on their own, and they look magical. When the line lives only in someone's head (“make it nicer”), the loop has nothing reliable to aim at. The practical skill is turning fuzzy goals into checkable ones.

What you will learn

  • What is a definition of done
  • Tasks where the definition of done is exact
  • Tasks where the definition of done is fuzzy
  • Why AI is so strong exactly where we can measure
  • How to write a better definition of done

In this lesson

  1. What is a definition of done?
  2. Tasks where the definition of done is exact
  3. Tasks where the definition of done is fuzzy
  4. Why AI is so strong exactly where we can measure
  5. Seeing it in code
  6. Vague, exact and rubric-based definitions
  7. How to write a better definition of done
  8. Worked example, step by step
  9. Practice: try it yourself
  10. Limits: when done cannot be defined

Key terms in this lesson

Definition of done
The explicit conditions a piece of work must meet to count as finished.
Verifier
A procedure, usually a program, that checks an output and returns pass, fail or a score without a human.
Rubric
A written list of specific criteria used by a human or LLM judge to score fuzzy outputs.
Verifiable reward
A training signal computed automatically by checking an answer, such as running tests or comparing to a known result.
Goodhart's law
When a measure becomes a target, it tends to stop being a good measure, because it gets optimised directly.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 12.3 Graph Engineering: Stateful Workflows for Agents · 12.5 LangChain: Composable Components for LLM Applications →