Modern AI Engineering

Lesson 14.4 · 24 min · video lesson

Agent Observability: Traces, Spans, and Debug Signals

A customer says our agent took 30 seconds and then refunded the wrong order: how do we find out exactly what it did, step by step?

In short: AI agent observability is the ability to see and understand what an agent did on every run: each model call, prompt, tool call, result, decision, token count, latency and cost. It is built from traces made of nested spans, plus metrics and logs. Observability lets us debug failures, control cost and latency, detect quality drift and feed real failures back into evaluation.

What you will learn

  • What is an AI Agent?
  • What is Observability?
  • What is AI Agent Observability?
  • Why do we need AI Agent Observability?
  • How is AI Agent Observability different from traditional Observability?
  • The Three Pillars of Observability
  • Traces and Spans
  • What should we observe inside an AI Agent?
  • Key Metrics for AI Agent Observability
  • How AI Agent Observability works
  • Tools and Frameworks for AI Agent Observability
  • Observability vs Evaluation
  • Challenges in AI Agent Observability
  • Best Practices

In this lesson

  1. What is an AI agent, and what is observability?
  2. What is AI agent observability, and why do we need it?
  3. How is AI agent observability different from traditional observability?
  4. The three pillars of observability
  5. Traces and spans
  6. What to observe inside an agent, and the key metrics
  7. How AI agent observability works
  8. Tools and frameworks, and observability vs evaluation
  9. Challenges and best practices
  10. Common mistakes and how to spot them
  11. Practice: try it yourself

Key terms in this lesson

Observability
The ability to understand a system's internal behaviour from the telemetry it emits.
Telemetry
Data a system emits about itself: logs, metrics and traces.
Trace
The end-to-end record of one request, made of linked spans and identified by a trace ID.
Span
One timed unit of work in a trace, with a parent, status and attributes.
Attribute
A key-value detail on a span, such as model name, token count or tool argument.
OpenTelemetry
An open standard and toolkit for producing and exporting traces, metrics and logs.
p95 latency
The latency below which 95% of requests complete; it reveals the slow tail.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 14.3 Evaluating AI Agents: Metrics and Methods That Work · 15.1 LLM Guardrails: Filtering Inputs and Outputs for Safety →