Modern AI Engineering

Lesson 17.8 · 33 min · video lesson

Building a Real-Time Voice AI Agent from Scratch

Humans reply to each other within a fraction of a second; how do we build an AI that listens, thinks, calls tools and talks back that fast, and lets you interrupt it?

In short: A real-time voice agent streams audio in, detects when the user has finished speaking, understands the request, decides what to say (often calling tools), and streams synthesised speech back, all within roughly a second. We can build it as a cascaded pipeline (speech-to-text, LLM, text-to-speech), as a single speech-to-speech model, or as a hybrid. The hard parts are the latency budget, turn detection, interruptions (barge-in), telephony, scaling and safety, which is exactly what a system design interview probes.

What you will learn

  • What is a Voice AI Agent?
  • Why is Real-Time Voice hard?
  • Requirements
  • Back-of-the-envelope estimation
  • High-Level Architecture
  • Component 1: Audio Transport
  • Component 2: Voice Activity Detection and Turn Detection
  • Component 3: Speech-to-Text (STT)
  • Component 4: The Brain - LLM with Tools
  • Component 5: Text-to-Speech (TTS)
  • Approach 1: Cascaded Pipeline (STT -> LLM -> TTS)
  • Approach 2: Speech-to-Speech Model
  • Approach 3: Hybrid Approach
  • Cascaded vs Speech-to-Speech: Comparison
  • Latency Budget: Where every millisecond goes
  • Handling Interruptions (Barge-in)
  • Tool Calling in a Voice Agent
  • Memory and Context
  • Telephony: Connecting to real phone calls
  • Scaling the system
  • Edge Cases and how to handle them
  • Observability and Evaluation
  • Safety, Security, and Privacy
  • Cost
  • How to present this design in an interview

In this lesson

  1. What a voice AI agent is, and why real-time voice is hard
  2. Requirements and back-of-the-envelope estimation
  3. High-level architecture and the five components
  4. Code: voice activity detection and end-of-turn
  5. Three approaches: cascaded, speech-to-speech, hybrid
  6. The latency budget: where every millisecond goes
  7. Barge-in, tool calling, and memory
  8. Telephony and scaling the system
  9. Edge cases, observability, safety and cost
  10. Going one level deeper
  11. Practice: try it yourself
  12. How to present this design in an interview

Key terms in this lesson

Voice Activity Detection (VAD)
Classifying short audio frames as speech or non-speech.
Endpointing / turn detection
Deciding that the speaker has finished their turn so the agent can respond.
Barge-in
The user interrupting while the agent is speaking; the agent should stop and listen.
Cascaded pipeline
A voice agent built from separate STT, LLM and TTS models connected by text.
Speech-to-speech model
A single model that takes audio input and produces audio output directly.
Voice-to-voice latency
Time from the user finishing speaking to the agent's audio starting.
SIP trunk
A connection that carries phone calls between the telephone network and internet systems using the Session Initiation Protocol.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 17.7 LLM Routing: Directing Each Query to the Best Model · 17.9 System Design: HLD, LLD and UML Diagrams with Interview Questions →