How voice AI works: the pipeline

Voice is becoming a primary way people interact with AI — assistants, customer-service agents, and voice-first products. But building voice AI that feels natural is genuinely harder than text, and it starts with understanding the pipeline that turns speech into a response and back into speech.

The classic voice-AI pipeline — the cascade:

1. Speech-to-text (STT / ASR) — convert the user's spoken audio into text.
2. LLM — process that text and generate a response (in text).
3. Text-to-speech (TTS) — convert the response text back into spoken audio.

Three separately swappable stages: the user talks, STT transcribes, the LLM reasons and replies, TTS speaks the reply. This cascade is the foundational architecture (and there's an alternative — speech-native models — in the next lesson).

Why voice is harder than text:

• It's real-time and synchronous. Text chat tolerates a pause before a reply; voice conversation expects fast, natural turn-taking. Latency that's invisible in text is glaring in voice (next lesson).
• There's no screen to scan. In text, a user can skim a long answer; in voice, everything is linear and ephemeral — long responses fail, and the user can't re-read.
• Errors cascade. A single misheard word in STT feeds wrong text to the LLM, which confidently replies to the wrong input (the error stack, Module 3). Text doesn't have this transcription layer.
• Turn-taking is subtle. Knowing when the user has finished speaking (vs. just paused), handling interruptions, and not talking over the user are hard problems humans do effortlessly and machines struggle with (later this module).
• Audio is messy — accents, background noise, phone-line quality, and disfluencies all degrade recognition.

Why the pipeline framing matters: each stage (STT, LLM, TTS) is a component you choose and tune, each adds latency and potential error, and the seams between them are where you insert control (grounding, guardrails, logging — Module 3). Understanding voice AI as a pipeline lets you reason about where latency comes from, where errors enter, and where you can intervene — which is the foundation for building agents that feel natural rather than clunky.

The honest framing for this course: voice AI is powerful and increasingly important, but building it well means confronting real difficulties — latency, turn-taking, a cascading error stack — that the demos gloss over, plus genuine ethical and legal issues (voice cloning, disclosure law) that you can't ignore. So this course is practical and honest: understand the pipeline and its alternatives (Module 1), the components (Module 2), how to build reliable agents (Module 3), and the risks and law (Module 4). Get these right, and voice AI is a genuinely natural interface; get them wrong, and it's a frustrating one.

The mindset: voice AI works through a pipeline — speech-to-text, then an LLM, then text-to-speech (the cascade) — and it's harder than text because it's real-time and synchronous, has no screen to scan, cascades errors from a transcription layer, requires subtle turn-taking, and deals with messy audio. Understanding voice as a pipeline of swappable stages, each adding latency and error, with seams where you insert control, is the foundation for building agents that feel natural. Voice is a powerful interface, but building it well means confronting the real difficulties the demos hide.