Latency: the honest version

Latency is the make-or-break factor in voice AI — but it's also the most overstated thing in the field. Getting an honest picture of what's realistic (vs. the marketing) is essential, because building to a fantasy latency target leads to disappointment.

The human baseline — where the '200ms' comes from. In natural human conversation, the gap between turns is remarkably short. Peer-reviewed research (Stivers, Levinson, and colleagues, studying conversation across many languages) found that responses to simple questions peak at around 200 milliseconds after the speaker finishes, with typical gaps in the low hundreds of milliseconds. This is the scientific basis for "sub-second feels natural." But — critically — that 200ms is the human turn-transition gap, not a claim that any deployed voice agent responds that fast. Confusing the two is the field's most common overstatement.

What's realistic for deployed voice agents. Real voice agents don't hit 200ms end-to-end. A common engineering target is roughly ~800 milliseconds (P50) from when the user stops speaking to first audio back — under ~500ms feels near-human, ~800ms is usable but noticeable, and beyond ~1200ms feels sluggish. Even the fastest speech-native setups typically land in the few-hundred-milliseconds range in good conditions, and cascades commonly land higher. So the honest framing is: aim for a few hundred milliseconds to ~800ms first-audio, not "human speed."

Where the latency comes from (the budget). In a cascade, latency stacks across components:

• Endpointing / turn detection — waiting to be sure the user finished speaking (this alone is often ~150-200ms+).
• STT — transcribing (first partial results can be fast, but there's latency).
• LLM time-to-first-token — the model starting to generate.
• TTS time-to-first-audio — the synthesizer starting to speak.
• Network and audio capture/playback — especially over telephony.

These add up, which is why cascades are slower than speech-native (one model) — and why every millisecond in each stage matters.

The critical caveat — vendor latency ≠ your latency. TTS and STT vendors advertise impressively low model-only latencies (e.g., tens of milliseconds time-to-first-audio). But real-world latency including network is meaningfully higher (often a few times higher). When you build a latency budget, use measured, real-world numbers, not the marketing model-only figures — or you'll design to a target you can't hit.

Telephony makes it worse. Real phone calls run over narrowband (8kHz) audio with PSTN/network overhead, adding latency and degrading recognition accuracy. A voice agent that hits ~800ms in a browser demo may be noticeably slower on an actual phone call.

The mindset: be honest about voice latency. The famous ~200ms is the human turn-taking gap (from conversation research), not what deployed agents achieve — claiming voice agents respond "as fast as humans" is the field's most common overstatement. Realistically, aim for a few hundred milliseconds to ~800ms first-audio: latency stacks across endpointing, STT, LLM time-to-first-token, TTS time-to-first-audio, and network — which is why cascades are slower than speech-native. Crucially, vendor model-only latency numbers understate real-world latency (which includes network), and telephony adds more. Build your latency budget from measured numbers, target realistic figures, and don't design to a fantasy.