Cascade vs. speech-native models

There are two fundamental architectures for voice AI: the cascade (STT → LLM → TTS) and speech-native / speech-to-speech models. Each has real trade-offs, and choosing between them is a key design decision. Understanding both — honestly — is essential.

The cascade (STT → LLM → TTS). The three-stage pipeline from the last lesson: transcribe, reason in text, synthesize speech. Its defining feature is that there's text in the middle, and the stages are separate and swappable.

Speech-native / speech-to-speech (S2S). A single multimodal model takes audio in and produces audio out, reasoning over the audio directly rather than routing through an intermediate text transcript as the primary interface. Examples include realtime voice models from major providers. Its defining feature is that there's no text bottleneck — the model works with audio end to end.

The real trade-offs (an honest comparison):

• Latency: speech-native is generally lower latency (one model, no stacked stages), while the cascade stacks STT + LLM + TTS latency. This is S2S's biggest advantage.
• Prosody and emotion: speech-native preserves tone, pacing, and emotion (it never flattens to text), while the cascade loses prosody at the text bottleneck and must re-synthesize it. S2S sounds more natural and expressive.
• Control and swappability: the cascade lets you hot-swap each component and insert logic between stages — RAG, PII redaction, guardrails, tool calls at the text seam. Speech-native is more of a black box locked to one provider, with no intermediate seam.
• Debuggability and compliance: the cascade gives you inspectable text at each stage — auditable logs, easy moderation. Speech-native is more opaque and harder to audit, which matters in regulated domains.
• Tool/function calling: more mature in the cascade (it inherits the LLM's tooling); newer and rougher in speech-native (though closing fast).
• Cost: the cascade is often more predictable per-minute; speech-native cost can grow as audio context accumulates over a conversation.

An important nuance: even "speech-to-speech" providers usually still produce a text transcript alongside the audio (for logging, tool calls, moderation). So the clean "no text at all" framing is an oversimplification — S2S reduces the text bottleneck's role but rarely eliminates transcripts entirely.

How to choose:

• Choose speech-native when low latency and natural, expressive speech are paramount and you can accept less control and harder auditing (e.g., a consumer voice assistant where feel is everything).
• Choose the cascade when you need control, debuggability, guardrails, mature tool-calling, and auditability — especially in regulated or complex domains where you must insert logic and inspect what's happening (the text seam is a real asset here).
• The gap is narrowing — speech-native tool-calling and control are improving fast, so re-evaluate as the technology moves.

The mindset: the two voice-AI architectures trade off cleanly. The cascade (STT → LLM → TTS) has text in the middle — giving you control, swappable components, seams to insert guardrails/RAG/tools, mature tool-calling, and auditability — at the cost of stacked latency and lost prosody. Speech-native/speech-to-speech models work audio-to-audio — lower latency and preserved emotion/prosody — but are more opaque, provider-locked, and harder to audit, with newer tool-calling. Choose speech-native when latency and natural feel dominate; choose the cascade when you need control, guardrails, and debuggability (especially in regulated domains). Note S2S usually still emits transcripts, and the gap is narrowing fast.