# Voice Agents 101: The Architecture Behind AI That Talks Back
**作者**: Manthan Gupta
**日期**: 2026-05-06T14:30:45.000Z
**来源**: [https://x.com/manthanguptaa/status/2052033331546698209](https://x.com/manthanguptaa/status/2052033331546698209)
---
The first time I built a voice agent and got on a call with it, the thing felt like talking to someone over a satellite phone in 2003. Long pauses, unnatural cadence, and occasional cuts where it would just keep talking over me. The text version of the same product was usable. The voice version was not. That was the moment I learned that text engineering and voice engineering are not the same craft.
The moment you introduce audio, you are dealing with a fundamentally different latency profile, a different failure mode surface, and a different set of tradeoffs than anything you would encounter in a text-based system. Voice agents aren’t just LLMs with a speaker attached. They are pipelines, and every stage of that pipeline fights against the same constraint: time.
This post covers how the voice agent pipeline actually works, the architectural choice between cascade and end-to-end speech models, the latency budget you are working within, what full-duplex means and why it is hard, and the design decisions that determine whether your voice agent feels natural or robotic.
## The Three-Stage Pipeline
At its core, the most common voice agent architecture is built on three sequential stages.
Speech-to-Text (STT) takes the raw audio from the microphone and converts it into text that the LLM can process. The dominant options here are OpenAI Whisper (open-source, extremely accurate, but has latency), Deepgram (low-latency streaming transcription, production-proven), AssemblyAI (good accuracy, streaming support), and a handful of others. The key metric is not just word error rate but time to transcript — how long it takes to produce a transcript. A 98% accurate transcription that takes 800ms to produce is often worse than a 95% accurate one that takes 150ms.
LLM is the brain. It takes the transcript, applies context and memory, and generates a response. The same models you would use in text apply here, but the prompt structure changes: voice conversations are shorter-turn, more casual, and the model absolutely cannot produce markdown, numbered lists, or long walls of text. Every response needs to be spoken aloud naturally.
Text-to-Speech (TTS) converts the LLM’s text response to audio. The options range from older, robotic systems to modern neural TTS that sounds remarkably human: ElevenLabs, Cartesia (exceptionally low-latency), OpenAI TTS, PlayHT, and others. The metric that matters most in production is time to first audio chunk — how long before the user hears anything at all.
Stitching these three stages together is straightforward but making them feel seamless is not.
## Cascade vs. End-to-End Speech Models
Before we get deeper into the cascade pipeline, you should know that there is now a second architectural family that didn’t really exist 18 months ago: end-to-end speech models.
Cascade is what we just described. STT transcribes audio to text, an LLM produces text, TTS turns that text back into audio. You get full control of every stage, complete observability, and the ability to swap components independently. The price you pay is latency (each stage adds time and you can’t always overlap them perfectly), context loss (the LLM never hears tone of voice or hesitation, only the transcribed words), and a flatter emotional surface (the LLM doesn’t know the user is frustrated, and the TTS doesn’t naturally react to the content).
End-to-end speech models bypass the transcription step entirely. The model takes audio in and produces audio out. OpenAI’s Realtime API (built on GPT-4o), Kyutai’s Moshi, and Sesame’s CSM are the prominent examples. Because there’s no STT step and no separate TTS step, latency drops dramatically. Moshi reports response latency in the 200-300ms range which is closer to human conversation than any cascade system can reach. Prosody and emotion are also better because the model directly hears and produces audio without flattening it through text.
The tradeoff is that you give up most of the things that make cascade systems engineerable. You can’t easily inspect what the model “heard.” You can’t swap a better STT in next quarter. Tool calling and structured output are weaker because they have to be encoded into the audio modality rather than handled as text. And the cost per minute of conversation is currently much higher than running a small fast LLM in a cascade.
For most production voice agents in 2026 customer service, telephony, voice front-ends to existing systems cascade is still the right architecture. The control surface, observability, and maturity of the component ecosystem outweigh the latency advantage of end-to-end models. But for use cases where natural conversation matters more than precise tool use (companions, language tutors, hands-free assistants), end-to-end is becoming genuinely competitive. It’s worth knowing both architectures exist before you commit to one.
The rest of this post focuses on cascade because that’s where the engineering surface is most interesting.
## The Latency Budget
For a voice conversation to feel natural, the total round-trip from the user finishing a sentence to hearing the first word of a response needs to be under roughly 500-800ms. Beyond that, users start perceiving a delay. Beyond 1.5 seconds, it feels broken. So, you are usually fighting against the clock to get your voice agent to feel natural.
Let’s break down where the time goes in a naive sequential implementation of a voice agent:
```
| Stage | Typical latency |
|-------|-----------------|
| Audio capture + VAD end-detection | 100–300ms |
| STT transcription | 150–400ms |
| LLM Time to First Token (TTFT) | 300–800ms |
| TTS first audio chunk | 100–300ms |
| Network overhead | 50–150ms |
| **Total** | **700ms–1.95s** |
```
Even at the optimistic end, you are already at the edge of acceptable. At the pessimistic end, it feels like talking to someone with a bad phone connection. This is why voice agents are fundamentally a latency engineering problem, not just an AI problem. The best voice experiences are built by teams that treat milliseconds as a first-class engineering concern.
The solution isn’t to make each stage faster in isolation (though that helps). The real unblock is streaming so that the stages can overlap rather than run sequentially.
## Streaming: The Only Way to Win on Latency
In a naive pipeline, you wait for the user to finish speaking, wait for the full transcript, send it to the LLM, wait for the full response, then send it all to TTS. That’s fully sequential and the latency compounds.
In a streaming pipeline, you start TTS as soon as the LLM starts generating tokens and you don’t wait for the full response. The LLM streams token by token, and the TTS system consumes those tokens and begins synthesizing audio in real time, typically buffering 5-15 tokens before generating the first chunk of audio to avoid stopping and starting.
The result is that the user hears the first word of the response while the LLM is still generating the second sentence. This is how production voice systems at companies like ElevenLabs’ Conversational AI, Bland.ai, and Retell AI achieve sub-600ms end-to-end response latency despite using powerful LLMs under the hood.
## Half-Duplex vs Full-Duplex
This is one of the most important architectural decisions in voice agent design, and it’s underappreciated.
Half-duplex is the push-to-talk model: one party speaks, then the other speaks. The system listens while the user talks, then responds, then listens again. Simple to implement. Feels like leaving voicemails back and forth. For simple command-response interfaces (think: “set a timer for 5 minutes”), half-duplex is fine. For natural conversation, it’s frustrating.
Full-duplex is how human conversations actually work: both parties can speak simultaneously, interrupt each other, say “uh-huh” mid-sentence, and react in real time. Building this is substantially harder. It requires the system to be listening and generating audio at the same time which means you need to solve the barge-in problem.
Barge-in is what happens when the user starts speaking while the agent is still talking. In a well-designed full-duplex system, the agent detects that the user has started speaking (via VAD), immediately stops its own audio output, cancels any pending TTS chunks, and begins processing the new input. In a poorly designed system, the agent just keeps talking while the user is trying to interrupt, which is deeply annoying.
The technical challenge of barge-in is threefold. First, you need a Voice Activity Detection (VAD) model that can distinguish the user’s voice from the agent’s own audio playback through the speaker, otherwise the agent hears itself and thinks it’s being interrupted. Second, you need to handle the latency gracefully: the moment VAD fires, you need to cut audio within milliseconds, not 300ms later. Third, you need to decide what to do with the partial audio captured during the barge-in. The question is: is it a real input or accidental noise?
The current state of the art uses Silero VAD, a lightweight neural VAD model that runs locally with very low latency (~10ms per 32ms audio chunk). Combined with echo cancellation (filtering out the agent’s own audio from the mic), it gives you a reasonable barge-in detection system.
## Turn-Taking Management
Humans don’t just talk and listen in strict alternation. We use backchannels (“mm-hmm”, “yeah”), we start talking slightly before the other person has fully finished, and we use prosodic cues (pitch, rhythm, trailing off) to signal turn-yielding. Current voice agent systems handle this clumsily.
The naive approach is energy-based end-of-speech detection: wait for N milliseconds of silence before deciding the user has finished speaking. The problem is that 500ms of silence is natural in thoughtful speech it doesn’t mean the user is done. Wait too long and the agent feels unresponsive. Wait too short and it cuts off the user.
Better approaches combine silence detection with prosodic analysis (does the pitch fall in a pattern consistent with sentence completion?) and semantic analysis (does the transcript so far form a complete thought?). Some systems use a small, fast LLM to predict whether the user’s utterance is likely complete given the transcript so far, which works surprisingly well.
This is an active research area and the LiveKit Agents framework supports pluggable turn detection strategies including model-based approaches.
## The Full Architecture
A production voice agent pipeline typically looks like this:
The conversation state (history, memory, context) lives in the LLM prompt, which gets reconstructed for each turn. The audio transport layer (how audio bytes travel between the user’s device and your server) is typically handled by WebRTC for browser-based agents or SIP/RTP for telephony-based ones.
## A Minimum Viable Voice Agent
To make this concrete, here is roughly what wiring up a voice agent looks like in practice. This is a simplified Pipecat-style pipeline showing how the pieces snap together and production code adds error handling, observability, and a lot more configuration, but the architecture is the same.
```
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.transports.services.daily import DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
async def main():
# WebRTC transport with built-in VAD and echo cancellation
transport = DailyTransport(
room_url, token, "Voice Bot",
DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=False,
),
)
# Three pipeline stages
stt = DeepgramSTTService(api_key=DEEPGRAM_KEY)
llm = OpenAILLMService(api_key=OPENAI_KEY, model="gpt-4o-mini")
tts = CartesiaTTSService(api_key=CARTESIA_KEY, voice_id=VOICE_ID)
# Voice-specific system prompt (more on this below)
messages = [{
"role": "system",
"content": "You are a voice assistant. Keep replies short and natural."
}]
# Wire the pipeline: audio in → STT → LLM → TTS → audio out
pipeline = Pipeline([
transport.input(),
stt,
LLMMessagesFrame(messages),
llm,
tts,
transport.output(),
])
task = PipelineTask(pipeline)
await PipelineRunner().run(task)
```
That’s it. Forty lines and you have a streaming voice agent that handles VAD, echo cancellation, barge-in, and full-duplex audio over WebRTC. The complexity hidden inside DailyTransport, SileroVADAnalyzer, and the streaming services is enormous, but the application code stays small. Frameworks like Pipecat and LiveKit Agents have done the heavy lifting on the orchestration so you can focus on the parts that are actually unique to your product.
## Choosing Your Components
The right component choices depend heavily on your use case:
For lowest latency: Deepgram Nova-2 (STT), a distilled 7-8B model (LLM), Cartesia Sonic or ElevenLabs Flash (TTS). This stack can hit sub-500ms consistently.
For highest accuracy: Whisper large-v3 or AssemblyAI (STT), GPT-4o or Claude Sonnet (LLM), ElevenLabs Turbo (TTS). Accuracy is better but latency is 800ms+.
For telephony deployment: Twilio handles audio transport, Deepgram for STT, small LLM, ElevenLabs or AWS Polly for TTS. The PSTN (phone network) introduces its own latency — typically 150-200ms of jitter buffer alone — so your target response time needs to account for it.
For fully self-hosted, privacy-first: Whisper.cpp (local STT), Llama-3 via Ollama (local LLM), Coqui/VITS or Piper (local TTS). Excellent for private deployments; latency depends on your hardware.
## What the LLM Prompt Looks Like
Voice-specific prompt engineering deserves a mention. Responses need to be designed for speech, not text. That means:
- No markdown formatting, bullet points, or headers. These appear verbatim in TTS and sound bizarre
- Short sentences and natural cadence. Long complex sentences with many sub-clauses are hard to follow when spoken
- Avoid lists and use “first… then… finally…” constructions instead
- Contractions and casual register feel more natural in speech
- Build in filler words sparingly (“let me check that for you”) when computation will take a moment
```
SYSTEM_PROMPT = """You are a voice assistant. Your responses will be spoken aloud.
Follow these rules strictly:
- Respond in natural conversational language, as you would speak it
- Never use bullet points, numbered lists, headers, or markdown
- Keep responses concise — 2-4 sentences for most answers
- Use natural speech patterns with contractions (you are, it is, I will)
- If you need more time to think, say 'let me think about that for a moment'
"""
```
## The Hard Problems That Remain
The basic pipeline I have described is well-understood at this point. The hard problems are what separate mediocre voice agents from good ones.
Noisy environments are the underrated killer of voice agents. A user calling from a coffee shop, or with background TV noise, or with kids in the same room sees STT accuracy drop substantially. The model still produces a transcript, but it’s wrong, and the LLM responds confidently to a query the user never asked. Tools like Krisp and RNNoise help significantly with stationary noise (HVAC, fans, traffic), but non-stationary noise (other voices, music with vocals, sudden loud sounds) is much harder. You can paper over this with voice-specific prompting that asks the user to repeat themselves when confidence is low, but it’s a workaround, not a solution.
Multi-speaker scenarios are the situation almost no production voice agent handles correctly. Imagine a family customer service call: spouse and kid in the same room, both occasionally chiming in. The agent’s STT will produce a Frankenstein transcript blending all three voices, and the LLM will respond to a query that nobody actually asked. Speaker diarization (figuring out who said what) is the answer. Pyannote-audio is the dominant open-source option, but adding it costs 200-400ms of latency on every turn and the accuracy on short utterances is still poor. Most teams just don’t solve this.
Long conversations hit context-window limits faster than text agents because every utterance needs to be transcribed, stored, and fed back as context for every subsequent turn. A 30-minute support call is easily 6,000-10,000 tokens of conversation history alone, before any system prompt or RAG context. You need a memory architecture that summarizes older turns, extracts persistent facts, and reloads relevant context on demand and crucially, all of that has to fit inside the same 500ms latency budget. This is one of the harder engineering problems in the space.
Accents and speaking styles vary STT accuracy dramatically. A model that scores 5% WER on a benchmark of American English speakers might score 15% on Indian English, 20% on Nigerian English, and 30% on heavily accented non-native speakers. Deepgram and AssemblyAI have improved substantially here, but if your product is global, you need to test specifically on the accents your users actually have, not the benchmarks the vendors quote.
## Conclusion
The voice agent space is moving fast and frameworks like LiveKit, Daily, and Pipecat have made the infrastructure significantly more accessible than it was even 18 months ago. The architectural patterns are converging. The challenge now is in the details: making the latency actually feel like presence, making the memory actually feel like continuity, and making the turn-taking actually feel like a conversation.
If you’re building anything in this space, I’d love to hear about it.
Read more such blogs from me at https://manthanguptaa.in/
If you found this interesting, I’d love to hear your thoughts. Share it on Twitter, LinkedIn, or reach out at guptaamanthan01[at]gmail[dot]com.
## References
- Silero VAD — Lightweight neural voice activity detection
- LiveKit Agents — Open-source voice agent framework
- Pipecat — Open-source framework for voice and multimodal AI applications
- Deepgram Nova-2 — Low-latency streaming STT
- Cartesia Sonic — Ultra-low-latency neural TTS
- Kyutai Moshi — Open end-to-end speech model with sub-300ms latency
- OpenAI Realtime API — End-to-end speech via GPT-4o
- Latency benchmarks from ElevenLabs, Bland.ai, and Retell.ai public documentation
## 相关链接
- [Manthan Gupta](https://x.com/manthanguptaa)
- [@manthanguptaa](https://x.com/manthanguptaa)
- [13K](https://x.com/manthanguptaa/status/2052033331546698209/analytics)
- [Bland.ai](https://bland.ai/)
- [Silero VAD](https://github.com/snakers4/silero-vad)
- [LiveKit Agents](https://github.com/livekit/agents)
- [https://manthanguptaa.in/](https://manthanguptaa.in/)
- [Twitter](https://twitter.com/manthanguptaa)
- [LinkedIn](https://www.linkedin.com/in/manthanguptaa/)
- [Silero VAD](https://github.com/snakers4/silero-vad)
- [LiveKit Agents](https://github.com/livekit/agents)
- [Pipecat](https://github.com/pipecat-ai/pipecat)
- [Deepgram Nova-2](https://deepgram.com/)
- [Cartesia Sonic](https://cartesia.ai/)
- [Kyutai Moshi](https://kyutai.org/Moshi.pdf)
- [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime)
- [Bland.ai](https://bland.ai/)
- [Retell.ai](https://retell.ai/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [10:30 PM · May 6, 2026](https://x.com/manthanguptaa/status/2052033331546698209)
- [13K Views](https://x.com/manthanguptaa/status/2052033331546698209/analytics)
- [View quotes](https://x.com/manthanguptaa/status/2052033331546698209/quotes)
---
*导出时间: 2026/5/7 21:50:56*
---
## 中文翻译
# 语音代理入门（Voice Agents 101）：能够对话的 AI 背后的架构
**作者**: Manthan Gupta
**日期**: 2026-05-06T14:30:45.000Z
**来源**: [https://x.com/manthanguptaa/status/2052033331546698209](https://x.com/manthanguptaa/status/2052033331546698209)
---
第一次我构建了一个语音代理并试着给它打电话时，感觉就像在用 2003 年的卫星电话跟人通话。长时间的停顿、不自然的节奏，还有偶尔的截断——它甚至会不停地打断我说话。同一个产品的文本版本还可以用，但语音版本根本没法用。那一刻我明白了，文本工程和语音工程根本不是一回事。
一旦引入音频，你就会面临与基于文本的系统完全不同的延迟状况、不同的故障模式以及不同的权衡取舍。语音代理不仅仅是连了个扬声器的 LLM（大语言模型）。它们是管道，而这个管道中的每一个阶段都在与同一个约束作斗争：时间。
这篇文章涵盖了语音代理管道的实际工作原理、级联与端到端语音模型之间的架构选择、你需要应对的延迟预算、全双工的含义及其难点，以及决定你的语音代理听起来自然还是机械的设计决策。
## 三阶段管道
核心来说，最常见的语音代理架构建立在三个连续的阶段之上。
语音转文本（STT）接收来自麦克风的原始音频，并将其转换为 LLM 可以处理的文本。这里的主流选择有 OpenAI Whisper（开源、极其准确，但有延迟）、Deepgram（低延迟流式转录，久经生产考验）、AssemblyAI（良好的准确率，支持流式传输）以及其他少数几家。这里的关键指标不仅仅是词错误率，还有转录生成时间——生成转录文本需要多长时间。一个准确率 98% 但耗时 800ms 生成的转录，往往比准确率 95% 但仅需 150ms 的要糟糕。
LLM 是大脑。它接收转录文本，应用上下文和记忆，并生成响应。这里可以使用与文本场景相同的模型，但提示词结构会发生变化：语音对话的回合更短、更随意，而且模型绝对不能输出 Markdown、编号列表或长篇大论。每一个响应都需要能被自然地朗读出来。
文本转语音（TTS）将 LLM 的文本响应转换为音频。其范围涵盖了从早期的机械系统到听起来非常像人的现代神经 TTS：ElevenLabs、Cartesia（延迟极低）、OpenAI TTS、PlayHT 等。在生产环境中，最重要的指标是首个音频块到达时间——用户要等多久才能听到任何声音。
将这三个阶段串联起来很简单，但要让它们感觉天衣无缝却很难。
## 级联模型 vs. 端到端语音模型
在我们深入探讨级联管道之前，你应该知道现在存在第二种架构家族，这在 18 个月前还并不存在：端到端语音模型。
级联就是我们刚才描述的。STT 将音频转录为文本，LLM 生成文本，TTS 将文本转回音频。你可以完全控制每一个阶段，拥有完整的可观测性，并且能够独立更换组件。你付出的代价是延迟（每个阶段都会增加时间，而且你无法总是完美地重叠它们）、上下文丢失（LLM 听不到语气或犹豫，只能听到转录的文字）以及更平淡的情感表面（LLM 不知道用户是否沮丧，TTS 也不会对内容做出自然的反应）。
端到端语音模型完全绕过转录步骤。模型直接接收音频并输出音频。OpenAI 的 Realtime API（基于 GPT-4o 构建）、Kyutai 的 Moshi 和 Sesame 的 CSM 是其中的突出代表。由于没有 STT 步骤，也没有独立的 TTS 步骤，延迟大幅下降。Moshi 报告的响应延迟在 200-300ms 范围内，这比任何级联系统所能达到的都更接近人类对话。由于模型直接听取并生成音频，而无需通过文本将其扁平化，因此韵律和情感也更好。
但代价是，你放弃了级联系统中那些使其可工程化的特性。你无法轻易检查模型“听”到了什么。你不能在下个季度直接换一个更好的 STT。工具调用和结构化输出会变得更弱，因为它们必须被编码到音频模态中，而不是作为文本处理。而且目前每分钟对话的成本远高于在级联系统中运行一个小型快速 LLM。
对于 2026 年的大多数生产级语音代理——无论是客户服务、电话通话，还是现有系统的语音前端——级联仍然是正确的架构。控制面、可观测性和组件生态系统的成熟度，超过了端到端模型的延迟优势。但对于自然对话比精确工具调用更重要的用例（陪伴型机器人、语言导师、免提助手），端到端正变得真正具有竞争力。在承诺使用其中一种之前，值得了解这两种架构的存在。
这篇文章的其余部分将集中在级联上，因为那是工程面上最有趣的地方。
## 延迟预算
为了让语音对话感觉自然，从用户说完一句话到听到响应的第一个字，总的往返时间需要控制在大约 500-800ms 以内。超过这个时间，用户就会开始感觉到延迟。超过 1.5 秒，感觉就像是坏了。因此，为了让语音代理感觉自然，你通常是在与时间赛跑。
让我们来分解一下，在一个简单的顺序语音代理实现中，时间都花在哪里了：
```
| 阶段 | 典型延迟 |
|-------|-----------------|
| 音频采集 + VAD 结束检测 | 100–300ms |
| STT 转录 | 150–400ms |
| LLM 首字生成时间 | 300–800ms |
| TTS 首个音频块 | 100–300ms |
| 网络开销 | 50–150ms |
| **总计** | **700ms–1.95s** |
```
即使按乐观的情况估算，你也已经处于可接受的边缘了。按悲观的情况估算，感觉就像是在和一个电话信号很差的人通话。这就是为什么语音代理从根本上说是一个延迟工程问题，而不仅仅是 AI 问题。最好的语音体验是由那些将毫秒视为一等工程问题的团队构建的。
解决方案不仅仅是孤立地让每个阶段都更快（尽管这有帮助）。真正的突破口在于流式传输，让各个阶段可以重叠运行，而不是按顺序运行。
## 流式传输：在延迟上获胜的唯一途径
在一个简单的管道中，你要等用户说完话，等完整的转录出来，发送给 LLM，等完整的响应，然后把所有内容发送给 TTS。这是完全顺序的，延迟会不断叠加。
在流式管道中，一旦 LLM 开始生成 token，你就启动 TTS，而无需等待完整的响应。LLM 一个接一个地流式传输 token，TTS 系统消耗这些 token 并开始实时合成音频，通常在生成第一个音频块之前缓冲 5-15 个 token，以避免停顿和断续。
结果就是，当 LLM 还在生成第二句话时，用户就已经听到了响应的第一个字。这就是 ElevenLabs 的 Conversational AI、Bland.ai 和 Retell AI 等公司的生产级语音系统，尽管底层使用了强大的 LLM，仍能实现低于 600ms 的端到端响应延迟的原因。
## 半双工 vs 全双工
这是语音代理设计中最重要但也最容易被忽视的架构决策之一。
半双工即“按键通话”模式：一方说话，然后另一方说话。系统在用户说话时监听，然后回应，然后再次监听。实现简单。感觉就像是来回留语音信箱。对于简单的命令-响应界面（例如：“设置一个 5 分钟的计时器”），半双工是可以的。但对于自然对话来说，这很令人沮丧。
全双工是人类对话的实际运作方式：双方可以同时说话，互相打断，在句子中间说“嗯哼”，并实时做出反应。构建这个要难得多。它要求系统同时进行监听和生成音频，这意味着你需要解决“插话”问题。
插话是指当用户在代理还在说话时开始说话的情况。在一个设计良好的全双工系统中，代理检测到用户开始说话（通过 VAD），立即停止自己的音频输出，取消任何待处理的 TTS 块，并开始处理新的输入。在一个设计糟糕的系统中，当用户试图打断时，代理只会继续喋喋不休，这非常令人恼火。
插话的技术挑战是三方面的。首先，你需要一个语音活动检测（VAD）模型，能够区分用户的声音和代理通过扬声器播放的自己的音频，否则代理会听到自己的声音并以为自己被打断了。其次，你需要优雅地处理延迟：一旦 VAD 触发，你需要在毫秒级内切断音频，而不是 300ms 之后。第三，你需要决定如何处理在插话期间捕获的部分音频。问题是：这是真实的输入还是意外的噪音？
目前最先进的技术使用 Silero VAD，这是一个轻量级的神经 VAD 模型，可以本地运行且延迟非常低（每 32ms 音频块约 10ms）。结合回声消除（从麦克风中过滤掉代理自己的音频），它为你提供了一个合理的插话检测系统。
## 轮次管理
人类并不是严格交替地说话和倾听。我们使用反向通道（“嗯嗯”、“是的”），我们在对方完全说完之前就开始说话，并且我们使用韵律线索（音调、节奏、声音渐弱）来发出轮次交接的信号。目前的语音代理系统处理得很笨拙。
简单的方法是基于能量的语音结束检测：在决定用户说完话之前，等待 N 毫秒的静音。问题在于，在深思熟虑的讲话中，500ms 的静音是很自然的，这并不意味着用户说完了。等太久会让代理感觉反应迟钝。等太短则会截断用户。
更好的方法结合了静音检测与韵律分析（音调是否以符合句子结束的模式下降？）和语义分析（目前的转录是否构成了一个完整的思路？）。有些系统使用一个小型、快速的 LLM 来根据目前的转录预测用户的 utterance 是否可能完成，这出奇地有效。
这是一个活跃的研究领域，LiveKit Agents 框架支持可插拔的轮次检测策略，包括基于模型的方法。
## 完整架构
生产级语音代理管道通常如下所示：
对话状态（历史、记忆、上下文）存在于 LLM 提示词中，每个轮次都会重建。音频传输层（音频字节如何在用户设备和你的服务器之间传输）通常由基于浏览器的代理的 WebRTC 或基于电话的代理的 SIP/RTP 处理。
## 最小可行性语音代理
具体来说，这大致就是实际连接语音代理的样子。这是一个简化的 Pipecat 风格管道，展示了各个部分如何组合在一起。生产代码会添加错误处理、可观测性和更多的配置，但架构是一样的。
```
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.transports.services.daily import DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
async def main():
# 具有内置 VAD 和回声消除的 WebRTC 传输
transport = DailyTransport(
room_url, token, "Voice Bot",
DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=False,
),
)
# 三个管道阶段
stt = DeepgramSTTService(api_key=DEEPGRAM_KEY)
llm = OpenAILLMService(api_key=OPENAI_KEY, model="gpt-4o-mini")
tts = CartesiaTTSService(api_key=CARTESIA_KEY, voice_id=VOICE_ID)
# 语音特定的系统提示词（更多内容见下文）
messages = [{
"role": "system",
"content": "You are a voice assistant. Keep replies short and natural."
}]
# 连接管道：音频输入 → STT → LLM → TTS → 音频输出
pipeline = Pipeline([
transport.input(),
stt,
LLMMessagesFrame(messages),
llm,
tts,
transport.output(),
])
task = PipelineTask(pipeline)
await PipelineRunner().run(task)
```
就这样。四十行代码，你就有了一个流式语音代理，可以处理 VAD、回声消除、插话和通过 WebRTC 的全双工音频。隐藏在 DailyTransport、SileroVADAnalyzer 和流式服务内部的复杂性是巨大的，但应用代码保持得很小。像 Pipecat 和 LiveKit Agents 这样的框架已经在编排方面完成了繁重的工作，这样你就可以专注于那些真正对产品独特的部分。
## 选择你的组件
正确的组件选择很大程度上取决于你的用例：
为了最低延迟：Deepgram Nova-2 (STT)、一个蒸馏过的 7-8B 模型 (LLM)、Cartesia Sonic 或 ElevenLabs Flash (TTS)。这个堆栈可以持续实现低于 500ms 的延迟。
为了最高准确率：Whisper large-v3 或 AssemblyAI (STT)、GPT-4o 或 Claude Sonnet (LLM)、ElevenLabs Turbo (TTS)。准确率更好，但延迟在 800ms 以上。
用于电话部署：Twilio 处理音频传输，Deepgram 用于 STT，小型 LLM，ElevenLabs 或 AWS Polly 用于 TTS。PSTN（电话网络）引入了自己的延迟——仅抖动缓冲通常就有 150-200ms ——所以你的目标响应时间需要考虑到这一点。
对于完全自托管、隐私优先：Whisper.cpp（本地 STT）、通过 Ollama 运行的 Llama-3（本地 LLM）、Coqui/VITS 或 Piper（本地 TTS）。非常适合私有部署；延迟取决于你的硬件。
## LLM 提示词是什么样的
针对语音的提示词工程值得一提。响应需要为语音而设计，而不是为文本。这意味着：
- 没有 Markdown 格式、项目符号或标题。这些会在 TTS 中逐字朗读，听起来很奇怪。
- 短句子和自然的节奏。当大声朗读时，带有许多从句的长难句很难跟上。
- 避免列表，改用“首先……然后……最后……”的结构。
- 缩略语和随意的语域在语音中感觉更自然。
- 当计算需要一点时间时，适当地加入填充词（“让我帮你查一下”）。
```
SYSTEM_PROMPT = """You are a voice assistant. Your responses will be spoken aloud.
Follow these rules strictly:
- Respond in natural conversational language, as you would speak it
- Never use bullet points, numbered lists, headers, or markdown
- Keep responses concise — 2-4 sentences for most answers
- Use natural speech patterns with contractions (you are, it is, I will)
- If you need more time to think, say 'let me think about that for a moment'
"""
```
## 仍然存在的难题
我描述的基本管道目前已经很成熟了。难题在于区分平庸的语音代理和优秀的语音代理。
噪声环境是语音代理的低估杀手。在咖啡店打电话的用户，有背景电视噪音，或者家里有