如何构建 AI 语音代理：全流程指南

✍ Avid🕐 2026-05-19📦 25.0 KB 🟢 已读 𝕏 文章列表

文章揭示了构建语音代理的核心要素：并非追求最强模型，而是建立低延迟的实时管道。作者阐述了三种主流架构，并详细拆解了链条式流程中 STT（听力）、LLM（大脑）、TTS（嘴巴）、检索（记忆）与执行（手）这五个关键组件的选型、调优与避坑指南，强调对话设计的重要性。

# How to Build a Voice Agent using AI (Full Guide) **作者**: Avid **日期**: 2026-05-18T16:30:42.000Z **来源**: [https://x.com/Av1dlive/status/2056412170402091070](https://x.com/Av1dlive/status/2056412170402091070) --- ![Image](/doc/2057/image/img_7b93dfad2fea.jpg) here is the truth no one tells ai builders . voice agents don't need the best model. all they need is > TLDR; if you find reading boring or your attention span is cooked, you can use the skill file I made to get the entire article and paste it to your agent > ➡️https://github.com/codejunkie99/voice-agent-builder all you need to build is: - a real-time pipeline with a real latency budget - five components wired in the right order - grounding strong enough to keep the model honest - a weekly review loop that compounds OpenAI shipped GPT-Realtime-2 on May 7, 2026. Salesforce AI Research published the VoiceAgentRAG paper on March 1, the same week Deepgram Flux moved from beta to GA. The pieces stopped being the problem. What stayed the problem is how you wire them, and what you write the agent to say. I spent the last three months building voice agents that actually answer the phone. I am not going to pretend any of it was clean. - The first build sounded like a kiosk. I scrapped it in two days. - The second build "booked" four phantom appointments in the first hour before I noticed. - The third build leaked memory because I forgot to invalidate the context cache after the background extractor wrote new facts. - By the time anything worked, the system was the fourth rewrite. The version I would defend now has a small set of properties I will spend the next 6,000 words explaining. - The pipeline has one job inside one budget. Five components, sub-700ms end-to-end, no exceptions. - Knowledge lives in your documents and gets retrieved with a dual-agent cache, not pulled out of the model's head. - Conversation design is the discipline of writing for ears, not eyes. Most teams treat this as cosmetic. It is not. - Every turn writes a structured log I can replay against the current config 90 days later. This article is what those 90 days actually taught me, plus the two or three bets I would make first if I started over today.🔽🔽 ## What a voice agent actually is A voice agent is not a chatbot with a microphone bolted on. It is not a TTS wrapper around a text API. It is a real-time audio system. Latency-constrained. Five components coordinating inside a 300 to 800 millisecond window. The pipeline in the order events actually happen: 1. User speaks 2. Audio gets captured 3. Streaming STT transcribes word by word, while the person is still speaking 4. The agent reads the transcript and retrieves relevant knowledge from your documents 5. The LLM generates a reply 6. TTS speaks the reply aloud 7. User hears it Every single one of those arrows is a component you can choose, tune, and swap. I tried building it the chatbot way first. STT completes, send to LLM, wait for full response, send to TTS, wait for full audio, play. It felt awful. Like talking to a kiosk. Two days in I deleted it. The reason it felt awful is not that the latency numbers were bad. They were fine on paper. The reason is that humans do not converse in turns. They converse in overlapping streams. - The agent has to start formulating a response while the user is still finishing the sentence. - The TTS has to begin speaking before the LLM has finished writing. - The STT has to keep listening while the agent is talking, so it knows when to shut up. A voice agent that cannot be interrupted is not a voice agent. It is voicemail. ## The three architectures There are only three. Pick by what you need to control. Chained pipeline - Separate STT, LLM, TTS services wired together - Three independent models, each specialized for its job - Text flows between them - Latency lands around 600 to 700ms on a well-tuned managed platform - Most controllable, most debuggable, easiest to upgrade one layer at a time Half-cascade - Audio goes directly into a multimodal model that hears the audio, not the transcript - Catches frustration in someone's voice, a question implied by rising tone, a mid-sentence language switch - Output still routes through a specialized TTS for audio control - Latency drops to 300 to 500ms Native speech-to-speech - One model, audio in, audio out - No transcription layer, no text handoffs - Every major lab shipped a native voice model in 2026 - Latency drops to 200 to 300ms, below the threshold where callers stop noticing they are talking to AI Which to start with 1. Start with the chained pipeline. The best tooling exists for it. Move to speech-to-speech once you have proven your product on the pipeline and want a step-function latency improvement. 2. I tried speech-to-speech first for everything. It was excellent for booking flows. 3. It fell apart on a 12-step intake form because the single model could not hold the state machine in its head without context bloat by turn nine. 4. I moved that one to a chained pipeline with a real state machine layer and the completion rate jumped from 61% to 89% in three days. 5. Tool scoping per state was the entire fix. ## The five components you have to wire Every chained pipeline has the same five components. Five jobs that need to be filled before your agent takes its first call. ## The ears (Streaming STT) The STT model converts incoming audio into text in real time, word by word, while the person is still speaking. This is the most consequential component in your stack. A transcription error here cascades through everything below. What to look for in 2026: - Streaming accuracy. Accurate while the person is speaking, not just after they finish. - Word Error Rate. 6 to 8% on real production audio is good. Past 12% will frustrate users on every third call. - Built-in end-of-turn detection. The single biggest UX upgrade of 2026. Why end-of-turn detection matters: - Generic STT returns transcripts. It does not tell you when the speaker has finished. - Without it, your agent either interrupts mid-sentence or waits two awkward seconds. - The 2026 wave of streaming STT models ships with end-of-turn detection inside the same network that produces the transcript. - The model emits a turn-complete signal when it has decided the speaker is done. - The signal uses semantic context, not just acoustic silence. It catches trailing off and ignores breath pauses. - Switch to this if your provider has shipped it. The pause before the agent starts speaking drops 200 to 400ms on every turn. ## The brain (LLM) The LLM reads the transcript, the conversation history, the retrieved knowledge, and decides what to say. It also decides actions, not just words. Voice-specific rules: - Use the small fast model, not the flagship. Frontier reasoning models take 1500ms to generate the first word. That is dead air. Smaller models in the same family almost always win on voice turns. - Escalate to the big model only for specific complex tool calls that need real planning. - Cap the system prompt at 800 tokens. It reloads on every turn. A 4000-token prompt adds latency to every single message. Function calling, in plain English: - You define each function with a description of what it does and what information it needs. - The LLM reads the description and decides when to call it based on conversation state. - No conditional logic tree. The LLM matches intent to function from natural language. The most common production failure with function calling is not what you would expect: - The LLM does not throw an error when it cannot call a function. It narrates the action instead. - "I've confirmed your booking." Nothing was called. User thinks they are booked. They are not. - The fix is to scope tools to the current state. A "collect name" state must not expose book_appointment. A "confirm details" state must not expose check_availability. - The state machine is the safety rail, not the system prompt. ## The knowledge (RAG) RAG is the mechanism that lets your agent answer from your documents instead of the model's training data. Why you cannot skip this: - LLMs are trained on the public internet up to a cutoff date. - They know a lot about the world. They know nothing specific about your products, prices, policies, customers. - Without RAG, an agent asked "what's in the enterprise plan?" will hallucinate confidently. - With RAG, it retrieves the actual answer from your documentation before responding. The basic mechanic: - User asks a question. - System embeds the query. - Vector database returns top relevant document chunks. - Chunks get injected into the LLM's context. - LLM is instructed to answer only from that context. The voice-specific challenge: - A typical vector database query adds 50 to 300ms to the pipeline. - Combined with STT, LLM, and TTS, that blows your latency budget. - The fix is the dual-agent cache pattern. Whole section on this below. ## The mouth (TTS) The TTS converts text to spoken audio. Sounds simple. Actually a major differentiator in perceived quality. What matters: - Time-to-first-audio. A TTS that takes 200ms to start speaking burns a third of your latency budget on the output layer alone. - Voice quality. Humans are extraordinarily sensitive to synthetic speech. Subtle artifacts, unnatural pacing, misplaced stress all read as a verdict on the whole system. - Pick the voice intentionally. It is a trust signal before the user has heard a sentence. ## The hands (Functions and integrations) Functions are actions the LLM can take mid-conversation: - Book appointments - Look up order statuses - Send confirmation SMS - Transfer to a human - Update records in your CRM This is the architectural shift that makes modern voice agents dramatically more capable than press-1-for-billing systems. ## The latency budget you have to fit inside The most important non-obvious thing about voice agents: every millisecond of processing time is a millisecond of silence the caller sits inside. The math: - Humans expect a conversational reply within 500 to 700ms of finishing a sentence - Past one second feels like the system is struggling - Past two seconds, callers start talking over the agent That 700ms is your entire budget, split across every component. Per-component budget, fast lane vs slow lane: - Transport. 20-50ms peer-to-peer. 50-100ms over relays. - STT first interim. 100-150ms on a cache hit. 150-250ms on a miss. - End-of-turn detection. Model-integrated, ~50ms. Silence-threshold, 300-600ms. - RAG retrieval. Sub-millisecond on a cache hit. 80-150ms on local BM25 + rerank. - LLM time-to-first-token. 150-250ms with a small model. 400-600ms with a frontier model. - TTS time-to-first-audio. 60-100ms on the fast tier. 150-250ms on the quality tier. - Network overhead. 40-80ms total inside one region. 100-160ms total across regions. - End-to-end. ~440ms on the fast lane. ~700-900ms on the slow lane. The two biggest unlocks in 2026: 1. Model-integrated end-of-turn detection. Moves 200 to 400ms off every turn. Single biggest upgrade you can make this year. 2. Speculative prefetch with a dual-agent cache. Gets retrieval from "miss with vector search" to "hit with cache lookup" on roughly 40% of turns. Everything else is rounding error compared to those two. ## The dual-agent RAG pattern Standard RAG inside a voice loop is a problem. The vector database query takes 80 to 300ms and blows your latency budget on every turn. The 2026 research answer comes from Salesforce AI Research's VoiceAgentRAG paper, published in March. The insight is simple. - In a real conversation, the next question is usually predictable from the current one. - Someone asking about pricing will probably follow up about the enterprise tier. - Someone asking about installation will probably ask about compatibility next. So you run two agents at the same time. The background agent (Slow Thinker) - Runs while the user is listening to the current response - Predicts the three to five most likely follow-up questions using the LLM - Pre-fetches relevant document chunks for each prediction - Stores them in a local in-memory cache before the user has finished hearing the current answer The foreground agent (Fast Talker) - Handles the next live question by checking the in-memory cache first - A cache lookup takes sub-millisecond time vs 110ms for a remote vector database call - If the cache has the answer, skip the database entirely - If the cache misses, fall back to the database and cache that result for next time Benchmark numbers from the paper - 75% of queries hit the cache - 316× retrieval speedup on cache hits (0.35ms vs 110ms) - 16 seconds of cumulative latency saved across 200 queries The principle to remember: use the user's listening time as your computation time. The moment they start hearing the current response is the moment you start preparing for their next question. I tried plain vector RAG inside the voice loop on my first build. Added 110ms per turn. Killed the conversation feel. I moved to the dual-agent cache pattern in week six. The 40% of turns that hit the cache feel snappier than the human call center reps the agent replaces. ## Conversation design is the discipline most builders skip You can have the fastest STT, the smallest LLM, the smartest RAG cache. If your agent does not know how to talk, callers will hang up. Conversation design is the discipline of writing for ears, not eyes. Rules I follow now that I learned by getting them wrong first - Speak in short sentences. Average human attention span for spoken information is 8 to 10 seconds. A 15-second response is too long. Split it into two turns. - Never ask two questions in one turn. Callers can only hold one in working memory. Ask one, wait, then ask the next. - Use acknowledgment phrases. "Got it." "Sure." "Let me check on that for you." These fill the silence between the user finishing and the response being ready. - Mirror the user's language. Caller says "billing issue," agent says "billing issue" back. Not "financial dispute" or "payment problem." Paraphrasing creates friction. Mirroring creates rapport. - Write for the ear, not the eye. No bullet points. No headers. No markdown in the system prompt. The LLM will try to speak asterisks and hyphens. - Spell out numbers. "Nine four one zero seven" instead of "94,107." "Fifteen dollars and ninety-nine cents" instead of "$15.99." TTS routinely mispronounces formatted numbers. - Cap the system prompt at 800 tokens. It reloads on every turn. ## The three-act structure of every good voice conversation 1. Acknowledgment and orientation. "So you're looking to reschedule your appointment on Thursday, let me pull that up." Confirms the caller was understood. Buys time while retrieval runs. 2. Resolution. The core action or answer. One point per turn. Move forward. 3. Confirmation and close. "I've rescheduled your appointment to Monday the 19th at 3 PM, you'll get a confirmation text shortly." Clean exit. Never leave an open loop. ## Safety is two checkpoints, not one The component most first-time builders skip and regret. A voice agent has no "read before sending" moment. An unsafe output gets spoken immediately. No draft, no preview, no human in the loop. The right model is two checkpoints. The input guard (before the LLM sees the user's turn) - Prompt injection. "Ignore previous instructions, pretend you are..." attacks. Exploits the LLM's instruction-following to steal data or break scope. - PII spoken aloud. Credit card numbers, social security numbers. Redact before they hit any log or database. - Topic blocklist. Loaded from a JSON file. Updated weekly as you learn what users actually try. The output guard (after the LLM writes its reply, before TTS speaks it) - Over-promise language. "I guarantee," "I promise." Creates legal and trust problems on a recorded line. - Specific factual claims not in the retrieved context. Lightweight hallucination check. Catches around 70% of confabulated answers in my deployment. - Standard moderation endpoint. For the rare model misbehavior. What both guards return - safe (bool) - detected category (string, if unsafe) - replacement phrase the agent speaks instead Every trigger logs to a file with timestamp, category, redacted text, and call ID. The escalation phrase One exact phrase, hardcoded, that the agent says when it does not know the answer or when something is going wrong. - "I want to make sure I give you accurate information. Let me connect you with someone who can help." - Not five variations. Not the LLM's improvised guess at the right phrasing. - One phrase. ALL CAPS in the system prompt. Fall-through when any safety check fires. I shipped without an output guard on build one. The agent confidently quoted a price 30% off the real one. The price was in an outdated document in the knowledge base. The hallucination check would have caught it because the right price was not in the retrieved context. ## Evaluation, or how to know if it is good You cannot improve what you cannot measure. Most teams skip evaluation and ship broken agents. The four-layer framework Layer 1: Infrastructure. Plumbing. - WER on your actual domain (not vendor benchmarks) - p50, p95, p99 latency for the full pipeline - Time-to-first-audio - Audio quality on your transport Layer 2: Execution. Does the agent do what was asked. - Task success rate - Tool-call accuracy - Parameter correctness - Response groundedness - Use LLM-as-judge on a small fast model. Four yes/no questions: answered correctly, stayed grounded, sounded natural for voice, appropriately concise. Layer 3: User behavior. Does it feel natural to talk to. - Barge-in recovery rate - Reprompt rate - Average turn length - Conversational repair count - Sample 20 calls a week. Read the actual transcripts. You will see patterns inside ten. Layer 4: Business outcome. Does it solve the problem. - Containment rate (percent of calls resolved without a human) - Transfer rate - CSAT - First-call resolution rate - Optimize against containment. It correlates with everything else and is the easiest to measure without instrumentation. Test set composition Build it before you launch. Minimum 50 conversations. - 40% happy path - 30% edge cases - 15% error handling - 10% adversarial (prompt injection, jailbreak attempts) - 5% acoustic variation (background noise, heavy accent, speakerphone) For each scenario: - Which tool should have been called - With what parameters - What the agent should have said ## The weekly review loop Every Monday morning. 30 minutes. 1. Pull metrics 2. Sample 20 calls (7 escalated, 7 resolved, 6 random) 3. Read the transcripts 4. Name the single most common failure type 5. Make one change (one variable at a time, always) 6. A/B test it for 48 hours 7. Ship the winner ## Grounding is a trust system Most builders think about RAG as a performance feature, a way to get more accurate answers. That framing undersells it. In a voice agent, the accuracy of every answer is a direct statement about how trustworthy your product is. A caller who hears a wrong answer about pricing or coverage or policy, said confidently in a natural-sounding voice, will not just be frustrated. They will feel deceived. The implementation of the trust promise has four parts. 1. Source of truth - Your documents, not the model's training data - The system prompt has to say this explicitly, in capital letters: ANSWER ONLY FROM THE CONTEXT PROVIDED - The model will still drift toward general knowledge sometimes, but the explicit instruction cuts the rate by an order of magnitude 2. Graceful refusal - When the agent cannot find an answer, it says so directly - The exact phrase matters - "I want to make sure I give you accurate information, let me check on that" buys you a graceful transfer - "I'm not sure" sounds like incompetence - "Based on my information" sounds like a hedge from a lawyer - Pick one phrase, hardcode it, never let the LLM improvise here 3. Confidence-aware response - Top BM25 score on retrieved chunks is a useful proxy for confidence - Score above 0.6: agent answers with confidence - Score 0.3 to 0.6: agent answers but adds an "I think" hedge - Score below 0.3: agent does not answer, offers to transfer - 20-line change in system prompt construction code. Cuts hallucinations roughly in half. 4. Knowledge base hygiene - Outdated documents produce outdated answers, which are dangerous answers - I run a Friday audit: read the bottom 5% of confidence-scored responses from the week - Half the time the answer was right but the retrieval found a stale chunk - Update the chunk, re-embed, next week is quieter ## What to watch out for Six failure modes that will hit you. VAD in the pipeline instead of the transport - Problem. Agent triggers on its own TTS output, enters a barge-in loop, or fails to detect end of turn entirely. - Solution. VAD analyzer goes on the transport. Always. Pair it with an echo guard that ignores STT transcripts matching recent assistant output. Tools available in the wrong state - Problem. LLM calls book_appointment in a state still collecting the patient's name. Or invents a booking that never happened. - Solution. Scope tools per state. One state, only its own functions. The state machine is the safety rail, not the system prompt. Function handler throws and never calls the result callback - Problem. LLM hangs waiting for a tool result that never comes. Or hallucinates one. - Solution. Every handler wraps in try/except. Every branch sends a result back. Every failure has a spoken fallback. Never an empty result. Validating user data in the prompt instead of in code - Problem. LLM accepts "john@" as a real email on call 12. Rejects a valid one with a plus sign on call 47. - Solution. Validation lives in Python. Regex for email, date parser for dates, name length check, a re-ask response when validation fails. Context window grows unbounded over a long call - Problem. p95 latency drifts upward over the week without code changes. By turn 20 you are sending 12K tokens per turn. - Solution. Sliding window of last N turns plus system prompt. Or milestone-based context resets at the end of each discrete stage. TTS reads codes and IDs literally - Problem. Confirmation code "A3X7" comes out "ay three ex seven" with no pause. Patient asks you to repeat anyway. - Solution. NATO phonetic alphabet expansion with SSML break tags. Sounds slower. Reads correctly the first time. ## Things I would do differently - Build the turn log schema on day one, not week four. The replay endpoint is the most valuable tool I built and I built it after I needed it. - Use semantic end-of-turn detection from the start instead of fighting silence thresholds. - Move to a real state machine the day the system prompt crosses 300 words. Do not try to encode a state machine in prose. - Stop validating in prompts. The LLM is not a parser. Python is a parser. Use Python. - Cache the five most likely RAG documents at call start. Skip vector search inside the turn loop. - Build the small-talk gate before you build retrieval. "Hi" is the cheapest 200ms win in the system. - Run the eval set before the first production call. 50 conversations minimum. - Put a durable extraction queue in from day one. A pending_extractions Postgres table with a single retry worker takes 200 lines and saves you a real outage. - Run an async LLM judge on every 50th call. Score on groundedness, relevance, brevity. Pipe it to a dashboard. The drift is real. - Run the weekly review loop. Sample 20 calls every Monday. Make one change. A/B test. Ship the winner. ## Conclusion Voice agents look like AI. They run like real-time systems. Teams that ship treat them that way. Teams that ship six months late think a better prompt fixes a system problem. Own your pipeline. Own your logs. Keep them in plain files where any failure is one replay away. The first agent took me a weekend. The production system took ten weeks. It has been getting better every day since, without me touching it. The user does not measure that. They notice the agent answered "thanks" without making them wait. ## Disclaimers and Disclosures This article was researched and written by the author, and it was edited by an AI model. The thumbnail was taken off Pinterest. This article was researched and written by the author while working on voice agents in deeper infrastructure. It is based on evolving notes and deep research using Perplexity, Claude, and ChatGPT, along with system design and API design from a few undergraduate-level college books. It has been thoroughly edited by Minimax M2.7 and Claude Opus 4.7 for grammatical errors and formatting. ## 相关链接 - [Avid](https://x.com/Av1dlive) - [@Av1dlive](https://x.com/Av1dlive) - [30K](https://x.com/Av1dlive/status/2056412170402091070/analytics) - [https://github.com/codejunkie99/voice-agent-builder](https://github.com/codejunkie99/voice-agent-builder) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [12:30 AM · May 19, 2026](https://x.com/Av1dlive/status/2056412170402091070) - [30.3K Views](https://x.com/Av1dlive/status/2056412170402091070/analytics) - [View quotes](https://x.com/Av1dlive/status/2056412170402091070/quotes) --- *导出时间: 2026/5/19 09:40:00* --- ## 中文翻译 # 如何利用 AI 构建语音代理（完整指南） **作者**: Avid **日期**: 2026-05-18T16:30:42.000Z **来源**: [https://x.com/Av1dlive/status/2056412170402091070](https://x.com/Av1dlive/status/2056412170402091070) --- ![Image](/doc/2057/image/img_7b93dfad2fea.jpg) 真相是这样的，没有人告诉 AI 构建者。语音代理并不需要最好的模型。它们只需要 > TLDR; 如果你觉得阅读很无聊，或者你的注意力已经不行了，你可以使用我制作的技术文件来获取整篇文章，并将其粘贴给你的代理 > ➡️https://github.com/codejunkie99/voice-agent-builder 你需要构建的全部内容是： - 一个具有真正实时延迟预算的流水线 - 五个按正确顺序连接的组件 - 足够强的 Grounding（知识锚定）以保持模型诚实 - 一个能够产生复利效应的每周回顾循环 OpenAI 于 2026 年 5 月 7 日发布了 GPT-Realtime-2。Salesforce AI Research 在 3 月 1 日发表了 VoiceAgentRAG 论文，同一周 Deepgram Flux 从测试版正式发布（GA）。组件已不再是问题。依然存在的问题在于你如何连接它们，以及你让代理说什么。过去三个月，我一直在构建真正能接听电话的语音代理。我不会假装这个过程有多干净。 - 第一个版本听起来像个自助服务终端。我两天后就把它废弃了。 - 第二个版本在第一小时内就“预订”了四个幽灵预约，这才发现。 - 第三个版本因为内存泄漏而崩溃，因为我忘记在后台提取器写入新事实后使上下文缓存失效。 - 等到任何东西能正常工作时，系统已经是第四次重写了。我现在能捍卫的版本具有一小部分特性，我将在接下来的 6,000 字中解释它们。 - 流水线在一个预算内完成一项工作。五个组件，端到端延迟低于 700 毫秒，无一例外。 - 知识存在于你的文档中，并通过双代理缓存进行检索，而不是从模型的脑海中提取。 - 对话设计是为耳朵写作、而非为眼睛写作的学科。大多数团队将其视为表面功夫。其实不然。 - 每一轮对话都会写入一个结构化日志，我可以在 90 天后针对当前配置重放它们。这篇文章是这 90 天真正教会我的东西，加上如果今天让我重新开始我会首先做的两三个赌注。🔽🔽 ## 语音代理到底是什么语音代理不是安装了麦克风的聊天机器人。也不是文本 API 的语音合成（TTS）封装。它是一个实时音频系统。受延迟约束。五个组件在 300 到 800 毫秒的窗口内协调工作。按事件实际发生顺序排列的流水线： 1. 用户说话 2. 音频被捕获 3. 流式 STT（语音转文字）在用户仍在说话时逐字转录 4. 代理阅读转录文本并从你的文档中检索相关知识 5. LLM 生成回复 6. TTS 大声读出回复 7. 用户听到每一个箭头代表一个你可以选择、调整和交换的组件。我起初尝试用聊天机器人的方式构建它。STT 完成，发送给 LLM，等待完整响应，发送给 TTS，等待完整音频，播放。感觉很糟。就像跟自助服务终端说话一样。两天后我删了它。感觉糟糕的原因不是延迟数字不好。纸面上它们看起来还行。原因是人类不会轮流对话。他们以重叠的流的形式交流。 - 代理必须在用户还在说完一句话时就开始构思回复。 - TTS 必须在 LLM 写完之前就开始说话。 - STT 必须在代理说话时保持监听，以便它知道何时闭嘴。无法被中断的语音代理不是语音代理。它是语音信箱。 ## 三种架构只有三种。根据你需要控制的内容来选择。链式流水线 - 独立的 STT、LLM、TTS 服务连接在一起 - 三个独立的模型，每个都专精于其工作 - 文本在它们之间流动 - 在调整得当的托管平台上，延迟大约在 600 到 700 毫秒 - 最可控，最易调试，最容易一次升级一层半级联 - 音频直接进入多模态模型，它听的是音频而不是转录文本 - 能捕捉某人声音中的挫败感、语调上升暗示的疑问、句子中间的语言切换 - 输出仍通过专用 TTS 路由以进行音频控制 - 延迟降至 300 到 500 毫秒原生语音到语音 - 一个模型，音频进，音频出 - 没有转录层，没有文本交接 - 2026 年，每个主要实验室都发布了一个原生语音模型 - 延迟降至 200 到 300 毫秒，低于呼叫者开始意识到他们在与 AI 交谈的阈值从哪一个开始 1. 从链式流水线开始。针对它的最佳工具已经存在。在你通过流水线验证了产品并想要阶跃式延迟改进后，再转到语音到语音。 2. 我起初对所有事情都尝试了语音到语音。它对预订流程来说非常出色。 3. 但在 12 步的 intake 表单上它崩溃了，因为单个模型无法在不造成上下文膨胀的情况下，在第 9 轮时在其脑海中维持状态机。 4. 我将其转移到一个具有真正状态机层的链式流水线，完成率在三天内从 61% 跃升至 89%。 5. 针对每个状态进行工具范围界定是唯一的修复方法。 ## 你必须连接的五个组件每个链式流水线都有相同的五个组件。在你的代理接通第一个电话之前，必须填补这五个职位。 ## 耳朵（流式 STT） STT 模型将传入的音频实时转换为文本，逐字转换，甚至在人还在说话时。这是你的技术栈中后果最严重的组件。这里的转录错误会级联影响到下面的所有环节。在 2026 年要注意什么： - 流式准确性。在人说话时准确，而不仅仅是说完后。 - 词错误率。在真实生产音频上 6% 到 8% 是好的。超过 12% 会在每三次通话中让用户感到挫败。 - 内置的轮次结束检测。这是 2026 年最大的用户体验升级。为什么轮次结束检测很重要： - 通用的 STT 返回转录文本。它不告诉你说话者何时完成。 - 没有它，你的代理要么会在句子中间打断，要么会尴尬地等待两秒钟。 - 2026 年的新一代流式 STT 模型在生成转录文本的同一网络内内置了轮次结束检测。 - 当模型判定说话者完成时，它会发出轮次完成信号。 - 该信号使用语义上下文，而不仅仅是声学静默。它能捕捉话音渐弱并忽略呼吸停顿。 - 如果你的提供商已推出此功能，请切换过去。代理开始说话前的停顿每轮可减少 200 到 400 毫秒。 ## 大脑（LLM） LLM 阅读转录文本、对话历史、检索到的知识，并决定说什么。它还决定行动，而不仅仅是文字。语音特定规则： - 使用小型快速模型，而不是旗舰模型。前沿推理模型生成第一个词需要 1500 毫秒。那是死寂时间。同一系列中的较小模型在语音轮次中几乎总是获胜。 - 仅在需要真正规划的特定复杂工具调用时升级到大型模型。 - 将系统提示词限制在 800 个 token。它每轮都会重新加载。4000 token 的提示词会增加每条消息的延迟。函数调用，用通俗英语解释： - 你用描述来定义每个函数，说明它的作用以及需要什么信息。 - LLM 阅读描述，并根据对话状态决定何时调用它。 - 没有条件逻辑树。LLM 从自然语言中将意图与函数匹配。函数调用中最常见的生产性故障并非你所料： - 当 LLM 无法调用函数时，它不会抛出错误。相反，它会叙述这个动作。 - “我已经确认了你的预订。” 什么也没调用。用户以为他们预订好了。其实没有。 - 修复方法是将工具范围限定在当前状态。一个“收集姓名”的状态绝不能暴露 `book_appointment`。一个“确认详情”的状态绝不能暴露 `check_availability`。 - 状态机是安全护栏，而不是系统提示词。 ## 知识（RAG） RAG 是一种机制，让你的代理能够根据你的文档回答问题，而不是依靠模型的训练数据。为什么你不能跳过这个： - LLM 的训练数据来自公共互联网，截止到某个日期。 - 它们对世界了解很多。但对你的产品、价格、政策、客户一无所知。 - 没有 RAG，当被问及“企业计划里有什么？”时，代理会自信地产生幻觉。 - 有了 RAG，它会在响应之前从你的文档中检索实际答案。基本机制： - 用户提问。 - 系统嵌入查询。 - 向量数据库返回最相关的文档块。 - 块被注入到 LLM 的上下文中。 - 指示 LLM 仅根据该上下文回答。语音特有的挑战： - 典型的向量数据库查询会给流水线增加 50 到 300 毫秒。 - 结合 STT、LLM 和 TTS，这会耗尽你的延迟预算。 - 修复方法是双代理缓存模式。下面有专门的一节来讲这个。 ## 嘴巴（TTS） TTS 将文本转换为口语音频。听起来很简单。实际上是感知质量的主要差异化因素。什么很重要： - 首音频时间。一个需要 200 毫秒才能开始说话的 TTS 仅在输出层就烧掉了你三分之一的延迟预算。 - 语音质量。人类对合成语音非常敏感。微妙的伪影、不自然的节奏、错误的重音都会被解读为对整个系统的裁决。 - 有意地选择声音。在用户听到一句话之前，这是一个信任信号。 ## 双手（函数和集成）函数是 LLM 在对话中可以采取的行动： - 预约 - 查询订单状态 - 发送确认短信 - 转接给人工 - 更新 CRM 中的记录这是架构上的转变，使现代语音代理比“按 1 选择账单”系统的能力显著增强。 ## 你必须适应的延迟预算关于语音代理，最不明显但最重要的一点是：每一毫秒的处理时间都是呼叫者忍受的一毫秒沉默。计算如下： - 人类期望在句子结束后的 500 到 700 毫秒内得到对话回复 - 超过一秒钟感觉系统在挣扎 - 超过两秒钟，呼叫者开始对着代理说话这 700 毫秒是你的全部预算，分配给每个组件。各组件预算，快车道 vs 慢车道： - 传输。点对点 20-50 毫秒。中继传输 50-100 毫秒。 - STT 首次 interim（中间结果）。缓存命中 100-150 毫秒。未命中 150-250 毫秒。 - 轮次结束检测。模型集成，约 50 毫秒。静音阈值，300-600 毫秒。 - RAG 检索。缓存命中小于 1 毫秒。本地 BM25 + 重排 80-150 毫秒。 - LLM 首个 token 时间。小型模型 150-250 毫秒。前沿模型 400-600 毫秒。 - TTS 首音频时间。快速层级 60-100 毫秒。质量层级 150-250 毫秒。 - 网络开销。同一区域内总共 40-80 毫秒。跨区域总共 100-160 毫秒。 - 端到端。快车道约 440 毫秒。慢车道约 700-900 毫秒。 2026 年最大的两个解锁点： 1. 模型集成的轮次结束检测。每轮减少 200 到 400 毫秒。今年你能做出的最大升级。 2. 带有双代理缓存的推测性预取。将检索从“向量搜索未命中”变为“缓存查找命中”，大约覆盖 40% 的轮次。与这两点相比，其他一切都是误差。 ## 双代理 RAG 模式在语音循环中使用标准 RAG 是个问题。向量数据库查询需要 80 到 300 毫秒，并且每轮都会耗尽你的延迟预算。 2026 年的研究答案来自 Salesforce AI Research 于 3 月发表的 VoiceAgentRAG 论文。见解很简单。 - 在真实的对话中，下一个问题通常可以从当前问题预测出来。 - 询问定价的人很可能会跟进询问企业版。 - 询问安装的人很可能会接着询问兼容性。所以你要同时运行两个代理。后台代理（慢思考者） - 在用户正在听当前回复时运行 - 使用 LLM 预测三到五个最可能的后续问题 - 为每个预测预取相关的文档块 - 在用户听完当前答案之前将它们存储在本地内存缓存中前台代理（快语者） - 通过首先检查内存缓存来处理下一个实时问题 - 缓存查找时间小于 1 毫秒，而远程向量数据库调用需要 110 毫秒 - 如果缓存有答案，完全跳过数据库 - 如果缓存未命中，回退到数据库并为下次缓存该结果论文中的基准数据 - 75% 的查询命中缓存 - 缓存命中的检索速度提升 316 倍（0.35 毫秒 vs 110 毫秒） - 在 200 个查询中节省了 16 秒的累积延迟要记住的原则：利用用户的倾听时间作为你的计算时间。当他们开始听到当前回复的那一刻，就是你开始准备下一个问题的时刻。我在第一次构建时尝试在语音循环中使用纯向量 RAG。每轮增加 110 毫秒。破坏了对话的感觉。我在第六周转移到了双代理缓存模式。40% 命中缓存的轮次感觉比代理所替代的人工呼叫中心代表更敏捷。 ## 对话设计是大多数构建者跳过的学科你可以拥有最快的 STT，最小的 LLM，最聪明的 RAG 缓存。如果你的代理不知道如何说话，呼叫者会挂断。对话设计是为耳朵写作，而非为眼睛写作。我现在遵循的规则，是我先搞砸后才学到的： - 说短句。人类对口语信息的平均注意力持续时间是 8 到 10 秒。15 秒的回复太长了。把它分成两轮。 - 永远不要在一轮中问两个问题。呼叫者只能在工作记忆中保持一个。问一个，等待，然后再问下一个。 - 使用确认短语。“明白了。”“当然。”“让我帮你查一下。”这些填补了用户结束和响应准备好之间的沉默。 - 镜像用户的语言。呼叫者说“账单问题”，代理也回说“账单问题”。而不是“财务纠纷”或“付款问题”。释义会产生摩擦。镜像会产生共鸣。 - 为耳朵写作，而不是为眼睛。没有要点。没有标题。系统提示词中没有 markdown。LLM 会尝试读出星号和连字符。 - 拼写出数字。“九四一零七”而不是“94,107”。“十五美元九十九美分”而不是“$15.99”。TTS 经常会误读格式化的数字。 - 将系统提示词限制在 800 个 token。它每轮都会重新加载。 ## 每一个良好语音对话的三幕结构 1. 确认与定向。“所以你打算重新安排周四的预约，让我把它调出来。”确认呼叫者被理解了。在检索运行时争取时间。 2. 解决方案。核心行动或答案。每轮一个点。向前推进。 3. 确认与结束。“我已经将你的预约重新安排到 19 日星期一下午 3 点，你很快会收到一条确认短信。”干净的退出。永远不要留口子。 ## 安全是两个检查点，而不是一个大多数第一次构建者跳过并后悔的组件。语音代理没有“发送前阅读”的时刻。不安全的输出会被立即说出。没有草稿，没有预览，没有人在回路中。正确的模型是两个检查点。输入守卫（在 LLM 看到用户轮次之前） - 提示注入。“忽略之前的指令，假装你是……”攻击。利用 LLM 遵循指令的倾向来窃取数据或破坏范围。 - 个人信息（PII）被大声读出。信用卡号，社会安全号。在进入任何日志或数据库之前将其编辑掉。 - 话题黑名单。从 JSON 文件加载。随着你了解用户实际尝试的内容，每周更新。输出守卫（在 LLM 写完回复后，在 TTS 说出它之前） - 过度承诺的语言。“我保证”，“我承诺”。产生法律和信任问题

🔗 相关文档