构建客户支持语音代理实战指南

✍ Sumanth🕐 2026-07-17📦 10.1 KB 🟢 已读 𝕏 文章列表

文章介绍了如何使用 Telnyx AI Assistant Builder 构建生产级客户支持语音代理。教程涵盖了低延迟音频流、中断处理等工程挑战，并演示了通过 FastAPI Webhook 注入实时业务上下文（如排队时间、系统状态）的方法，无需自行搭建 STT、LLM 和 TTS 管道。

# How to build a Customer Support Voice Agent **作者**: Sumanth **日期**: 2026-07-13T14:13:07.000Z **来源**: [https://x.com/Sumanth_077/status/2076671269391696042](https://x.com/Sumanth_077/status/2076671269391696042) --- ![Image](/doc/3091/image/img_c88a98553802.jpg) Voice AI agent demo often looks simple: Caller → Speech-to-Text → LLM → Text-to-Speech → Caller In production systems this looks very different. Between those components lies the real engineering involved in low-latency audio streaming, turn detection, interruption handling, telephony codecs, webhook orchestration, and voicemail detection. In this tutorial, we'll build a customer support voice agent without assembling that pipeline yourself. Using Telnyx AI Assistant Builder, you'll configure the entire voice stack from a portal and write a small FastAPI webhook that injects live business context into every call. ![Image](/doc/3091/image/img_4b43e7e3b3f3.jpg) ## What a working customer support voice agent actually requires The same core building blocks, regardless of the tools you choose: 1. Sub-second round-trip latency. STT, the language model, TTS, and the telephony layer all have to fit inside about one second, or the conversation feels awkward. 2. Grounded answers. The agent has to speak from a knowledge base, not from vibes. Hallucinated policies hurt more than they help. 3. Live context per call. Queue wait, incidents in the last hour, whether the branch is open right now. None of that can live in a static prompt. 4. Graceful handoff. Some questions do not get answered by AI. The agent needs to know when to stop trying and pass the caller to a human, cleanly. 5. Testable without a phone number. You should be able to iterate on the agent's instructions without dialling a number and burning a call minute every time. This article covers all of them. Instead of assembling providers, you configure one platform in a portal and write code only for the context the assistant needs when a call begins. ## The Telnyx AI Assistant Builder Telnyx's AI Assistant Builder is a portal for creating a voice assistant. You give it a name, pick a language model, pick a voice, pick a transcription provider, fill in the Instructions field (the system prompt that carries the assistant's persona, response format, and knowledge base), and attach a phone number. The platform handles the rest: - STT (Telnyx native, Deepgram, or Azure) - The language model (Kimi K2.5 on Telnyx infrastructure with no API key, GPT-4o via OpenAI, and several others) - TTS (Telnyx native, ElevenLabs, AWS, Azure, Inworld) - The phone number and PSTN routing - Conversation history and playback - An embeddable browser widget that connects to the same assistant over WebRTC ![Image](/doc/3091/image/img_3b61b452a5c2.jpg) The Instructions field is where the assistant's persona lives. For the tutorial this article follows, the persona is a fictional retail bank: > You are a customer support agent for a full-service retail bank serving personal and business customers. > Keep all responses under 2 sentences -- callers are listening, not reading. > Speak calmly and professionally. If a question is not covered in the knowledge base below, acknowledge it warmly and offer to transfer the caller to a specialist. > System status: {{system_status}} > Active incidents: {{todays_incidents}} > Queue wait: {{current_queue_wait}} > Business hours: {{business_hours}} > Support email: {{support_email}} > --- KNOWLEDGE BASE --- > ACCOUNTS > We offer three personal account tiers. The Everyday Checking account carries no > minimum balance, includes a free debit card and online bill pay, and reimburses > up to $10 in ATM fees each month... ## Dynamic Variables The Instructions field holds information that stays the same across every call. Information that changes over time, such as the current queue wait time, active incidents, or whether a branch is open, is handled by Dynamic Variables. Anywhere in the Instructions field you write {{variable_name}}, Telnyx replaces it with a value when the call begins. Those values come from a webhook you host. The flow is simple: 1. Caller dials the number 2. Telnyx sends an HTTP POST to your webhook URL 3. Your webhook returns a JSON object of key-value pairs 4. Telnyx substitutes them into the system prompt 5. The model reads the substituted prompt for the rest of the call For a support agent, every call starts with fresh values for queue wait times, active incidents, business hours, and anything else you want to keep current. The webhook is just a single FastAPI route: ``` from fastapi import FastAPI, HTTPException, Request from fastapi.responses import JSONResponse import telnyx, os telnyx_client = telnyx.Telnyx( api_key=os.getenv("TELNYX_API_KEY"), public_key=os.getenv("TELNYX_PUBLIC_KEY"), ) app = FastAPI() def get_live_context() -> dict: return { "system_status": "All systems operational", "current_queue_wait": "Under 2 minutes", "business_hours": "Monday to Friday, 9AM to 5PM EST; Saturday, 9AM to 1PM EST", "todays_incidents": "None", "support_email": "support@bank.com", } @app.post("/webhooks/dynamic-variables") async def dynamic_variables(request: Request) -> JSONResponse: raw_body = await request.body() try: telnyx_client.webhooks.unwrap( raw_body.decode("utf-8"), dict(request.headers) ) except Exception: raise HTTPException(status_code=401, detail="Invalid webhook signature") return JSONResponse(get_live_context()) ``` That is the whole webhook. Roughly fifteen effective lines of code. No STT integration. No language model API key. No TTS work. No telephony. Just the piece that has to change per call. ## Full Workflow Here is the full flow, from dialed number to answered question: 1. Caller dials the Telnyx phone number. Telnyx routes the call to the AI Assistant configured for that number. 2. Telnyx fires the Dynamic Variables webhook. POST to your FastAPI endpoint with call metadata in the body. 3. The webhook verifies the signature and returns a context dict. Live queue wait, incidents, business hours. 4. Telnyx substitutes those values into the Instructions field. The {{variable}} placeholders become their current values. 5. The model reads the fully-substituted prompt. Persona, knowledge base, and live context in one system message. 6. STT, language model, and TTS run on Telnyx infrastructure. No hops to external providers. The audio stays inside one platform. 7. The conversation happens. Caller asks about card fraud, wire transfer times, or the current queue wait. The model answers from the KB and the live context. ![Image](/doc/3091/image/img_381e47b42542.jpg) ## Getting the agent running Clone the repo: ``` git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git cd Hands-On-AI-Engineering/voice_apps/customer_support_voice_agent uv venv source .venv/bin/activate uv pip install -e . ``` Set up credentials: ``` cp .env.example .env # Edit .env and update with with your TELNYX_API_KEY and TELNYX_PUBLIC_KEY ``` Run the webhook server: ``` python main.py ``` In a separate terminal, expose it via ngrok so Telnyx can reach it during development: ``` ngrok http 8000 ``` Copy the ngrok HTTPS URL, then open the assistant in the Telnyx portal and paste it Dynamic Variables webhook URL field: ![Image](/doc/3091/image/img_40489214153c.jpg) Three ways to test the agent: - Portal test. Open the assistant in the Telnyx portal and click Test. A browser-based call interface opens. No phone required. - Phone call. Dial the number you attached to the assistant. Real PSTN, real audio, real telephony codecs. - Browser widget. Open demo.html from the repo in any browser. It embeds the Telnyx AI Agent widget, which connects to the assistant over WebRTC. No server required. Sample prompts that shows what this agent can handle: - "I think my debit card was stolen. What should I do?" Grounded answer from the KB, with the 24-hour hotline number and the app-based card freeze option. - "How long does a wire transfer take, and what does it cost?" Same source, same call. - "How busy are you right now?" Answered from the live queue wait injected via the webhook. The full setup and code is available here. ## What you get from this Voice AI used to mean stitching together a speech-to-text service, an LLM, text-to-speech, and then writing the code to keep everything in sync. Every new integration added more moving parts to manage. With Telnyx, all of that is handled for you. The only code left is the webhook that answers one question: What does the agent need to know right now? That lets you focus on your business logic instead of the infrastructure. Checkout the Repo: https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent ## 相关链接 - [Sumanth](https://x.com/Sumanth_077) - [@Sumanth_077](https://x.com/Sumanth_077) - [19K](https://x.com/Sumanth_077/status/2076671269391696042/analytics) - [Telnyx's AI Assistant Builder](https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant) - [portal](https://portal.telnyx.com/#/ai/assistants/edit/assistant-7fdd43fa-b002-4d18-956f-0f86d4e96433) - [here](https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/voice_apps/customer_support_voice_agent) - [https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent](https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [10:13 PM · Jul 13, 2026](https://x.com/Sumanth_077/status/2076671269391696042) - [19.3K Views](https://x.com/Sumanth_077/status/2076671269391696042/analytics) - [View quotes](https://x.com/Sumanth_077/status/2076671269391696042/quotes) --- *导出时间: 2026/7/17 11:26:32* --- ## 中文翻译 # 如何构建客户支持语音代理 **作者**: Sumanth **日期**: 2026-07-13T14:13:07.000Z **来源**: [https://x.com/Sumanth_077/status/2076671269391696042](https://x.com/Sumanth_077/status/2076671269391696042) --- ![Image](/doc/3091/image/img_c88a98553802.jpg) 语音 AI 代理的演示通常看起来很简单：呼叫者 → 语音转文字 → 大语言模型 (LLM) → 文字转语音 → 呼叫者在生产环境中，情况看起来截然不同。在这些组件之间，涉及着低延迟音频流传输、回合检测、中断处理、电话编解码器、Webhook 编排和语音信箱检测等真正的工程问题。在本教程中，我们将构建一个客户支持语音代理，而无需自己去组装这条流水线。使用 Telnyx AI Assistant Builder，您可以在一个门户中配置整个语音技术栈，并编写一个小型的 FastAPI webhook，将实时的业务上下文注入到每个通话中。 ![Image](/doc/3091/image/img_4b43e7e3b3f3.jpg) ## 一个能正常工作的客户支持语音代理真正需要什么无论您选择哪种工具，核心构建模块都是一样的： 1. 亚秒级往返延迟。语音转文字 (STT)、语言模型、文字转语音 (TTS) 和电话层都必须容纳在大约一秒的时间内，否则对话会让人感到尴尬。 2. 有依据的答案。代理必须基于知识库发言，而不是凭感觉。凭空臆造的政策只会带来伤害，而不是帮助。 3. 每次通话的实时上下文。排队等待时间、过去一小时的事件、分支机构当前是否营业。这些都无法存在于静态提示词中。 4. 优雅的转接。有些问题 AI 无法回答。代理需要知道何时停止尝试，并将呼叫者干净利落地转接给人工客服。 5. 无需电话号码即可测试。您应该能够迭代优化代理的指令，而无需每次都拨打电话并消耗通话时长。本文涵盖了所有这些内容。无需组装多个提供商，您只需在门户中配置一个平台，并仅需编写代码来处理助手在通话开始时所需的上下文。 ## Telnyx AI Assistant Builder Telnyx 的 AI Assistant Builder 是一个用于创建语音助手的门户。您给它起个名字，选择一个语言模型，选择一种声音，选择一个转录提供商，填写“指令”字段（即承载助手人设、响应格式和知识库的系统提示词），并绑定一个电话号码。平台负责处理其余的工作： - STT（Telnyx 原生、Deepgram 或 Azure） - 语言模型（Telnyx 基础设施上的 Kimi K2.5，无需 API 密钥；通过 OpenAI 的 GPT-4o，以及其他几种） - TTS（Telnyx 原生、ElevenLabs、AWS、Azure、Inworld） - 电话号码和 PSTN 路由 - 对话历史记录和回放 - 一个可嵌入的浏览器小部件，通过 WebRTC 连接到同一个助手 ![Image](/doc/3091/image/img_3b61b452a5c2.jpg) “指令”字段是助手人设的所在地。对于本文遵循的教程，人设是一家虚构的零售银行： > 您是一家服务于个人和企业客户的全方位零售银行的客户支持代理。 > 保持所有回复在 2 句话以内——呼叫者是在听，不是在读。 > 说话要冷静且专业。如果以下知识库未涵盖某个问题，请热情地表示知晓，并提出将呼叫者转接给专家。 > 系统状态：{{system_status}} > 当前事件：{{todays_incidents}} > 排队等待：{{current_queue_wait}} > 营业时间：{{business_hours}} > 支持邮箱：{{support_email}} > --- 知识库 --- > 账户 > 我们提供三个个人账户层级。日常支票账户没有最低余额要求，包括免费的借记卡和在线账单支付，并且每月补偿高达 10 美元的 ATM 手续费... ## 动态变量 “指令”字段包含在每次通话中保持不变的信息。随时间变化的信息，例如当前的排队等待时间、当前事件或分支机构是否营业，由动态变量处理。在“指令”字段的任何位置编写 {{variable_name}}，Telnyx 会在通话开始时将其替换为一个值。这些值来自您托管的 webhook。流程很简单： 1. 呼叫者拨打电话号码 2. Telnyx 向您的 webhook URL 发送 HTTP POST 请求 3. 您的 webhook 返回一个键值对 JSON 对象 4. Telnyx 将这些值替换到系统提示词中 5. 模型在通话的剩余时间读取替换后的提示词对于支持代理，每次通话开始时，排队等待时间、当前事件、营业时间以及您希望保持最新的任何其他信息都会更新为最新值。 Webhook 只是一个单一的 FastAPI 路由： ``` from fastapi import FastAPI, HTTPException, Request from fastapi.responses import JSONResponse import telnyx, os telnyx_client = telnyx.Telnyx( api_key=os.getenv("TELNYX_API_KEY"), public_key=os.getenv("TELNYX_PUBLIC_KEY"), ) app = FastAPI() def get_live_context() -> dict: return { "system_status": "所有系统运行正常", "current_queue_wait": "不到 2 分钟", "business_hours": "周一至周五，美国东部时间上午 9 点至下午 5 点；周六，美国东部时间上午 9 点至下午 1 点", "todays_incidents": "无", "support_email": "support@bank.com", } @app.post("/webhooks/dynamic-variables") async def dynamic_variables(request: Request) -> JSONResponse: raw_body = await request.body() try: telnyx_client.webhooks.unwrap( raw_body.decode("utf-8"), dict(request.headers) ) except Exception: raise HTTPException(status_code=401, detail="Invalid webhook signature") return JSONResponse(get_live_context()) ``` 这就是整个 webhook。大约十五行有效代码。没有 STT 集成。没有语言模型 API 密钥。没有 TTS 工作。没有电话通信。只有每次通话必须更改的那部分。 ## 完整工作流从拨号到回答问题的完整流程如下： 1. 呼叫者拨打 Telnyx 电话号码。Telnyx 将呼叫路由到为该号码配置的 AI 助手。 2. Telnyx 触发动态变量 webhook。向您的 FastAPI 端点发送 POST 请求，请求正文中包含通话元数据。 3. Webhook 验证签名并返回上下文字典。实时排队等待时间、事件、营业时间。 4. Telnyx 将这些值替换到“指令”字段中。{{variable}} 占位符变为它们的当前值。 5. 模型读取完全替换后的提示词。人设、知识库和实时上下文都包含在一个系统消息中。 6. STT、语言模型和 TTS 在 Telnyx 基础设施上运行。无需跳转到外部提供商。音频保留在一个平台内。 7. 对话进行。呼叫者询问关于信用卡欺诈、电汇时间或当前排队等待时间的问题。模型根据知识库和实时上下文进行回答。 ![Image](/doc/3091/image/img_381e47b42542.jpg) ## 让代理运行起来克隆仓库： ``` git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git cd Hands-On-AI-Engineering/voice_apps/customer_support_voice_agent uv venv source .venv/bin/activate uv pip install -e . ``` 设置凭证： ``` cp .env.example .env # 编辑 .env 并更新您的 TELNYX_API_KEY 和 TELNYX_PUBLIC_KEY ``` 运行 webhook 服务器： ``` python main.py ``` 在单独的终端中，通过 ngrok 暴露它，以便 Telnyx 在开发期间可以访问它： ``` ngrok http 8000 ``` 复制 ngrok HTTPS URL，然后在 Telnyx 门户中打开助手，将其粘贴到动态变量 webhook URL 字段中： ![Image](/doc/3091/image/img_40489214153c.jpg) 三种测试代理的方法： - 门户测试。在 Telnyx 门户中打开助手并点击测试。会打开一个基于浏览器的通话界面。无需电话。 - 电话呼叫。拨打您绑定到助手的号码。真实的 PSTN、真实的音频、真实的电话编解码器。 - 浏览器小部件。在任何浏览器中打开仓库中的 demo.html。它嵌入了 Telnyx AI Agent 小部件，通过 WebRTC 连接到助手。无需服务器。展示此代理可以处理功能的示例提示词： - “我认为我的借记卡被盗了。我该怎么办？”根据知识库给出有依据的答案，提供 24 小时热线号码和基于应用程序的卡片冻结选项。 - “电汇需要多长时间，费用是多少？”同样的来源，同一个电话。 - “你们现在有多忙？”根据通过 webhook 注入的实时排队等待时间回答。完整的设置和代码可在此处获取。 ## 您能从中获得什么语音 AI 过去意味着将语音转文字服务、LLM、文字转语音拼接在一起，然后编写代码以保持所有内容同步。每次新的集成都会增加更多需要管理的活动部件。使用 Telnyx，所有这些都为您处理好了。剩下的唯一代码就是回答一个问题的 webhook：代理现在需要知道什么？这使您可以专注于业务逻辑而不是基础设施。查看仓库：https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent ## 相关链接 - [Sumanth](https://x.com/Sumanth_077) - [@Sumanth_077](https://x.com/Sumanth_077) - [19K](https://x.com/Sumanth_077/status/2076671269391696042/analytics) - [Telnyx's AI Assistant Builder](https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant) - [portal](https://portal.telnyx.com/#/ai/assistants/edit/assistant-7fdd43fa-b002-4d18-956f-0f86d4e96433) - [here](https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/voice_apps/customer_support_voice_agent) - [https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent](https://github.com/Sumanth077/Hands-On-AI-Engineering/tree/main/audio/customer_support_voice_agent) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [10:13 PM · Jul 13, 2026](https://x.com/Sumanth_077/status/2076671269391696042) - [19.3K Views](https://x.com/Sumanth_077/status/2076671269391696042/analytics) - [View quotes](https://x.com/Sumanth_077/status/2076671269391696042/quotes) --- *导出时间: 2026/7/17 11:26:32*

🔗 相关文档