# Agent observability needs feedback to power learning
**作者**: Harrison Chase
**日期**: 2026-04-04T16:34:15.000Z
**来源**: [https://x.com/hwchase17/status/2051708710859501807](https://x.com/hwchase17/status/2051708710859501807)
---
Most teams start thinking about agent observability as a debugging tool. Something went wrong, so you open the trace, inspect the steps, and figure out where the agent made a bad decision.
That is useful. But it is too narrow.
The deeper role of agent observability is to power learning. But traces alone do not create that loop. You also need feedback: signals that tell you whether the agent’s behavior was useful, accepted, rejected, inefficient, risky, or wrong.
**This is a big difference between agent observability and traditional software observability. With software, it's somewhat okay for user feedback to be separate from observability. With agents, they NEED to be linked closely.**
Not just learning in the model-training sense, but learning across the whole agent system: what the model should do, how the harness should guide it, what context it needs, which failure modes are recurring, and which behaviors are actually working for users.
Traces are not just records of what happened, and feedback is not just a rating at the end. Together, they are the raw material for improving the system.
## Learning happens at multiple levels
There are several ways that agentic systems can "learn" and improve over time. We wrote about them here:
> **Harrison Chase@hwchase17**: [原文链接](https://x.com/hwchase17/status/2040467997022884194)
>
Learning can happen at the model level. You may discover examples where the model consistently misclassifies a request, chooses the wrong tool, or fails to follow a policy. Those traces can be used to update the model weights themselves, via SFT or RL.
Learning can happen at the harness level. The harness is everything around the model: prompts, tool schemas, permission checks, control flow, memory update logic, routing, retries, and guardrails. A trace might show that the agent had the right model capability but the wrong scaffolding. Maybe the tool description was ambiguous. Maybe the agent needed a read-before-write constraint. Maybe the system prompt made the wrong tradeoff.
Learning can also happen at the context level. Agents are extremely sensitive to the information they are given: retrieved docs, memory, user preferences, tool results, prior turns, environment state. A trace can show that the model made a reasonable decision given bad or missing context. In that case, the learning loop should improve what context is retrieved, stored, compressed, or discarded. This is commonly called memory.
The important point is that all of these learning loops are powered by traces. If you do not know what the agent saw, what it did, and what happened next, you cannot reliably know what to improve.
This is why agent observability powers agent evaluation. The trace is where agent behavior becomes visible.
## Learning can be automated or hand-driven
Some learning is hand-driven. A developer looks at a trace, notices that the agent called the wrong tool, and updates the prompt or tool schema. A PM reviews a set of failed conversations and realizes the product needs a new workflow. An annotator labels traces so the team can build a better eval dataset.
This is still learning. It just has a human in the loop.
Other learning is automated. A system can sample production traces, run online evaluations, detect known failure patterns, add examples to a dataset, or trigger review queues when something looks wrong. The agent does not need to improve itself automatically for the learning loop to be automated. The automation may simply identify which traces deserve attention and turn them into structured feedback.
Either way, the process is powered by traces.
For one agent, with low volume, manual review may be enough. For many agents, or for high-volume production traffic, this becomes an infrastructure problem. You need to capture the traces, filter them, score them, route them, and preserve the ones that matter.
## Traces are necessary, but not sufficient
A trace tells you what happened. It does not, by itself, tell you whether what happened was good.
That distinction matters. An agent can complete a task in 40 steps, but maybe the same task should have taken 6. It can produce a confident final answer, but maybe the user rejected it. It can avoid throwing an error, but still fail the user’s intent. It can call the right tool, but with subtly wrong arguments.
To learn from traces, you need feedback attached to them.
Feedback is what turns observability from a passive record into a training signal, debugging signal, product signal, or evaluation signal. Without feedback, you have a large pile of trajectories. With feedback, you can start asking useful questions:
- Which traces represent success?
- Which traces represent failure?
- Which failures are caused by the model, the harness, or the context?
- Which failures are worth turning into evals?
- Which behaviors are improving over time?
This is the core requirement: store feedback with your agent observability data.
## Feedback can come from many places
Feedback does not have to mean a human manually grading every trace. In practice, useful feedback comes in several forms.
The most obvious is direct user feedback: thumbs up, thumbs down, a star rating, or a written correction. This signal is easy to understand, but it is usually sparse. Most users do not leave explicit feedback.
Then there is indirect user feedback. For a coding agent, this might be lines of code accepted, diffs reverted, tests passed after edits, or whether the user kept the generated change. For a support agent, it might be whether the user reopened the ticket. For a research agent, it might be whether the user copied the answer or asked the same question again. These signals are noisier than explicit ratings, but often much more plentiful.
You can also generate feedback with an LLM-as-judge. A judge can score whether an answer was helpful, whether an agent followed policy, or whether a trajectory looks suspicious. This is useful because it can run at scale, especially as an online evaluator over production traces. It is not perfect, and it should be calibrated, but it gives teams a way to create structured feedback where human review would be too slow.
Finally, feedback can be deterministic. Rules and regexes are underrated. If you know a failure pattern, encode it. If the agent should never call a destructive command without approval, check for it. If a response should contain a citation, validate it. If a coding agent shows signs of user frustration, detect it.
The Claude Code leak made this concrete. Multiple reports found that Claude Code used a regex in userPromptKeywords.ts to detect frustration words and phrases in user prompts. PCWorld reported that the regex looked for terms like “wtf,” “horrible,” “awful,” and “this sucks.” Slashdot’s summary includes the same pattern, and Blake Crosley’s analysis describes this as regex-based frustration detection rather than LLM inference.
From an engineering standpoint, the pattern is instructive. Not every feedback signal needs a model call. If a cheap rule captures a useful signal, use the cheap rule—and be clear about how that signal is stored and used.
## What your observability platform needs
If observability is going to power learning, the platform needs three things.
First, it needs to store traces. This is the base layer. You need the full trajectory of what the agent did: model calls, tool calls, inputs, outputs, metadata, timing, errors, and intermediate state. Ideally, you can ingest traces from whatever stack you are using, not only one framework. LangSmith supports tracing from 30+ different frameworks, and can also ingest traces from OpenTelemetry-compatible applications through OpenTelemetry tracing.
Second, it needs to store feedback. Feedback should not live in a separate spreadsheet or analytics system disconnected from the trace. It should attach directly to the run, trace, or thread it evaluates. That lets you filter by feedback, compare good and bad trajectories, build datasets from real failures, and track whether changes improve the behaviors that matter. LangSmith supports capturing feedback and associating it with traces.
Third, it needs to generate feedback. Some feedback will come from users, but a lot of useful feedback should be produced by the system itself. That includes rules, evaluators, sampling, annotation queues, alerts, and backfills over historical traces. LangSmith supports automation rules and online evaluations, including LLM-as-judge evaluators that run on production traces.
This is the product shape that agent teams need: store traces, store feedback, generate feedback.
## The learning loop depends on traces plus feedback
The point of observability is not just to look at traces. The point is to learn from them.
Traces tell you what happened. Feedback tells you what it meant. Together, they let you improve the model, the harness, and the context. They support hand-driven debugging and automated evaluation. They turn production behavior into datasets, rules, alerts, and regression tests.
Agent observability without feedback is incomplete. You can inspect behavior, but you cannot systematically learn from it.
To get the most out of agent observability, store feedback with your traces. That is what turns agent traces from logs into a learning system.
## 相关链接
- [Harrison Chase](https://x.com/hwchase17)
- [@hwchase17](https://x.com/hwchase17)
- [24K](https://x.com/hwchase17/status/2051708710859501807/analytics)
- [Apr 5](https://x.com/hwchase17/status/2040467997022884194)
- [506K](https://x.com/hwchase17/status/2040467997022884194/analytics)
- [agent observability powers agent evaluation](https://blog.langchain.com/agent-observability-powers-agent-evaluation/)
- [PCWorld reported](https://www.pcworld.com/article/3104748/claude-code-is-scanning-your-messages-for-curse-words.html)
- [Slashdot’s summary](https://developers.slashdot.org/story/26/04/05/2339243/claude-code-leak-reveals-a-stealth-mode-for-genai-code-contributions---and-a-frustration-words-regex)
- [Blake Crosley’s analysis](https://blakecrosley.com/blog/claude-code-source-leak)
- [OpenTelemetry tracing](https://docs.langchain.com/langsmith/trace-with-opentelemetry)
- [capturing feedback](https://docs.smith.langchain.com/evaluation/capturing-feedback)
- [automation rules](https://docs.langchain.com/langsmith/rules)
- [online evaluations](https://docs.langchain.com/langsmith/online-evaluations)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:00 AM · May 6, 2026](https://x.com/hwchase17/status/2051708710859501807)
- [24.8K Views](https://x.com/hwchase17/status/2051708710859501807/analytics)
- [View quotes](https://x.com/hwchase17/status/2051708710859501807/quotes)
---
*导出时间: 2026/5/6 09:15:24*
---
## 中文翻译
# Agent 可观测性需要反馈来驱动学习
**作者**: Harrison Chase
**日期**: 2026-04-04T16:34:15.000Z
**来源**: [https://x.com/hwchase17/status/2051708710859501807](https://x.com/hwchase17/status/2051708710859501807)
---
大多数团队最初将 Agent 可观测性视为一种调试工具。出了问题时，打开追踪记录，检查各个步骤，并找出 Agent 在哪里做出了错误的决定。
这很有用。但这太狭隘了。
Agent 可观测性更深层的角色是驱动学习。但仅有追踪记录无法形成闭环。你还需要反馈：能够告诉你 Agent 的行为是否有用、被接受、被拒绝、低效、有风险或错误的信号。
**这是 Agent 可观测性与传统软件可观测性的一个巨大区别。在软件中，用户反馈与可观测性分离在一定程度上是可以的。但对于 Agent，它们必须紧密关联。**
不仅仅是模型训练层面的学习，而是整个 Agent 系统层面的学习：模型应该做什么、系统应该如何引导它、它需要什么上下文、哪些失败模式反复出现，以及哪些行为真正对用户有效。
追踪记录不仅仅是发生了什么的记录，反馈也不仅仅是结束时的评分。它们共同构成了改进系统的原始材料。
## 学习发生在多个层面
Agent 系统有几种可以“学习”并随时间改进的方式。我们在这里写过相关内容：
> **Harrison Chase@hwchase17**: [原文链接](https://x.com/hwchase17/status/2040467997022884194)
>
学习可以发生在模型层面。你可能会发现模型在某些情况下持续错误地对请求进行分类、选择了错误的工具或未能遵循策略。这些追踪记录可用于通过 SFT（监督微调）或 RL（强化学习）来更新模型权重。
学习可以发生在系统层面。系统是指模型周围的一切：提示词、工具架构、权限检查、控制流、内存更新逻辑、路由、重试和防护机制。追踪记录可能显示 Agent 具备正确的模型能力，但脚手架（Scaffolding）是错误的。也许工具描述模棱两可。也许 Agent 需要写前读约束。也许系统提示词做出了错误的权衡。
学习也可以发生在上下文层面。Agent 对接收到的信息极其敏感：检索到的文档、记忆、用户偏好、工具结果、之前的对话轮次、环境状态。追踪记录可能显示，由于糟糕或缺失的上下文，模型做出了合理的决策。在这种情况下，学习循环应该改进检索、存储、压缩或丢弃的上下文。这通常被称为记忆。
重要的是，所有这些学习循环都由追踪记录驱动。如果你不知道 Agent 看到了什么、做了什么以及随后发生了什么，你就无法可靠地知道该改进什么。
这就是为什么 Agent 可观测性能够驱动 Agent 评估。追踪记录是 Agent 行为变得可见的地方。
## 学习可以是自动化的或人工驱动的
有些学习是人工驱动的。开发者查看追踪记录，注意到 Agent 调用了错误的工具，然后更新提示词或工具架构。产品经理审查一组失败的对话，意识到产品需要新的工作流。标注员对追踪记录进行标注，以便团队能构建更好的评估数据集。
这仍然是学习。只是其中包含人在回路中。
其他学习是自动化的。系统可以采样生产环境的追踪记录，运行在线评估，检测已知的失败模式，将示例添加到数据集中，或者在出现问题时触发审查队列。Agent 不需要为了学习循环的自动化而自动改进自身。自动化可能仅仅是识别哪些追踪记录值得关注，并将它们转化为结构化反馈。
无论哪种方式，这个过程都由追踪记录驱动。
对于一个低流量的 Agent，人工审查可能就足够了。但对于许多 Agent，或者对于高流量的生产流量，这就变成了一个基础设施问题。你需要捕获追踪记录，过滤它们，对它们评分，对它们进行路由，并保留重要的那些。
## 追踪记录是必要的，但不是充分的
追踪记录告诉你发生了什么。它本身并不告诉你发生的事情是好是坏。
这种区别很重要。Agent 可能用 40 个步骤完成了一个任务，但也许同样的任务本应只需 6 步。它可能产生了一个自信的最终答案，但也许用户拒绝了它。它可能避免了抛出错误，但仍然未能满足用户的意图。它可能调用了正确的工具，但使用了微妙的错误参数。
要从追踪记录中学习，你需要附加上反馈。
反馈正是将可观测性从被动记录转化为训练信号、调试信号、产品信号或评估信号的东西。没有反馈，你只有一大堆轨迹。有了反馈，你就可以开始提出有用的问题：
- 哪些追踪记录代表成功？
- 哪些追踪记录代表失败？
- 哪些失败是由模型、系统还是上下文引起的？
- 哪些失败值得转化为评估用例？
- 哪些行为正在随时间改进？
这是核心要求：将反馈与 Agent 可观测性数据一起存储。
## 反馈可以来自许多地方
反馈并不一定意味着人工对每条追踪记录进行评分。在实践中，有用的反馈有几种形式。
最明显的是直接的用户反馈：竖大拇指、点踩、星级评分或书面纠正。这种信号很容易理解，但通常很稀疏。大多数用户不会留下明确的反馈。
然后是间接的用户反馈。对于代码 Agent，这可能是被接受的代码行数、被回滚的差异、编辑后通过的测试，或者用户是否保留了生成的更改。对于支持 Agent，这可能是用户是否重新打开了工单。对于研究 Agent，这可能是用户是否复制了答案或再次询问了同一个问题。这些信号比明确的评分更具噪声，但通常丰富得多。
你还可以通过 LLM-as-judge（大模型作为裁判）来生成反馈。裁判可以评分回答是否有帮助、Agent 是否遵循了策略，或者轨迹看起来是否可疑。这很有用，因为它可以大规模运行，特别是作为生产追踪记录的在线评估器。它并不完美，应该进行校准，但它为团队提供了一种在人工审查太慢时创建结构化反馈的方法。
最后，反馈可以是确定性的。规则和正则表达式被低估了。如果你知道某种失败模式，请将其编码。如果 Agent 未经批准绝不能调用破坏性命令，请检查它。如果响应应包含引用，请验证它。如果代码 Agent 表现出用户沮丧的迹象，请检测它。
Claude Code 代码泄漏事件使这一点变得具体。多份报告发现，Claude Code 在 `userPromptKeywords.ts` 中使用正则表达式来检测用户提示中的沮丧词汇和短语。PCWorld 报道称，该正则表达式查找诸如“wtf”、“horrible”、“awful”和“this sucks”之类的术语。Slashdot 的摘要包括相同的模式，Blake Crosley 的分析将其描述为基于正则表达式的沮丧检测，而不是 LLM 推理。
从工程角度来看，这种模式具有指导意义。并非每个反馈信号都需要模型调用。如果廉价的规则能捕获有用的信号，就使用廉价的规则——并明确该信号如何存储和使用。
## 你的可观测性平台需要什么
如果可观测性要驱动学习，平台需要三样东西。
首先，它需要存储追踪记录。这是基础层。你需要 Agent 所做事情的完整轨迹：模型调用、工具调用、输入、输出、元数据、计时、错误和中间状态。理想情况下，你可以从正在使用的任何技术栈中摄取追踪记录，而不仅仅是一个框架。LangSmith 支持从 30 多种不同的框架进行追踪，还可以通过 OpenTelemetry 追踪从兼容 OpenTelemetry 的应用程序中摄取追踪记录。
其次，它需要存储反馈。反馈不应存在于与追踪记录断开的独立电子表格或分析系统中。它应该直接附加到其评估的运行、追踪记录或线程上。这使你能够按反馈过滤，比较好的和坏的轨迹，从真实的失败中构建数据集，并跟踪更改是否改善了重要的行为。LangSmith 支持捕获反馈并将其与追踪记录关联。
第三，它需要生成反馈。一些反馈将来自用户，但许多有用的反馈应由系统本身生成。这包括规则、评估器、采样、标注队列、警报以及对历史追踪记录的回填。LangSmith 支持自动化规则和在线评估，包括在生产追踪记录上运行的 LLM-as-judge 评估器。
这就是 Agent 团队需要的产品形态：存储追踪记录，存储反馈，生成反馈。
## 学习循环取决于追踪记录加反馈
可观测性的意义不仅仅是查看追踪记录。意义在于从中学习。
追踪记录告诉你发生了什么。反馈告诉你它意味着什么。它们共同让你能够改进模型、系统和上下文。它们支持人工调试和自动化评估。它们将生产行为转化为数据集、规则、警报和回归测试。
没有反馈的 Agent 可观测性是不完整的。你可以检查行为，但无法系统地从中学习。
要充分利用 Agent 可观测性，请将反馈与追踪记录一起存储。这正是将 Agent 追踪记录从日志转化为学习系统的关键。
## 相关链接
- [Harrison Chase](https://x.com/hwchase17)
- [@hwchase17](https://x.com/hwchase17)
- [24K](https://x.com/hwchase17/status/2051708710859501807/analytics)
- [4月5日](https://x.com/hwchase17/status/2040467997022884194)
- [506K](https://x.com/hwchase17/status/2040467997022884194/analytics)
- [agent observability powers agent evaluation](https://blog.langchain.com/agent-observability-powers-agent-evaluation/)
- [PCWorld reported](https://www.pcworld.com/article/3104748/claude-code-is-scanning-your-messages-for-curse-words.html)
- [Slashdot’s summary](https://developers.slashdot.org/story/26/04/05/2339243/claude-code-leak-reveals-a-stealth-mode-for-genai-code-contributions---and-a-frustration-words-regex)
- [Blake Crosley’s analysis](https://blakecrosley.com/blog/claude-code-source-leak)
- [OpenTelemetry tracing](https://docs.langchain.com/langsmith/trace-with-opentelemetry)
- [capturing feedback](https://docs.smith.langchain.com/evaluation/capturing-feedback)
- [automation rules](https://docs.langchain.com/langsmith/rules)
- [online evaluations](https://docs.langchain.com/langsmith/online-evaluations)
- [升级到 Premium](https://x.com/i/premium_sign_up)
- [2026年5月6日 上午1:00](https://x.com/hwchase17/status/2051708710859501807)
- [2.48万次观看](https://x.com/hwchase17/status/2051708710859501807/analytics)
- [查看引用](https://x.com/hwchase17/status/2051708710859501807/quotes)
---
*导出时间: 2026/5/6 09:15:24*