# start ugly, write evals anyway.
**作者**: hrushikesh
**日期**: 2026-03-10T16:36:51.000Z
**来源**: [https://x.com/Hrushikeshhhh/status/2099590015336808865](https://x.com/Hrushikeshhhh/status/2099590015336808865)
---
TLDR; understanding the importance of evals, and how they can make yours and your agent life easier without spending exponential tokens and time to complete a task.
pre -
Agent = Model + Harness.
- Model - the one thats brings the intelligence, capability to think and act.
- Harness - the one that lets the model act by giving tools, state, memory, envs, etc..
imo there are - model evals, and harness evals. and this article talks more about the harness eval, which helps us understand - does this harness support agent to to do what it should be doing.
im assuming you know what is a harness, if not, read this -
> **Viv@Vtrivedy10**: [原文链接](https://x.com/Vtrivedy10/status/2031408954517971368)
>
--
## what is eval, compared to tests for sdlc
back when we were building our search agent, who can search across of your entire data estate and point you to right things or possibly get you an answer which you are looking for -- we struggled a lot with getting the accurate responses, as the models were not that great to do better tool calling and also writing better markdown and html (this is early 2025).
because this was very new for teams who are now building agentic systems (moving from standard SDLC to ADLC), but assessing/evaluating these systems didnt have a playbook.
why, you may ask! the non determinism enters the request lifecycle:
- the context keeps changing in the background, so does the thinking, context window, exploration (tool calls), and understanding of data results different tokens.
there is a beautiful writing on why is evals not equals tests here --
> **Ankur Goyal@ankrgyl**: [原文链接](https://x.com/ankrgyl/status/2084667136618762550)
>
TLDR of above post;
- Tests = does it work?
- Evals = what can it do?
- Mature Evals = what should it be doing?
> not all harness are the same, can be managed or open source - which can then be of different structure based on the needs.
before metrics, get the nouns straight. this saved us a lot of arguing in review:
- task - one test case. input + starting state + what counts as success.
- trial - one run of that task. you run many, because the same task passes today and fails tomorrow.
- transcript (trace / trajectory) - the full record of a trial. reasoning, tool calls, intermediate results, tokens.
- outcome - the end state, not the final message. the agent saying "i created the connector" is transcript. the connector actually existing is outcome.
- grader - the logic that scores it. code, llm-as-judge, or a human. one task can have many.
- eval harness - the infra that runs all of this. different from the agent harness. yes, confusing. two harnesses.
- suite - a bunch of tasks that share a goal.
take a pov of junior engineer and is given an engineering task - yes, you being the senior engineer you give him access to laptop, saas tools, notebook, internet, etc.. and ask him to finish the task.
the junior engg has the capability of using these tools + internet can gather information and complete the task in 4 days not 1 day. the harness can help j engg do more work but "more work doesnt mean its getting better."
in a similar way, harness helps agent do more things but doesnt mean its getting better in any sense.
when you give a new tool/notebook, to help structure things - did it worsen the quality of work, failures, time? how would you track? - evals.
--
## how to approach evals
ive seen most teams struggle with the cold start problem with evals, mainly for - what are good metrics to evaluate/assess an agent, and how can i build them.
the cold start is smaller than people think. you do not need 500 tasks. 20-50 tasks pulled from real failures/hand written is a fine start. early on, every change has a big, obvious effect, so a small sample is enough to see it. the tasks get harder to write the longer you wait - early on your product requirements are your test cases, later you are reverse engineering success criteria out of a live system.
where to get those 20-50: whatever you already check by hand before a release, plus your bug tracker, plus the support queue. real complaints make better tasks than synthetic ones.
two things that quietly decide whether your eval is worth anything:
write tasks two people would grade the same way. if two domain experts read the task and disagree on pass/fail, that ambiguity shows up as noise in your metric forever. also write a reference solution for each task - a known-good output that passes all graders. it proves the task is solvable and that your graders arent broken.
build the negative half. one sided evals create one sided agents. if you only test "did it search when it should", you ship an agent that searches for everything. we felt this hard with the search agent - undertriggering and overtriggering are the same bug with opposite signs, and you only catch it if the suite has both kinds of task.
if im starting on evals today, i will only build these 2 metrics to gain confidence in the agent:
1. task completion
2. correctness
was agent able to complete the task? yes! is that the right answer? yes!
and you are good for internal testing, not yet ready for the production.
--
## grade what it produced, not the path it took
the instinct is to assert the exact sequence of tool calls. dont. agents keep finding valid approaches you didnt think of, and every brittle path assertion punishes that. path assertions also couple your eval to internal implementation details of the harness, so you cant refactor without breaking your own tests.
outcome grading is your gate. transcript reading is your debugger.
one more: build in partial credit. an agent that identified the problem and verified the user but failed the last step is meaningfully better than one that died at turn 2. a single binary hides that entire continuum, and the continuum is where your improvement is.
--
## then watch the trajectories
you will now see the traces flowing from internal testing, and you start monitoring the trajectories - which will show you the real insights into thinking, tool calls and can help you understand the behaviour of the agent for variety of tasks.
the signal you might want to capture and evaluate are:
- distraction - when agent is off track, does it align itself on the task or goes haywire.
- tokens spent - does it even need to spend millions of tokens to complete the task or can be done in lesser as well.
- latency - the time agent takes to complete a task. mainly, when its customer facing, there is not a lot of attention and time customer spends waiting for the agent to say i cant help!
- context bloat - how much unnecessary information is flowing into the context window.
- consistency - same task, same conditions, five runs. same answer? this one surprised me the most.
that last one deserves its own section.
--
## your eval is also a thing that can be broken
this is the part i underrated for the longest time.
a low score is not automatically the agent's fault. it can be the grader, the task spec, or the harness.
a good rule: 0% pass across many trials usually means a broken task, not an incapable agent. and failures should feel fair - you should be able to look at a failed transcript and say yeah, that was on the agent.
so, read the transcripts. not a sample after a big release, weekly. it is the single highest leverage habit in this whole thing and the one everyone skips because it doesnt produce a number.
same applies to llm-as-judge. it needs its own calibration set against human grading, its own version history, and a disagreement analysis. give the judge an explicit way out ("return unknown if you dont have enough information") so it stops hallucinating a verdict. and grade one dimension per judge instead of one judge grading six dimensions - much cleaner signal.
> worth budgeting for too - judging in production tends to run somewhere around 30-50% of your inference cost.
--
## evals need to evolve
one interesting learning i had over the time building and maintaining both the agent and evals is, todays criteria of metrics doesnt makes sense tomorrow for a better model/harness. evals need to evolve.
we started with search agent, but as both models and harness are becoming more capable, it turned out to be a generic agent which can perform actions, setup data connectors, generate presentation ready artifacts, and more - the evals which were evaluating the search tasks doesnt do a good job anymore. so we evolved.
> all new and successful patterns needs to feed back into evals. can give you regression and can-be-improved insights.
then hill climb.
once you crack all these for you agent, you are ready to hill climb with these evals, and [@Vtrivedy10](https://x.com/@Vtrivedy10) did a great work explaning how you can do it --
> **Viv@Vtrivedy10**: [原文链接](https://x.com/Vtrivedy10/status/2041927488918413589)
>
as you go deeper you will realise you need an eval to evaluate your evals. thanks for reading!
## 相关链接
- [hrushikesh](https://x.com/Hrushikeshhhh)
- [@Hrushikeshhhh](https://x.com/Hrushikeshhhh)
- [10K](https://x.com/Hrushikeshhhh/status/2099590015336808865/analytics)
- [Mar 11](https://x.com/Vtrivedy10/status/2031408954517971368)
- [825K](https://x.com/Vtrivedy10/status/2031408954517971368/analytics)
- [Aug 4](https://x.com/ankrgyl/status/2084667136618762550)
- [33K](https://x.com/ankrgyl/status/2084667136618762550/analytics)
- [@Vtrivedy10](https://x.com/@Vtrivedy10)
- [Apr 9](https://x.com/Vtrivedy10/status/2041927488918413589)
- [441K](https://x.com/Vtrivedy10/status/2041927488918413589/analytics)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:04 AM · Sep 15, 2026](https://x.com/Hrushikeshhhh/status/2099590015336808865)
- [10.8K Views](https://x.com/Hrushikeshhhh/status/2099590015336808865/analytics)
- [View quotes](https://x.com/Hrushikeshhhh/status/2099590015336808865/quotes)
---
*导出时间: 2026/9/15 13:28:56*
---
## 中文翻译
# 从丑陋的开始，但无论如何要写评估。
**作者**：hrushikesh
**日期**：2026-03-10T16:36:51.000Z
**来源**：[https://x.com/Hrushikeshhhh/status/2099590015336808865](https://x.com/Hrushikeshhhh/status/2099590015336808865)
---
TLDR；理解评估的重要性，以及它们如何让你和你的智能体的生活更轻松，而不必花费指数级的 token 和时间去完成一个任务。
前置概念 -
Agent（智能体）= 模型 + 框架（Harness）。
- 模型 - 提供智能、思考和行动能力的部分。
- 框架 - 通过提供工具、状态、记忆、环境等让模型能够行动的部分。
在我看来，评估分为 - 模型评估和框架评估。本文更多讨论的是框架评估，它帮助我们理解 - 这个框架是否支持智能体去做它应该做的事情。
我假设你知道什么是框架，如果不知道，请阅读这篇 -
> **Viv@Vtrivedy10**: [原文链接](https://x.com/Vtrivedy10/status/2031408954517971368)
>
--
## 什么是评估，与软件开发生命周期（SDLC）中测试的对比
回想当初我们构建搜索智能体时——它可以跨你的整个数据资产进行搜索，为你指出正确的内容，或者可能给你一个你正在寻找的答案——我们在获得准确响应方面遇到了很大困难，因为当时的模型在工具调用以及编写更好的 markdown 和 html 方面还不够好（这是 2025 年初）。
因为这对现在正在构建智能体系统的团队来说是非常新的事物（从标准 SDLC 转向 ADLC），但评估这些系统并没有现成的 playbook。
你可能会问为什么！非确定性进入了请求生命周期：
- 上下文在后台不断变化，思考过程、上下文窗口、探索（工具调用）以及对数据结果的理解也随之变化，产生不同的 token。
这里有一篇关于为什么评估不等于测试的精彩文章 --
> **Ankur Goyal@ankrgyl**: [原文链接](https://x.com/ankrgyl/status/2084667136618762550)
>
上面帖子的大意；
- 测试 = 它能工作吗？
- 评估 = 它能做什么？
- 成熟的评估 = 它应该做什么？
> 并非所有框架都相同，可以是托管型的或开源的——它们可以根据需求呈现不同的结构。
在讨论指标之前，先把名词搞清楚。这为我们在评审中省去了很多争论：
- 任务（task）- 一个测试用例。输入 + 初始状态 + 什么算成功。
- 试验（trial）- 该任务的一次运行。你会运行很多次，因为同一个任务今天通过，明天可能失败。
- 记录（transcript，也称 trace / trajectory）- 一次试验的完整记录。推理过程、工具调用、中间结果、token。
- 结果（outcome）- 最终状态，而不是最后一条消息。智能体说“我创建了连接器”是 transcript；连接器真实存在才是 outcome。
- 评分器（grader）- 对其进行打分的逻辑。可以是代码、LLM 作为裁判（LLM-as-judge），或者人类。一个任务可以有多个评分器。
- 评估框架（eval harness）- 运行所有这些的基础设施。不同于智能体框架（agent harness）。是的，容易混淆，存在两个 harness。
- 套件（suite）- 一组共享同一目标的任务。
想象一个初级工程师接到一个工程任务——是的，你作为资深工程师，给他配备笔记本电脑、SaaS 工具、笔记本、互联网等，并要求他完成任务。
这位初级工程师有能力使用这些工具 + 互联网来收集信息并在 4 天内完成任务，而不是 1 天。框架可以帮助初级工程师做更多的工作，但“做更多的工作并不意味着它变得更好。”
类似地，框架帮助智能体做更多的事情，但并不意味着它在任何意义上变得更好。
当你给它一个新的工具/笔记本来帮助组织事物时——它是让工作质量变差了、故障变多了、时间变长了吗？你该如何追踪？——评估。
--
## 如何着手做评估
我看到大多数团队都在为评估的冷启动问题而挣扎，主要是 - 评估一个智能体有哪些好的指标，以及我如何构建它们。
冷启动比人们想象的要小。你不需要 500 个任务。从真实的失败案例中提取或手工编写的 20-50 个任务就是一个不错的开始。在早期，每一次改动都有明显而巨大的效果，所以一个小样本就足以观察到它。你等待的时间越长，任务就越难编写——早期你的产品需求就是你的测试用例，后期你则是从一个已经运行的系统中逆向工程出成功标准。
从哪里获得这 20-50 个任务：你每次发布前手工检查的任何内容，加上你的 bug 追踪器，加上支持工单队列。真实的投诉比合成的问题能做出更好的任务。
有两件事悄悄决定着你的评估是否有价值：
编写的任务要让两个人能以相同方式评分。如果两位领域专家读了任务后在通过/失败上产生分歧，这种模糊性就会永远作为噪声出现在你的指标中。另外为每个任务写一个参考解决方案——一个能通过所有评分器的已知良好输出。它证明了任务是可解的，而且你的评分器没有坏。
构建负面的一半。单边的评估会造就单边的智能体。如果你只测试“该搜索时它是否搜索了”，你发布的就是一个对所有东西都搜索的智能体。我们在搜索智能体上深刻体会到了这一点——欠触发和过触发是同一类 bug 的正负两个符号，只有当套件同时包含两种类型的任务时你才能发现它。
如果今天我开始做评估，我只会构建这两个指标来建立对智能体的信心：
1. 任务完成度
2. 正确性
智能体能够完成任务吗？能！那是正确的答案吗？是！
那么你就可以进行内部测试了，但还不准备好上生产环境。
--
## 评估它产出的结果，而不是它走的路径
直觉做法是断言确切的工具调用序列。别这么做。智能体总会找到你没有想到的有效方法，而每一个脆弱的路径断言都会惩罚这一点。路径断言还会将你的评估耦合到框架的内部实现细节上，导致你无法在不破坏自己测试的情况下进行重构。
结果评分是你的门槛。阅读 transcript 是你的调试器。
还有一点：构建部分得分机制。一个识别出问题、验证了用户但最后一步失败的智能体，明显好于一个在第 2 轮就死掉的智能体。单一的二元评分掩盖了整个连续谱，而改进恰恰存在于这个连续谱之中。
--
## 然后观察轨迹
你现在会看到来自内部测试的追踪数据流，你开始监控轨迹——它会向你展示对思考过程、工具调用的真正洞察，并帮助你理解智能体在各类任务中的行为。
你可能想要捕获和评估的信号有：
- 分心 - 当智能体偏离轨道时，它是重新对齐任务还是变得毫无头绪。
- 花费的 token - 它是否真的需要花费数百万 token 来完成任务，还是可以用更少的 token 完成。
- 延迟 - 智能体完成任务所需的时间。尤其是面向客户时，客户等待智能体说“我帮不了你”的注意力和时间并不多。
- 上下文膨胀 - 有多少不必要的信息流入了上下文窗口。
- 一致性 - 相同任务、相同条件，运行五次。答案相同吗？这一条最让我惊讶。
最后一条值得单独一节来讨论。
--
## 你的评估本身也是可能出错的东西
这是我长期以来低估的部分。
低分并不自动意味着是智能体的错。可能是评分器、任务规格或框架的问题。
一个好用的规则：跨多次试验的通过率都是 0%，通常意味着任务有问题，而不是智能体能力不足。失败应该是“感觉公平”的——你应该能够查看一个失败的 transcript 并说：对，这次确实怪智能体。
所以，去读 transcript。不是在大版本发布后抽样读，而是每周读。这是整个过程中杠杆率最高的习惯，也是所有人都会跳过的习惯，因为它不产生数字。
这同样适用于 LLM 作为裁判（LLM-as-judge）。它需要自己的一套与人工评分对照的校准集、自己的版本历史，以及分歧分析。给裁判一个明确的退路（“如果没有足够的信息，返回 unknown”），这样它就不会幻觉出一个判决。并且让每个裁判只评一个维度，而不是一个裁判评六个维度——信号会干净得多。
> 这也值得做预算——生产环境中的裁判评估通常会消耗大约 30-50% 的推理成本。
--
## 评估需要不断演进
在构建和维护智能体与评估的过程中，我得到的一个有趣的体会是：今天的指标标准到明天对于一个更好的模型/框架来说就不再适用了。评估需要不断演进。
我们最初做的是搜索智能体，但随着模型和框架都变得更加强大，它变成了一个通用智能体，可以执行操作、设置数据连接器、生成可直接用于演示的产物等等——原本用于评估搜索任务的评估不再好用。所以我们演进它。
> 所有新的和成功的模式都需要反馈到评估中。可以给你回归检测和可改进之处的洞察。
然后进行登山式优化（hill climb）。
一旦你为你的智能体搞定了所有这些，你就可以用这些评估来进行登山式优化，[@Vtrivedy10](https://x.com/@Vtrivedy10) 在解释如何做到这一点方面做得很棒 --
> **Viv@Vtrivedy10**: [原文链接](https://x.com/Vtrivedy10/status/2041927488918413589)
>
随着你越深入，你会意识到你需要一个评估来评估你的评估。感谢阅读！
## 相关链接
- [hrushikesh](https://x.com/Hrushikeshhhh)
- [@Hrushikeshhhh](https://x.com/Hrushikeshhhh)
- [10K](https://x.com/Hrushikeshhhh/status/2099590015336808865/analytics)
- [Mar 11](https://x.com/Vtrivedy10/status/2031408954517971368)
- [825K](https://x.com/Vtrivedy10/status/2031408954517971368/analytics)
- [Aug 4](https://x.com/ankrgyl/status/2084667136618762550)
- [33K](https://x.com/ankrgyl/status/2084667136618762550/analytics)
- [@Vtrivedy10](https://x.com/@Vtrivedy10)
- [Apr 9](https://x.com/Vtrivedy10/status/2041927488918413589)
- [441K](https://x.com/Vtrivedy10/status/2041927488918413589/analytics)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:04 AM · Sep 15, 2026](https://x.com/Hrushikeshhhh/status/2099590015336808865)
- [10.8K Views](https://x.com/Hrushikeshhhh/status/2099590015336808865/analytics)
- [View quotes](https://x.com/Hrushikeshhhh/status/2099590015336808865/quotes)
---
*导出时间： 2026/9/15 13:28:56* 
