# Eval Engineering: the step that turns a $200 model into a $200,000 system (complete build)
**作者**: Argona
**日期**: 2026-07-24T12:08:00.000Z
**来源**: [https://x.com/Argona0x/status/2082127026538868839](https://x.com/Argona0x/status/2082127026538868839)
---
Everyone is renting the same brain now.
The model you can have for a couple of hundred a month is, give or take, the model a company with a thousand engineers is running. That was supposed to level everything. Instead it made the gap between two teams renting the identical brain wider than the gap between two models.
A travel agent, the software kind, answered a question about a trip: an exchange rate to one decimal place, the temperature for the week, the opening hours of a museum. Specific, clean, useful-sounding. All of it invented.
The search tool had come back empty, and the model quietly filled the hole and handed its own invention over as a fact it had looked up.
Then the team measured it across 100 real sessions and got two numbers that should never sit that far apart.
How good the answers were: 83.9%.
How much of those answers was grounded in what the tools actually returned: 32.3%.
The agent wrote beautifully and told the truth about a third of the time.
Nobody caught it, because the writing was the only part anyone ever looked at.
83.9% response quality against 32.3% faithfulness, one and the same run
Same agent, same day: the left bar is what the demo shows, the right bar is what a customer actually gets.
A better model would not have caught this. What was missing was the layer that decides whether an answer was right, and then does something with the verdict.
That layer is the whole distance between an AI that impresses people and an AI that gets paid. It is the cheapest line in the stack and the only one nobody can sell you, because it encodes your own definition of correct.
This one is not a skim. Open a terminal beside it. The setup starts with a single command you already own, and it ends somewhere that sounds unreasonable: pull requests that merge with nobody reading them.
> Before we get into it, follow me on X and join my Telegram channel where
> post more AI content every day. Both are free.
> X - https://x.com/Argona0x
> Telegram - https://t.me/+r0clI4-MMC03ZjAy
# What eval engineering is
A company running agents in production said the quiet part in one sentence. Across successive versions of their agent they changed no model, no prompts, no human nudges, only the evals — and the scores moved across every dimension.
Same brain. Different examiner. Better product.
Their own conclusion afterwards was blunter: the evals should have been in the system on day one, not month nine.
A thermometer tells you the room is cold. A thermostat turns the heat on.
Almost everyone building with AI owns a thermometer at best: a dashboard, a feeling, a Friday afternoon where somebody scrolls outputs and says it seems worse than last week.
Eval engineering is the wiring that runs from the reading to the furnace.
← thermometer against thermostat, with the verdict returning into the graph
The whole difference is the return arrow on the right: the verdict goes back into the graph and changes what runs next.
This arrived in a fixed order. One agent in a loop came first, so it could try, look at what came back, and try again. Then many agents laid out as a graph, so work that never depended on each other ran side by side instead of standing in line.
Loops, then graphs, then evals.
> **Argona@Argona0x**: [原文链接](https://x.com/Argona0x/status/2080626046903157126)
>
> a new kind of engineer is showing up. they don't write prompts, they design how the AI works. it's called graph engineering, and the people who get it now are about to make everyone else look slow
The graph was the last upgrade that made you faster. The judge decides whether that speed was worth having.
Twenty agents running at once, all reporting to one frozen judge, is twenty times as many places for a wrong answer to look finished.
It is also the only part of the system that gets more valuable every week you run it. The model is a rental. The examiner is yours, and every failure you feed it stays there permanently.
Which makes the price of entry strange. The examiner costs almost nothing to install, and the model it examines costs about $200 a month.
# Step 1 · The eval engine you already own
You assume this starts with a platform: a contract, a per-seat price, a quarter of setup before the first useful number.
That assumption is why most people never build the layer at all, and it stopped being true about four weeks ago.
## The blind test
1. Two researchers ran the honest test. They took 100 real production traces from a live voice agent, had a human expert mark every failure by hand, built a taxonomy of 39 labeled failures, then hid the labels and handed the same pile to every evaluation system on the market.
Find them yourself.
The interesting row was not the winner:
1. Braintrust Loop: 87.2% of the human-flagged failures recovered.
2. Codex on GPT-5.5 High: 84.6% recall, 82.8% precision.
3. LangSmith: 79.5%.
4. Arize AX: 74.4% recall, and the cleanest precision of the group at 91.0%.
A general coding agent, on a subscription you already pay for, landed above two dedicated evaluation platforms. One of them wrote the conclusion plainly: you can get similar results from using your coding agent.
Then the platforms did something stranger than losing. They moved in.
Inside four weeks, four evaluation vendors shipped their expertise as a <skill installed into somebody else's coding agent: LangChain on 22 July, Galileo, AWS under Apache 2.0, and Arize putting its trace extraction into skills aimed at, in their words, your favorite coding agent.
Nobody has counted those four as one move.
It is a category unbundling itself into the terminal you already have open.
## Installing the examiner
Installing all of it takes three lines.
```
npx skills add langchain-ai/langchain-skills --skill '*' --yes --global
npx skills add langchain-ai/langsmith-skills --skill '*' --yes --global
uv tool install evalkit --from git+https://github.com/awslabs/Agent-EvalKit.git
```
Inside Claude Code the same thing installs as a plugin:
```
/plugin marketplace add langchain-ai/langchain-skills
/plugin install langchain-skills@langchain-skills
```
← real terminal run: the skills install, evalkit init scaffolds the folder
A real run on a real machine: the skills land as global files, and evalkit init lays out the folder your evaluation will live in.
Two details in there are worth more than the commands.
1. Two repositories, two jobs: langchain-skills builds the evals, langsmith-skills pulls the real production runs to build them from. Almost everyone installs the first, then wonders where the material comes from.
2. Not a Claude Code feature: those skills follow the open skills specification, so the identical files load into Codex, Cursor, Windsurf and Goose.
The AWS kit runs on six commands, each a phase writing into an eval/ folder the next one reads:
```
evalkit init my-agent-evaluation
# then, inside your coding agent:
/evalkit.plan Evaluate my agent at ./my_agent for grounding and tool accuracy
/evalkit.data
/evalkit.trace
/evalkit.run_agent
/evalkit.eval
/evalkit.report
```
The last command is the one worth the whole setup. /evalkit.report hands back prioritized recommendations pointing at specific locations in your code.
The examiner is installed. Now it has to be allowed to do something.
# Step 2 · Make the score change the next edge
You will build your first evaluation, get a number, look at it, and feel nothing happen.
That feeling is correct. A number in a report has no path back to the run it measured.
One line fixes the whole discipline, and it belongs to a 21-year-old founder in Tokyo with 258 followers whose post on it got two likes:
> A score that never changes behavior is analytics. An eval that changes the next edge is engineering.
His wiring is six rules. Each takes a verdict and does something structural to the run in progress:
```
low context recall → reject the handoff
bad tool use → retry or swap the node
hallucination → quarantine the branch
schema failure → block the edge
compliance risk → route to human review
verified completion → terminate the run
```
← six verdicts, six structural actions on the run in progress
Every one of those six is a routing decision the graph executes. The eval steers the run mid-flight, one edge at a time.
## Rules for the judge itself
The examiner needs hygiene of its own, and almost nobody sets this part up correctly.
1. Judge from another family: a model recognizes its own writing and grades it kinder once it does. One evaluation team runs Sonnet as the judge for output generated by Haiku, on purpose. One line on X gets it in fewer words: same family generates and grades, so the blind spots are shared.
2. Write the rubric as one line: the working form is literally Pass iff [the independently observable successful outcome]. One primary verdict, never a bundle of proxy scores.
3. Split the work by kind: the judge decides the semantic calls, plain code decides the objective ones. Did the test pass, does the file exist, did the state change.
4. Never reward the shape of an answer: the rule written into the skill forbids scoring on response length, keywords, citation count, exact phrasing, tool-call count or similarity to a reference. Reward the shape and the agent learns the shape.
5. Pin the judge and log its version: an examiner that silently upgrades makes every score before and after incomparable, and you will not notice for a month. The version pin is the one people skip, and the one that makes a month of scores unreadable after the fact.
Optimize against a judge long enough and the agent learns to look right rather than be right.
Now the score has somewhere to go. The next problem is where good tests come from, because inventing them at a desk is why everybody's first set is useless.
# Step 3 · Turn a failed run into an eval
Tests you invent from imagination protect you from failures you already imagined.
The ones that cost money are sitting in your logs right now, wearing a timestamp.
The whole loop is five steps long:
```
mine traces -> identify a failure -> build an eval -> improve the agent -> rerun
```
## Where the tests come from
The first step carries all the weight. The skill that does this professionally names which runs to pull, and it starts with 25 complete traces, no more, chosen so good and bad behavior sit next to each other:
1. A normal request that finished: your baseline for what working looks like.
2. A request the user confirmed: the rare trace where you know the answer was good.
3. A request the user corrected or rephrased: the correction is the label, free of charge.
4. A run with a failed, empty or repeated tool call: repetition means a loop, emptiness means an invented answer is coming.
5. A run with an external failure: a timeout or a rate limit, where the only thing tested is how your agent behaves when the world says no.
Each one gets written up in four lines, and this template is the part worth stealing:
```
Observed behavior: what the user asked and what the agent actually did
Comparison: what worked and what did not
Attribution: agent behavior, dependency behavior, or unclear
Eval candidate: the capability to preserve or improve
```
Attribution is where beginners lose a week.
The same lookup called twice with identical arguments is a loop in your agent. A 429 coming back is somebody else's limit, and it only becomes your eval if your agent was supposed to recover from it.
## From finding to folder
Then the finding becomes a folder, and the folder is the format the tooling runs:
```
evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
```
← 25 traces → one failure → a Harbor task folder: what the agent sees, what stays hidden
One capability per folder. The instruction and the environment are visible to the agent being tested. The expected outcome, the rubric and the judge's credentials are not, and that separation is the only reason the score means anything.
Three rules stop the failures that make people quit:
1. Never treat the recorded answer as truth: the trace tells you what your agent did, never what it should have done. Take the answer key from tests, source records, policy, known state or a person.
2. Test the test before you trust it: hand the verifier two fake results by hand, one clearly correct and one plausible but wrong. If either goes the wrong way, the rubric is broken, not the agent.
3. Watch for the environment giving away the answer: if the setup hands over the result before the agent reaches the tool it was supposed to use, the task passes forever and measures nothing.
Anything that costs money or writes to production gets simulated rather than called, so the suite runs as often as you like without a bill.
One instruction starts all of it, in the agent you already have open:
```
Use the eval-engineering skill.
Map this repository's agent: entrypoint, tools, backing data, and what a good
result looks like. Then read the 25 traces in ./traces and propose two or three
eval candidates grounded in what actually failed there.
Recommend one. Do not implement until I choose.
Build it as a Harbor task under evals/, keep the rubric and expected outcome
hidden from the target, and test the verifier on one passing and one plausible
wrong result before the real run.
```
The interview step is deliberate. The team that wrote the skill found that questioning the user beats one-shot generation every time, for the plain reason that the definition of correct lives in your head and nowhere in the model.
Run it a handful of times and every failure stops being an incident and becomes a permanent test.
# Step 4 · Let the graph merge its own work
Every pull request an agent opens lands in the same place: a human queue.
You become the bottleneck of your own automation, and the fleet you built runs at the speed of one tired reviewer.
The way out looks nothing like trusting the model more. When an agent opens a pull request, four signals are already available, and a confidence score is computed from them on the spot:
1. Guardrails result: a deterministic pass or fail on the blocking standards. No model involved.
2. Recent eval trajectory: how this exact version of this agent has been scoring lately.
3. Historical revert rate: how often this agent, on this repository, on this class of change, has had its work rolled back before.
4. Sandbox outcome: did it run.
bove the threshold it merges itself. Below it, a human gets it with the failing signal named, so the review starts at the problem instead of at line one.
← four signals → one confidence score → merge itself, or route to a human
Three of those four are history and deterministic checks, and exactly one of them touches the model at all.
Trust in an agent is an actuarial calculation.
What you are building is a track record with a price on it, the same way an insurer builds one, and it gets sharper every week whether or not the models improve.
## What it looks like when it runs
At the same company that changed nothing but its evals, 19 of every 20 pull requests on the fully autonomous agent merge with no human involved.
Across the top agents, about three quarters of merged work goes in without a single human edit, the revert rate stays in the low single digits, and Guardrails alone bounces one pull request in five before a person ever sees it.
Somebody running this pattern on his own repositories put it in a way no vendor would:
> In the past 90 days, I've approved and merged around 1,500 pull requests. I haven't looked at a line of code. How can I put so much trust in agents? I don't trust them at all. I also don't trust my ability to code review their work. But I do trust my ability to constrain their work.
The constraint is the product.
And the warning worth more than the formula comes from a team that ran 285 iterations of a self-improving codebase and came out with 1,094 merged pull requests and zero regressions
Their own line is the one to keep: 38 green tests coexisted with a completely broken product.
A suite can go all green while the product it guards falls apart, which is why the loop has to converge on the spec rather than on the score.
Turn it on the careful way:
1. Run it in shadow first: the gate scores every pull request and merges none of them, for at least 4 hours of real traffic.
2. Set a deviation threshold of 2%: if the automated verdict and the human verdict disagree more than that, the gate stays closed.
3. Sample traces at 1% to 5%: full capture on everything is a cost you do not need to carry.
A two-person shop where every agent-written change waits on one reviewer can take on about as much work as that reviewer can read.
With the gate closed on the risky slice and open on the boring 80%, the same two people start quoting on the contracts they used to decline.
The model bill does not move. Everything else does.
# Step 5 · The evals to build this week
A suite that is too slow or too vague never gets run, so this is the part to save.
Start with three measurements, not twelve. The three that exposed the travel agent are a working default for any agent that calls tools:
1. Faithfulness: is the answer grounded in what the tools actually returned. This is the one that sat at 32.3% while every other number on the dashboard looked fine.
2. Tool parameter accuracy: right tool, right arguments.
3. Response quality: is the output coherent and useful to the person who asked.
If your agent ships code, score the change instead. The five dimensions used in production on every pull request: intent and decision, execution and artifact, completeness and usefulness, instruction and boundary, efficiency.
Pick the dataset type on purpose. Four exist: final_response for the answer alone, single_step for one decision in isolation, trajectory for the whole path the agent took, and RAG for retrieval quality.
Grading only the final response is how an agent reaches a correct answer through a broken sequence with nobody noticing.
Size it so it stays alive. Hold out 300 to 800 cases, and keep 500 of them running
in under 5 minutes.
A suite that takes longer than a coffee break stops being run.
The first five evals, in order:
1. Empty tool result: the tool returns nothing, and the agent has to say so instead of inventing the number.
2. Repeated call: the same lookup with the same arguments twice. That is a loop, and the eval fails it.
3. Boundary refusal: asked for something outside its permissions, the agent declines cleanly instead of hunting for a route around.
4. Handoff integrity: what the previous node produced is what the next node reads, with nothing invented in between.
5. Verified completion: done means a real signal says done, never the agent's own word for it.
← this week: three measurements, one dataset type, five evals (the save-card)
Three measurements, one dataset type, five evals. That is an afternoon of work, and every week after it the suite is worth more than it was the week before.
# The people who go first
The model was never the interesting part. It is a rental, identical for everyone, and it will be replaced twice before the end of the year.
What survives every replacement is the examiner you built around it: the failures you turned into permanent tests, the rules that let a verdict change the next edge, the track record that lets a machine merge its own work.
That is what decides whether the $200 on your card statement produces a demo or produces a business.
Most people will go back to scrolling outputs on a Friday and deciding it feels about right.
The ones who go first spend one afternoon wiring the thermostat, and then spend the next year with an agent that cannot break the same thing twice.
Three lines hold the whole discipline:
1. Measure the path the agent took, never only the answer it landed on.
2. A verdict that does not change the next edge is a report.
3. Any failure you do not turn into a permanent test, you will meet again.
Install one skill. Pull 25 traces. Build one eval today, and add one every time something breaks.
> If you want to stay up to date with everything happening in AI, follow me on X and Telegram:
> X - https://x.com/Argona0x
> Telegram - https://t.me/+r0clI4-MMC03ZjAy
## 相关链接
- [Argona](https://x.com/Argona0x)
- [@Argona0x](https://x.com/Argona0x)
- [13K](https://x.com/Argona0x/status/2082127026538868839/analytics)
- [https://x.com/Argona0x](https://x.com/Argona0x)
- [https://t.me/+r0clI4-MMC03ZjAy](https://t.me/+r0clI4-MMC03ZjAy)
- [Jul 24](https://x.com/Argona0x/status/2080626046903157126)
- [344K](https://x.com/Argona0x/status/2080626046903157126/analytics)
- [https://x.com/Argona0x](https://x.com/Argona0x)
- [https://t.me/+r0clI4-MMC03ZjAy](https://t.me/+r0clI4-MMC03ZjAy)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:32 PM · Jul 28, 2026](https://x.com/Argona0x/status/2082127026538868839)
- [13.7K Views](https://x.com/Argona0x/status/2082127026538868839/analytics)
- [View quotes](https://x.com/Argona0x/status/2082127026538868839/quotes)
---
*导出时间: 2026/7/29 08:59:32*
---
## 中文翻译
# 评估工程：将 200 美元的模型转变为 20 万美元系统的步骤（完整构建指南）
**作者**: Argona
**日期**: 2026-07-24T12:08:00.000Z
**来源**: [https://x.com/Argona0x/status/2082127026538868839](https://x.com/Argona0x/status/2082127026538868839)
---
现在每个人都在租用同一个大脑。
你每个月花几百美元就能拥有的模型，跟一家拥有千名工程师的公司所运行的模型，大致上是一样的。这本该让一切变得平等。结果，它反而让两个租用相同大脑的团队之间的差距，变得比两个不同模型之间的差距还要大。
一个旅行社——软件意义上的那种——回答了一个关于旅行的问题：精确到小数点后一位的汇率、一周的气温、一家博物馆的开放时间。具体、干净、听起来很有用。但所有内容都是编造的。
搜索工具什么也没查到，模型便悄悄填补了这个空白，并把自己编造的东西当作它查到的事实交了出去。
然后，团队在 100 个真实会话中对此进行了测量，得到了两个本不该相差如此之远的数字。
答案的质量有多高：83.9%。
这些答案中有多少是基于工具实际返回的内容：32.3%。
这个智能体写得很漂亮，但只有三分之一的时候在说真话。
没人发现这一点，因为文字是人们唯一会看的部分。
83.9% 的响应质量对比 32.3% 的忠实度，同一次运行
同一个智能体，同一天：左边的柱状图是演示展示的内容，右边的柱状图是客户实际得到的内容。
更好的模型也发现不了这个问题。缺失的是那一层：判断答案是否正确，然后根据裁决采取行动。
这一层是让 AI 仅停留在“令人印象深刻”和 AI 能够“真正赚到钱”之间的全部距离。它是技术栈中最便宜的一条线，也是唯一没人能卖给你的线，因为它编码了你自己对“正确”的定义。
本文不是那种略读概览。请在旁边打开一个终端。设置从你已经拥有的一条命令开始，结束于一个听起来不可思议的地方：无需任何人阅读即可合并的拉取请求（Pull Requests）。
> 在开始之前，请在 X 上关注我并加入我的 Telegram 频道，我会在那里每天发布更多 AI 内容。两者都是免费的。
> X - https://x.com/Argona0x
> Telegram - https://t.me/+r0clI4-MMC03ZjAy
# 什么是评估工程
一家在生产环境中运行智能体的公司用一句话道出了这个安静的真相。在智能体的连续几个版本中，他们没有更改模型、没有更改提示词、没有人工干预，只更改了评估——结果每个维度的分数都发生了变化。
同一个大脑。不同的考官。更好的产品。
他们事后得出的结论更加直白：评估应该在系统构建的第一天就存在，而不是第九个月。
温度计告诉你房间很冷。恒温器会打开暖气。
几乎所有使用 AI 构建应用的人最多只拥有一个温度计：一个仪表盘、一种感觉、某个周五下午有人滚动查看输出并说“看起来比上周差”。
评估工程就是从读数到炉子之间的布线。
← 温度计对比恒温器，裁决结果返回到图中
整个区别在于右边的返回箭头：裁决回到图中，并改变接下来运行的内容。
事物发展遵循固定的顺序。首先是一个循环中的单个智能体，这样它可以尝试、观察返回结果，然后再试一次。然后是许多智能体以图的形式布局，这样互不依赖的工作可以并行运行，而不是排队等待。
先循环，再图，然后评估。
> **Argona@Argona0x**: [原文链接](https://x.com/Argona0x/status/2080626046903157126)
>
> 一种新型的工程师正在出现。他们不写提示词，他们设计 AI 的工作方式。这被称为图工程，现在就懂这一点的人将让其他人看起来很慢
图是让你变快的最后一次升级。法官决定这种速度是否值得拥有。
二十个智能体同时运行，都向一个冻结的法官汇报，这就意味着错误答案看起来像最终完成的地方增加了二十倍。
这也是系统中唯一一个你运行的每一周都会变得更值钱的部分。模型是租来的。考官是你自己的，你喂给它的每一次失败都会永久留在那里。
这使得准入成本显得很奇怪。安装考官几乎不需要成本，而它检查的模型每月大约花费 200 美元。
# 第 1 步 · 你已经拥有的评估引擎
你假设这始于一个平台：一份合同、一个按座位的定价、在第一个有用的数字出现之前需要四分之一的设置时间。
这个假设就是大多数人根本没有构建这一层的原因，而且这种情况在四周前就已经不再是事实了。
## 盲测
1. 两名研究人员进行了诚实测试。他们从一个活跃的语音智能体那里提取了 100 条真实的生产轨迹，让人工专家手动标记每一个失败，建立了一个包含 39 个标记失败的分类体系，然后隐藏标签并将同一堆数据交给市场上所有的评估系统。
自己去找找看。
有趣的一行不是获胜者：
1. Braintrust Loop：恢复了 87.2% 的人工标记失败。
2. GPT-5.5 High 上的 Codex：84.6% 的召回率，82.8% 的精确率。
3. LangSmith：79.5%。
4. Arize AX：74.4% 的召回率，以及组中最干净的精确率 91.0%。
一个通用的编码智能体，在你已经订阅的套餐中，其表现高于两个专门的评估平台。其中一个平台直白地写下了结论：你可以通过使用你的编码智能体获得类似的结果。
然后这些平台做了一件比失败更奇怪的事情。它们进来了。
在四周内，四家评估供应商将它们的专业知识作为技能安装到了别人的编码智能体中：7 月 22 日的 LangChain，Galileo，AWS 在 Apache 2.0 协议下，以及 Arize 将其轨迹提取放入技能中，用他们的话说，针对你最喜欢的编码智能体。
没有人把这四家视为同一举动。
这是一个类别正在自行解构，融入你已经打开的终端中。
## 安装考官
安装所有内容只需要三行。
```
npx skills add langchain-ai/langchain-skills --skill '*' --yes --global
npx skills add langchain-ai/langsmith-skills --skill '*' --yes --global
uv tool install evalkit --from git+https://github.com/awslabs/Agent-EvalKit.git
```
在 Claude Code 中，同样的内容作为插件安装：
```
/plugin marketplace add langchain-ai/langchain-skills
/plugin install langchain-skills@langchain-skills
```
← 真实的终端运行：技能安装，evalkit init 搭建文件夹
真实机器上的真实运行：技能作为全局文件落地，evalkit init 布局了你的评估将要生存的文件夹。
其中有两个细节比命令本身更有价值。
1. 两个仓库，两项工作：langchain-skills 构建评估，langsmith-skills 拉取真实的生产运行来构建它们。几乎每个人都安装了第一个，然后不知道材料从哪里来。
2. 不是 Claude Code 的独有功能：这些技能遵循开放的技能规范，所以相同的文件可以加载到 Codex、Cursor、Windsurf 和 Goose 中。
AWS 工具包运行六个命令，每个阶段向 eval/ 文件夹写入内容，供下一个阶段读取：
```
evalkit init my-agent-evaluation
# 然后，在你的编码智能体中：
/evalkit.plan Evaluate my agent at ./my_agent for grounding and tool accuracy
/evalkit.data
/evalkit.trace
/evalkit.run_agent
/evalkit.eval
/evalkit.report
```
最后一个命令值得整个设置。/evalkit.report 返回优先级建议，指向代码中的具体位置。
考官已安装。现在必须允许它做点什么。
# 第 2 步 · 让分数改变下一个边
你会构建你的第一个评估，得到一个数字，看着它，然后感觉什么也没发生。
这种感觉是对的。报告中的数字没有路径回到它测量的运行中。
一行代码就能修正整个学科，它属于东京一位 21 岁的创始人，他有 258 个粉丝，关于这个的帖子只得到了两个赞：
> 一个永远不改变行为的分数是分析。一个改变下一个边的评估是工程。
他的布线是六条规则。每一条接收一个裁决，并对正在进行的运行做一些结构性改变：
```
low context recall → reject the handoff
bad tool use → retry or swap the node
hallucination → quarantine the branch
schema failure → block the edge
compliance risk → route to human review
verified completion → terminate the run
```
← 六种裁决，对正在进行的运行采取六种结构性操作
这六个中的每一个都是图执行的路由决策。评估在运行中途引导流向，一次一个边。
## 法官本身的规则
考官需要自己的卫生规范，几乎没有人正确设置这一部分。
1. 来自不同家族的法官：模型识别自己的写作并在识别后给予更宽松的评分。一个评估团队特意使用 Sonnet 作为 Haiku 生成输出的法官。X 上有一句话更简短地表达了这一点：同一家族生成并评分，所以盲点是共享的。
2. 将评分标准写成一行：工作形式字面上就是“当且仅当 [独立观察到的成功结果] 时通过”。一个主要裁决，绝不是一堆代理分数。
3. 按类型分拆工作：法官决定语义调用，普通代码决定客观调用。测试通过了吗？文件存在吗？状态改变了吗？
4. 绝不要奖励答案的形状：写入技能的规则禁止根据响应长度、关键词、引用数量、确切措辞、工具调用数量或与参考的相似性进行评分。奖励形状，智能体就会学会形状。
5. 固定法官版本并记录：一个默默升级的考官会使升级前后的每个分数都无法比较，而且你一个月都不会注意到。版本固定是人们跳过的步骤，也是导致一个月后的分数事后无法阅读的原因。
针对一个法官优化得足够久，智能体就会学会看起来正确，而不是真正正确。
现在分数有了去向。下一个问题是从哪里获得好的测试，因为在办公桌前凭空发明测试就是每个人的第一套测试都没用的原因。
# 第 3 步 · 将失败的运行转化为评估
你凭空想象出的测试只能保护你免受你已经想到的失败的影响。
那些真正要花钱的测试正坐在你的日志里，带着时间戳。
整个循环有五个步骤：
```
mine traces -> identify a failure -> build an eval -> improve the agent -> rerun
```
## 测试从哪里来
第一步承载了所有的重量。专业执行此任务的技能会指定要提取哪些运行，它从 25 条完整的轨迹开始，不多不少，选择时让好的行为和坏的行为并排出现：
1. 一个完成的正常请求：你关于“正常工作”是什么样子的基准。
2. 一个用户确认过的请求：你知道答案很好的罕见轨迹。
3. 一个用户纠正或改写的请求：纠正是标签，免费赠送。
4. 一个带有失败、空或重复工具调用的运行：重复意味着循环，空白意味着编造的答案即将到来。
5. 一个带有外部失败的运行：超时或速率限制，此时测试的唯一内容是你的智能体在世界说“不”时的表现。
每一条都会被写成四行，这个模板是值得借鉴的部分：
```
Observed behavior: what the user asked and what the agent actually did
Comparison: what worked and what did not
Attribution: agent behavior, dependency behavior, or unclear
Eval candidate: the capability to preserve or improve
```
归因是初学者浪费一周的地方。
用相同参数调用两次相同的查询是你的智能体中的循环。返回的 429 是别人的限制，只有当你的智能体应该从中恢复时，它才成为你的评估。
## 从发现到文件夹
然后发现变成了一个文件夹，而文件夹是工具运行的格式：
```
evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
```
← 25 条轨迹 → 一个失败 → 一个 Harbor 任务文件夹：智能体看到什么，什么保持隐藏
每个文件夹一种能力。指令和环境对被测试的智能体可见。预期结果、评分标准和法官的凭证不可见，这种分离是分数有意义的唯一原因。
三条规则阻止那些让人放弃的失败：
1. 永远不要将记录的答案视为真理：轨迹告诉你你的智能体做了什么，从未告诉你它应该做什么。从测试、源记录、策略、已知状态或人那里获取答案键。
2. 在信任之前测试测试：手动给验证器两个假结果，一个明显正确，一个看似合理但错误。如果任何一个走向错误的方向，那是评分标准坏了，不是智能体坏了。
3. 注意环境泄露答案：如果设置在智能体到达它应该使用的工具之前就交出了结果，任务将永远通过，并且什么也测不出来。
任何花费金钱或写入生产环境的内容都会被模拟而不是直接调用，这样套件可以尽可能频繁地运行而无需账单。
一条指令启动所有这一切，在你已经打开的智能体中：
```
Use the eval-engineering skill.
Map this repository's agent: entrypoint, tools, backing data, and what a good
result looks like. Then read the 25 traces in ./traces and propose two or three
eval candidates grounded in what actually failed there.
Recommend one. Do not implement until I choose.
Build it as a Harbor task under evals/, keep the rubric and expected outcome
hidden from the target, and test the verifier on one passing and one plausible
wrong result before the real run.
```
询问这一步是蓄意的。编写该技能的团队发现，询问用户总是胜过一次性生成，原因很简单：对“正确”的定义存在于你的头脑中，而不存在于模型中。
运行几次，每一次失败就不再是事故，而变成永久测试。
# 第 4 步 · 让图合并自己的工作
智能体打开的每个拉取请求都落在同一个地方：一个人工队列。
你成为自己自动化的瓶颈，你构建的机队以一个疲惫的审核员的速度运行。
出路看起来一点也不像是更信任模型。当一个智能体打开一个拉取请求时，四个信号已经可用，并当场从中计算出一个置信度分数：
1. 护栏结果：对阻塞标准的确定性通过或失败。不涉及模型。
2. 最近的评估轨迹：该智能体的该确切版本最近评分如何。
3. 历史回退率：该智能体、该仓库、该类别的更改，以前的工作回滚的频率。
4. 沙盒结果：它运行了吗。
高于阈值，它自己合并。低于阈值，人工获得它并附带失败的信号名称，因此审核从问题开始，而不是从第一行开始。
← 四个信号 → 一个置信度分数 → 自动合并，或路由给人工
这四个信号中，三个是历史记录和确定性检查，正好有一个接触模型。
对智能体的信任是一种精算计算。
你正在构建的是一份带有价码的业绩记录，就像保险公司建立的那样，无论模型是否改进，它每周都会变得更敏锐。
## 运行起来是什么样子
在那家除了评估什么都没改变的公司，完全自主智能体每 20 个拉取请求中有 19 个在没有任何人工参与的情况下合并。
在顶级智能体中，大约四分之三的合并工作没有任何人工编辑就进去了，回退率保持在个位数低位，仅护栏一项就会在人员看到之前将五分之一的拉取请求弹回。
在他自己的仓库上运行这种模式的人用一种供应商绝不会使用的方式表达了这一点：
> 在过去的 90 天里，我批准并合并了大约 1,500 个拉取请求。我从未看过一行代码。我怎么能对智能体投入如此多的信任？我根本不信任它们。我也不信任我代码审查它们工作的能力。但我确实信任我限制它们工作的能力。
约束就是产品。
而比公式更有价值的警告来自一个团队，他们运行了 285 次自我改进的代码库迭代，结果产生了 1,094 个合并的拉取请求和零回归
他们自己的一句话值得保留：38 个通过的测试与一个完全损坏的产品共存。
一个套件可能全部变绿，而它保护的产品却分崩离析，这就是为什么循环必须收敛于规范而不是分数。
以谨慎的方式开启它：
1. 先在影子模式下运行：门控对每个拉取请求评分，但都不合并，至少 4 小时的真实流量。
2. 设定 2% 的偏差阈值：如果自动裁决和人工裁决的分歧超过这个值，门控保持关闭。
3. 以 1% 到 5% 的比例采样轨迹：全量捕获所有内容是你不需要承担的成本。
一个两个人经营的店，每个智能体编写的更改都在等待一个审核员，所能承担的工作量大约就是那个审核员能读的量。
当门控对风险部分关闭，对无聊的 80% 开放时，这两个人开始报价他们过去拒绝的合同。
模型的账单不动。其他一切都在变。
# 第 5 步 · 本周要构建的评估
一个太慢或太模糊的套件永远不会运行，所以这是需要保存的部分。
从三个测量开始，而不是十二个。暴露旅行社问题的那三个是任何调用工具的智能体的有效默认值：
1. 忠实度：答案是否基于工具实际返回的内容。这就是那个停留在 32.3% 而仪表盘上其他所有数字看起来都很好的指标。
2. 工具参数准确性：正确的工具，正确的参数。
3. 响应质量：输出是否连贯且对提问者有用。
如果你的智能体交付代码，则对更改进行评分。在每个拉取请求的生产中使用的五个维度：意图和决策、执行和工件、完整性和有用性、指令和边界、效率。
有目的地选择数据集类型。存在四种类型：final_response 仅用于答案，single_step 用于孤立的一个决策，trajectory 用于智能体采取的整个路径，以及 RAG 用于检索质量。
仅对最终响应进行评分是智能体通过损坏的序列达到正确答案而没人注意到的方式。
调整大小以保持其活跃。保留 300 到 800 个案例，并让其中 500 个在 5 分钟内运行完成。
一个比喝咖啡时间还长的套件就不会再被运行了。
前五个评估，按顺序：
1. 空工具结果：工具什么也没返回，智能体必须说明这一点，而不是编造数字。
2. 重复调用：用相同参数进行两次相同的查询。这是一个循环，评估让它失败。
3. 边界拒绝：被要求做其权限之外的事情，智能体干净地拒绝，而不是寻找绕过的路。
4. 交接完整性：前一个节点产生的内容就是下一个节点读取的内容，中间没有任何编造。
5. 已验证完成：完成意味着一个真正的信号说完成，绝不是智能体自己说完成。
← 本周：三个测量，一种数据集类型，五个评估（保存卡）
三个测量，一种数据集类型，五个评估。这是一个下午的工作，之后的每一周，套件都比前一周更有价值。
# 走在前面的人
模型从来都不是有趣的部分。它是租来的，对每个人都一样，并且在年底前会被更换两次。
在每一次更换中幸存下来的是你围绕它构建的考官：你转化为永久测试的失败、让裁决改变下一个边的规则、让机器合并自己工作的业绩记录。
这才是决定你卡上那 200 美元产生的是演示还是业务的关键。
大多数人会回到周五滚动查看输出并决定感觉差不多的状态。
那些走在前面的人花了一个下午布线恒温器，然后花了一年的时间拥有一个不会两次打破同一个东西的智能体。
三行代码包含整个学科：
1. 测量智能体采取的路径，绝不只是它落地的答案。
2. 一个不改变下一个边的裁决是一份报告。
3. 任何你不转化为永久测试的失败，你都会再次遇到。
安装一个技能。提取 25 条轨迹。今天构建一个评估，每次有东西坏了就加一个。
> 如果你想了解 AI 领域发生的最新动态，请在 X 和 Telegram 上关注我：
> X - https://x.com/Argona0x
> Telegram - https://t.me/+r0clI4-MMC03ZjAy
## 相关链接
- [Argona](https://x.com/Argona0x)
- [@Argona0x](https://x.com/Argona0x)
- [13K](https://x.com/Argona0x/status/2082127026538868839/analytics)
- [https://x.com/Argona0x](https://x.com/Argona0x)
- [https://t.me/+r0clI4-MMC03ZjAy](https://t.me/+r0clI4-MMC03ZjAy)
- [Jul 24](https://x.com/Argona0x/status/2080626046903157126)
- [344K](https://x.com/Argona0x/status/2080626046903157126/analytics)
- [https://x.com/Argona0x](https://x.com/Argona0x)
- [https://t.me/+r0clI4-MMC03ZjAy](https://t.me/+r0clI4-MMC03ZjAy)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [11:32 PM · Jul 28, 2026](https://x.com/Argona0x/status/2082127026538868839)
- [13.7K Views](https://x.com/Argona0x/status/2082127026538868839/analytics)
- [View quotes](https://x.com/Argona0x/status/2082127026538868839/quotes)
---
*导出时间: 2026/7/29 08:59:32*