# Towards Automating Eval Engineering
**作者**: Viv
**日期**: 2026-07-22T17:04:58.000Z
**来源**: [https://x.com/Vtrivedy10/status/2079976006644072796](https://x.com/Vtrivedy10/status/2079976006644072796)
---
Today we’re releasing our Eval Engineering Skill, a skill that helps coding agents build evals using context from a repository and agent traces.
The skill inspects how an agent is structured, mines patterns from traces if available, and proposes abilities to test.
The skill is designed to interview the user who can give feedback on proposals and iteratively approve each eval. The end product is a set of executable evals in Harbor format.
## Building the Environment & Task
The skill first reads the repository and maps the agent surface including prompts, models, tools, skills, hooks, etc. It also identifies the data and services that back those behaviors such such as API calls.
Users can also point the agent to traces which can be retrieved using tools like the langsmith-cli. Traces show how tools behave in practice such as their arguments, results, and errors. These observed contracts help the skill reproduce relevant production behavior in a controlled environment.
Crawling the repo and traces gives the agent knowledge of a which abilities are important for the agent as it proposes eval tasks. We found that interviewing the user, leads to much better eval acceptance than one-shot generation. The user chooses from the proposed eval directions, and gives guidance on questions such as which tools & dependencies should run live or need to be simulated. For example, tool calls that incur costs or require writes to production can be simulated instead of being run on every eval invocation.
We tested this flow on our documentation Q&A agent, chat-langchain. For this agent, the environment required a data corpus exposed through agent search tools modeled on the production agent. The tasks included realistic documentation question pulled from real traces and a verifier that checked the answer using a golden answer string and cited documents.
## Eval Design is iterative
We found that while agents are sometimes able to one-shot evals, the best evals came from users providing feedback and specifying which capabilities were worth measuring in agents. Coding agents & skills provide a natural interface where domain knowledge on how to build a good eval are encoded and users can iterate over them over time.
For example, we found that when building verifiers, the first verifier was rarely the final one. A useful way to improve it was to run the eval and inspect both sides of the result:
- the agent trajectory, including its messages, tool calls, and actions.
- the verifier trajectory, evidence, reasoning, and final score.
This helped reveal if the task or verifier design was measuring what we cared about or if it could be reward hacked where agents could take shortcuts. These shortcuts could include overciting irrelevant sources to receive full credit on the eval, claim an action it never took, exploit exposed answer material, or satisfy a proxy without completing the task. Observing the traces for how agents solve problems often reveal the source of these failures. The task, environment, and verifier can then be revised and run again.
## Evals are in Harbor format
The skill builds evals as Harbor tasks:
1. An Instruction: the message given to the agent at start describing the task
2. An environment: given as a Dockerfile containing the setup for the task such as what tools to install or what data to populate in the filesystem
3. A verifier that scores whether the agent completed the task correctly.
The skill builds these components together as a Harbor task:
```
evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
```
Harbor runs the agent in the environment and records its trajectory, artifacts, reward, and errors. The same eval can then run against different models, prompts, tools, and agent versions.
## Why this matters
Continual learning can be thought of as a continuous data mining problem where production data is used to build evals that improve agents over time. Teams mine traces to find recurring user requests, errors, failed tool calls, and incorrect state changes. which become evals so the same behavior can be measured and prevented in the future.
Evals are training data for agents. Teams can fit agent behavior to them through harness engineering such as changing prompts & tools or fine-tuning. The eval provides a fixed target for deciding whether those changes improved the intended capability.
Containerized evals make this process faster. The task and environment remain stable while the agent configuration changes, so builders can swap models, tools, prompts, or complete agent versions and compare results directly. Multiple configurations can run in parallel.
Reproducible environments are critical to that signal. When an eval mirrors the relevant tools, data, permissions, state, and failure modes from production, builders get a stable testbed that is still representative of how the agent operates. They can experiment quickly without relying on changing production systems or writing to production state.
The resulting loop is:
mine traces -> identify a failure -> build an eval -> improve the agent -> rerun
## Try it today
The Eval Engineering Skill is available in the langchain-ai/langchain-skills repository.
Install the skill in Codex or Claude Code, open the repository containing the agent you want to evaluate, point to agent to a set of traces if available, and start with a simple prompt:
We’re looking forward to expanding this skill and building tooling to make it easier to automatically build evals and fit agents to them autonomously.
## 相关链接
- [Viv](https://x.com/Vtrivedy10)
- [@Vtrivedy10](https://x.com/Vtrivedy10)
- [90K](https://x.com/Vtrivedy10/status/2079976006644072796/analytics)
- [Harbor](https://github.com/harbor-framework/harbor)
- [langsmith-cli](https://docs.langchain.com/langsmith/langsmith-cli)
- [chat-langchain](https://github.com/langchain-ai/chat-langchain)
- [Harbor tasks](https://www.harborframework.com/docs/tasks)
- [Continual learning can be thought of as a continuous data mining problem](https://www.langchain.com/blog/improving-agents-is-a-data-mining-problem)
- [langchain-ai/langchain-skills repository.](https://github.com/langchain-ai/langchain-skills/tree/main/config/skills/eval-engineering)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:04 AM · Jul 23, 2026](https://x.com/Vtrivedy10/status/2079976006644072796)
- [90.6K Views](https://x.com/Vtrivedy10/status/2079976006644072796/analytics)
- [View quotes](https://x.com/Vtrivedy10/status/2079976006644072796/quotes)
---
*导出时间: 2026/7/23 10:56:36*
---
## 中文翻译
# 迈向评估工程的自动化
**作者**: Viv
**日期**: 2026-07-22T17:04:58.000Z
**来源**: [https://x.com/Vtrivedy10/status/2079976006644072796](https://x.com/Vtrivedy10/status/2079976006644072796)
---
今天我们发布了我们的评估工程技能，这是一项帮助编码代理利用仓库上下文和代理轨迹来构建评估的技能。
该技能会检查代理的结构，如果轨迹可用，则从中挖掘模式，并提出需要测试的能力。
该技能设计用于与用户进行交互，用户可以对提案提供反馈，并迭代地批准每个评估。最终产物是一组 Harbor 格式的可执行评估。
## 构建环境与任务
该技能首先读取仓库并映射代理的表面，包括提示词、模型、工具、技能、钩子等。它还会识别支持这些行为的数据和服务，例如 API 调用。
用户还可以将代理指向可以使用 langsmith-cli 等工具检索的轨迹。轨迹展示了工具在实际运行中的行为，例如它们的参数、结果和错误。这些观察到的契约有助于该技能在受控环境中重现相关的生产行为。
爬取仓库和轨迹让代理了解了哪些能力对代理很重要，因为它会提出评估任务。我们发现，与一次性生成相比，与用户交互能带来更好的评估接受度。用户从提议的评估方向中进行选择，并就诸如哪些工具和依赖项应该实时运行或需要模拟等问题提供指导。例如，产生成本或需要写入生产环境的工具调用可以被模拟，而不是在每次评估调用时都运行。
我们在我们的文档问答代理 chat-langchain 上测试了这个流程。对于该代理，环境需要一个通过代理搜索工具暴露的数据语料库，这些工具是基于生产代理建模的。任务包括从真实轨迹中提取的真实文档问题，以及一个使用黄金答案字符串和引用文档来检查答案的验证器。
## 评估设计是迭代的
我们发现，虽然代理有时能够一次性生成评估，但最好的评估来自于用户提供反馈并指定哪些能力值得在代理中进行测量。编码代理和技能提供了一个自然的接口，其中关于如何构建良好评估的领域知识被编码在内，用户可以随时间推移对其进行迭代。
例如，我们发现，在构建验证器时，第一个验证器很少是最终版本。改进它的一个有用方法是运行评估并检查结果的两端：
- 代理轨迹，包括其消息、工具调用和操作。
- 验证器轨迹、证据、推理和最终得分。
这有助于揭示任务或验证器设计是否正在衡量我们关心的内容，或者它是否可能被“奖励黑客”利用，即代理可能会走捷径。这些捷径可能包括过度引用不相关的来源以在评估中获得满分，声称它从未采取过的操作，利用暴露的答案材料，或者在不完成任务的情况下满足代理指标。观察代理解决问题的轨迹通常会揭示这些失败的根源。然后可以修改任务、环境和验证器并再次运行。
## 评估采用 Harbor 格式
该技能将评估构建为 Harbor 任务：
1. 指令：在开始时发送给代理的消息，用于描述任务
2. 环境：以 Dockerfile 形式给出，包含任务的设置，例如安装哪些工具或在文件系统中填充哪些数据
3. 验证器：用于对代理是否正确完成任务进行评分。
该技能将这些组件一起构建为一个 Harbor 任务：
```
evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
```
Harbor 在环境中运行代理，并记录其轨迹、产物、奖励和错误。然后可以对不同的模型、提示词、工具和代理版本运行相同的评估。
## 为什么这很重要
持续学习可以被认为是一个连续的数据挖掘问题，其中生产数据被用来构建评估，从而随时间推移改进代理。团队挖掘轨迹以找到重复的用户请求、错误、失败的工具调用和不正确的状态更改。这些将成为评估，以便可以衡量相同的行为并在未来防止它们。
评估是代理的训练数据。团队可以通过约束工程（例如更改提示词和工具或微调）使代理行为适应它们。评估为确定这些更改是否提高了预期能力提供了一个固定目标。
容器化的评估使这个过程更快。当代理配置更改时，任务和环境保持稳定，因此构建者可以交换模型、工具、提示词或完整的代理版本，并直接比较结果。多个配置可以并行运行。
可重现的环境对于该信号至关重要。当评估反映了生产环境中的相关工具、数据、权限、状态和失败模式时，构建者将获得一个仍然代表代理运行方式的稳定测试平台。他们可以快速进行实验，而无需依赖变化的生产系统或写入生产状态。
产生的循环是：
挖掘轨迹 -> 识别失败 -> 构建评估 -> 改进代理 -> 重新运行
## 立即试用
评估工程技能可在 langchain-ai/langchain-skills 仓库中获取。
在 Codex 或 Claude Code 中安装该技能，打开包含您想要评估的代理的仓库，如果可用，将代理指向一组轨迹，然后从一个简单的提示词开始：
我们期待扩展这一技能，并构建工具，以便更容易地自动构建评估并使代理自主适应它们。
## 相关链接
- [Viv](https://x.com/Vtrivedy10)
- [@Vtrivedy10](https://x.com/Vtrivedy10)
- [90K](https://x.com/Vtrivedy10/status/2079976006644072796/analytics)
- [Harbor](https://github.com/harbor-framework/harbor)
- [langsmith-cli](https://docs.langchain.com/langsmith/langsmith-cli)
- [chat-langchain](https://github.com/langchain-ai/chat-langchain)
- [Harbor tasks](https://www.harborframework.com/docs/tasks)
- [Continual learning can be thought of as a continuous data mining problem](https://www.langchain.com/blog/improving-agents-is-a-data-mining-problem)
- [langchain-ai/langchain-skills repository.](https://github.com/langchain-ai/langchain-skills/tree/main/config/skills/eval-engineering)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:04 AM · Jul 23, 2026](https://x.com/Vtrivedy10/status/2079976006644072796)
- [90.6K Views](https://x.com/Vtrivedy10/status/2079976006644072796/analytics)
- [View quotes](https://x.com/Vtrivedy10/status/2079976006644072796/quotes)
---
*导出时间: 2026/7/23 10:56:36*