利用 Auto-research 提升 Agent Harness 技能

✍ Alok Bishoyi🕐 2026-05-28📦 7.9 KB 🟢 已读 𝕏 文章列表

本文介绍了 AI Agent 自我优化的新趋势：通过优化 Harness 层（即 Skills）而非模型本身来提升性能。作者详细介绍了名为 evo 的开源优化工具，它利用并行探索、树搜索和门控机制，自动对 Skills 进行实验和迭代。文章展示了在 SealQA 基准测试中，evo 将准确率从 5/20 提升至 11/20 的实战案例，强调了“模型能力 x 优秀的 Harness x 紧密的验证循环”这一杠杆公式的重要性。

# Using Autoresearch to improve harness skills ( with detailed example run ) **作者**: Alok Bishoyi **日期**: 2026-05-27T12:18:57.000Z **来源**: [https://x.com/alokbishoyi97/status/2059610305408462898](https://x.com/alokbishoyi97/status/2059610305408462898) --- ![Image](/doc/2283/image/img_f7bc814f084c.jpg) ## self-improving agents are here The most interesting shift in AI right now is that agents are now improving themselves. You point one at a problem, leave it alone, and come back to find it has run experiments on its own behavior, kept what worked, discarded what didn't, and produced a version of itself that scores meaningfully higher on the task you cared about. The underlying model never changed. The wrapper around it did. ## what's actually changing The thing being optimized is the layer between you and the model. Most agent harnesses now expose some of their capabilities as a Skill. a Skill is a markdown file with a short description (of when this Skill is relevant) and a body (what the agent should do once it's loaded). Claude Code has Skills. Codex has them. Cursor exposes a similar surface. so does OpenClaw and a handful of other harnesses. The implementation details differ, but the idea is the same: the agent's procedure for a class of tasks is a plain markdown file. Anyone - including another agent - can read it, edit it, and ship a new version. This is what makes self-improvement tractable. The parameter space isn't model weights, which require a training run, a GPU cluster, and a labeled dataset. The parameter space is markdown in this case. An LLM can write markdown. Another LLM can grade it. ![Image](/doc/2283/image/img_622e5cc8a9c8.jpg) we used a Claude Code harness and Claude Skills in the run below, but nothing about the approach is Claude-specific. Point the same loop at a Codex skill or a Cursor agent definition and you get the same machine. ## what evo is evo is the loop. You give it a codebase, a definition of "better", and a budget. It sets up a benchmark (or uses one you already have), generates hypotheses about what to change, runs each hypothesis in its own isolated workspace, scores the result, and keeps a tree of attempts. The branches that score higher get extended. The branches that don't get pruned. An auditor checks each accepted change for cheating - verifying the optimizer isn't smuggling test answers into the parameter file. Three things make evo different from a single greedy hill-climb thats typical in autoresearch implementations : - parallel exploration - Multiple agents run at the same time, each in its own sandbox, each trying a different hypothesis. They don't have to wait their turn. - tree search, not linear - evo doesn't only keep the single highest-scoring branch. It keeps branches that win on different slices of the task - specialists alongside generalists. When those specialists can be merged, the result is usually better than either parent alone. - gates - any pass/fail check you can write (a regression test, a held-out slice, a no-cheating audit) is a gate. An experiment that fails a gate gets discarded even if its score is the best so far. Without gates, optimization loops find ways to game the metric. evo is open source. We use it ourselves. It's also the engine underneath the autoresearch platform that we are launching ( details below ) ![Image](/doc/2283/image/img_ff806646e6a4.jpg) ## how you use evo setup is two commands: ``` uv tool install evo-hq-cli # the optimizer evo install claude-code # wires them together ``` that's the full install. swap claude-code for codex, cursor, openclaw, hermes, pi or whichever host you already use. then two commands run the loop. /evo:discover : is the one-time setup. You point it at a codebase and tell it what you want better. It explores the repo, figures out a benchmark (or instruments one if none exists), proposes the optimization target, and registers any gates that should run on every experiment. /evo:optimize is the loop. It orchestrates agent runs in parallel, the per-agent budget, and the stopping condition (usually: stop when N consecutive rounds produce no improvement) From user's POV that's it. The dashboard streams scores and traces as experiments complete. You can leave it running for an hour or a week.When it's done, you get back the parameter file (Skill, prompt, config, whatever the target was) that scored highest, plus the full record of every hypothesis it tried It works on Claude Code, Codex, Cursor, OpenClaw, and a few others. Runs locally by default; you can swap to a cloud sandbox provider if you want experiments running in parallel without burning your laptop. ## what we did To make this concrete we pointed evo at a hard benchmark ( SealQA )and let it run. The target was a public set of 20 fact-seeking questions where the obvious search results disagree, and the right answer takes some reasoning to extract. The kind of question where a vanilla agent confidently answers wrong: "which country joined the EU most recently that also borders Russia" - the joint condition trips up most single-shot searches. "how many active volcanoes are currently erupting" - the answer depends on whether you mean today, this month, or this year, and the top sources disagree. Baseline: Claude Code with web search, no custom Skills. 5/20 right We set evo loose on the `.claude/skills/` directory. 50 experiments later, ending state: one 145-line Skill file with five trigger-gated sub-protocols, one for each question shape evo discovered the agent was systematically getting wrong. Score: 11/20. More than doubled. The underlying model did not change. The loop produced the wrapper. ## why this matters here is the how we model the leverage of an agentic system : > model capability × good harness × tight verification loops Until recently only the first factor moved on a regular cadence. New model, new behavior. Now the other two factors are moving too. and they're moving on a schedule you control, against an objective you defined, in a parameter space that anyone can update. And these are optimized against your benchmark, over your dataset This is what i mean when i say AI's 4-minute mile is here. The breakthrough isn't a bigger model. It's a tighter loop around the one we already have. We are building the evo platform to make this loop available to anyone, not just teams who can build the orchestration themselves. You define what "better" means for your AI. You point the platform at your repo, your agent stack, or your Skills directory. It builds the harness, runs the experiments in parallel, scores them honestly, and returns the version that scored highest along with the full audit trail of how it got there. The longer arc is continuous tuning. Users don't actually want a one-time autoresearch run - they want their systems to stay tuned as the problem drifts, as the model updates, as new failure modes appear. The goal is for the loop to run 24/7, watching the benchmark and keeping the configuration current. Systems, code, agents, Skills, models - anything where "better" has a definition. The full record of the SealQA run - every experiment, the final Skill, every hypothesis evo tried, the live dashboard - is here . So is the way to register if you want this loop running on something of yours. ## 相关链接 - [Alok Bishoyi](https://x.com/alokbishoyi97) - [@alokbishoyi97](https://x.com/alokbishoyi97) - [49K](https://x.com/alokbishoyi97/status/2059610305408462898/analytics) - [evo platform](https://evo-hq.com/) - [is here](https://evo-hq.com/shared/x/9k3b8t2qhr/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [8:18 PM · May 27, 2026](https://x.com/alokbishoyi97/status/2059610305408462898) - [49.4K Views](https://x.com/alokbishoyi97/status/2059610305408462898/analytics) - [View quotes](https://x.com/alokbishoyi97/status/2059610305408462898/quotes) --- *导出时间: 2026/5/28 09:15:35* --- ## 中文翻译 # 使用自动研究提升 Harness 技能（附带详细运行示例） **作者**: Alok Bishoyi **日期**: 2026-05-27T12:18:57.000Z **来源**: [https://x.com/alokbishoyi97/status/2059610305408462898](https://x.com/alokbishoyi97/status/2059610305408462898) --- ![Image](/doc/2283/image/img_f7bc814f084c.jpg) ## 自我改进的 Agent 已经到来目前 AI 领域最有趣的转变在于，Agent 现在可以自我改进。你只需把一个 Agent 对准某个问题，然后不管它，过一会儿你会发现它已经针对自己的行为进行了实验，保留了有效的方法，丢弃了无效的，并产生了一个在你关注的任务上得分显著更高的新版本。底层的模型从未改变。改变的是它的外壳。 ## 实际改变的是什么被优化的对象是你和模型之间的那一层。大多数 Agent Harness 现在将其部分能力作为“Skill”暴露出来。Skill 是一个 Markdown 文件，包含简短的描述（说明该 Skill 何时适用）和主体（Agent 加载后应该执行的操作）。 Claude Code 有 Skills。Codex 也有。Cursor 暴露了类似的接口。OpenClaw 和其他一些 Harness 也是如此。尽管实现细节各不相同，但核心思想是一样的：Agent 处理一类任务的流程就是一个普通的 Markdown 文件。任何人——包括另一个 Agent——都可以读取、编辑并发布新版本。这就是自我改进变得切实可行的原因。这里的参数空间不是模型权重——那需要训练运行、GPU 集群和标注数据集。这里的参数空间是 Markdown。大语言模型（LLM）可以编写 Markdown。另一个 LLM 可以给它打分。 ![Image](/doc/2283/image/img_622e5cc8a9c8.jpg) 在下面的运行中，我们使用了 Claude Code Harness 和 Claude Skills，但这种方法并非仅限于 Claude。将同样的循环指向 Codex Skill 或 Cursor Agent 定义，你会得到相同的机制。 ## evo 是什么 evo 就是那个循环。你给它一个代码库、“更好”的定义以及预算。它会设置一个基准（或使用你已有的基准），生成关于修改内容的假设，在各自隔离的工作空间中运行每个假设，对结果打分，并维护一棵尝试树。得分较高的分支会被延伸。得分不高的分支会被修剪。审计员会检查每个被接受的变更是否存在作弊行为——验证优化器没有将测试答案偷偷塞进参数文件中。有三点让 evo 区别于典型自动研究实现中的单步贪婪爬山算法： - **并行探索** - 多个 Agent 同时运行，每个都在自己的沙箱中，尝试不同的假设。它们不必排队等待。 - **树状搜索，而非线性** - evo 不仅保留得分最高的单一分支。它还保留在任务不同切片上获胜的分支——专家与通才并存。当这些专家可以被合并时，结果通常优于任一父代。 - **关卡** - 任何你能编写的通过/失败检查（回归测试、保留切片、防作弊审计）都是一个关卡。即使某次实验的得分是迄今为止最高的，如果未通过关卡，也会被丢弃。没有关卡，优化循环就会找到对策来通过指标。 evo 是开源的。我们自己也在使用它。它也是我们要发布的自动研究平台底层的引擎（详情见下文）。 ![Image](/doc/2283/image/img_ff806646e6a4.jpg) ## 如何使用 evo 安装只需两条命令： ``` uv tool install evo-hq-cli # 优化器 evo install claude-code # 将它们连接起来 ``` 这就是完整的安装过程。你可以把 claude-code 换成 codex、cursor、openclaw、hermes、pi 或任何你已经使用的主机。然后运行循环需要两条命令。 `/evo:discover`：这是一次性设置。你将代码库指向它，并告诉它你想改进什么。它会探索仓库，找出基准（如果不存在则创建一个），提出优化目标，并注册应该在每次实验中运行的关卡。 `/evo:optimize`：这是循环。它协调并行的 Agent 运行、每个 Agent 的预算以及停止条件（通常是：当连续 N 轮没有改进时停止）。从用户的角度来看，这就够了。当实验完成时，仪表板会流式传输得分和追踪信息。你可以让它运行一小时或一周。完成后，你会得到得分最高的参数文件（Skill、提示词、配置，无论目标是什么），以及它尝试过的每个假设的完整记录。它适用于 Claude Code、Codex、Cursor、OpenClaw 和其他一些工具。默认在本地运行；如果你希望在不耗尽笔记本资源的情况下并行运行实验，可以切换到云沙箱提供商。 ## 我们做了什么为了具体展示，我们将 evo 对准了一个高难度的基准，并让它运行。目标是一组包含 20 个事实性查询问题的公开数据集，其中明显的搜索结果相互矛盾，正确答案需要一些推理才能提取。这类问题通常会让普通的 Agent 自信地答错：“哪个最近加入欧盟的国家也与俄罗斯接壤”——这种联合条件会绊倒大多数单次搜索。“目前有多少活火山正在喷发”——答案取决于你是指今天、这个月还是今年，而且主要信息源各执一词。基线：带网络搜索的 Claude Code，无自定义 Skills。正确率 5/20。我们将 evo 放在 `.claude/skills/` 目录中运行。50 次实验后，最终状态：一个 145 行的 Skill 文件，包含五个受触发器控制的子协议，对应 evo 发现 Agent 系统性答错的每种问题形状。得分：11/20。翻了一倍多。底层模型没有改变。循环产生的是外壳。 ## 为什么这很重要以下是我们对 Agent 系统杠杆作用的建模： > 模型能力 × 良好的 Harness × 紧密的验证循环直到最近，只有第一个因素会定期变动。新模型，新行为。现在其他两个因素也在变动。而且它们按照你控制的进度变动，针对你定义的目标，在任何人都可以更新的参数空间中变动。并且这些是根据你的基准、在你的数据集上进行优化的。这就是我说 AI 的“4分钟英里”时刻已经到来的意思。突破不是更大的模型。它是围绕我们现有模型的更紧密的循环。我们正在构建 evo 平台，以让任何人都能使用这个循环，而不仅仅是那些能自己构建编排流程的团队。你定义你的 AI 中“更好”意味着什么。你将平台指向你的仓库、Agent 栈或 Skills 目录。它构建 Harness，并行运行实验，诚实地打分，并返回得分最高的版本以及它是如何达到这一点的完整审计追踪。更长远的愿景是持续调优。用户实际上并不想要一次性的自动研究运行——他们希望随着问题的偏移、模型的更新、新失败模式的出现，他们的系统能保持调优。目标是让循环 24/7 运行，监视基准并保持配置是最新的。系统、代码、Agent、Skills、模型——任何“更好”有定义的事物。 SealQA 运行的完整记录——每个实验、最终的 Skill、evo 尝试的每个假设、实时仪表板——都在这里。如果你想在你自己的东西上运行这个循环，注册方式也在这里。 ## 相关链接 - [Alok Bishoyi](https://x.com/alokbishoyi97) - [@alokbishoyi97](https://x.com/alokbishoyi97) - [49K](https://x.com/alokbishoyi97/status/2059610305408462898/analytics) - [evo platform](https://evo-hq.com/) - [is here](https://evo-hq.com/shared/x/9k3b8t2qhr/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [8:18 PM · May 27, 2026](https://x.com/alokbishoyi97/status/2059610305408462898) - [49.4K Views](https://x.com/alokbishoyi97/status/2059610305408462898/analytics) - [View quotes](https://x.com/alokbishoyi97/status/2059610305408462898/quotes) --- *导出时间: 2026/5/28 09:15:35*

🔗 相关文档