# The Harness Is the Product
**作者**: Sarthak Dalabehera
**日期**: 2026-08-07T05:52:43.000Z
**来源**: [https://x.com/s7rthaks/status/2085605033316647408](https://x.com/s7rthaks/status/2085605033316647408)
---
In February 2025, AI created 0% of our merged pull requests. By November, agents were creating 15%. Then Opus 4.5 and GPT 5.2 hit a tipping point. We rolled out Claude Code and Codex enterprise plans to every engineer. Within three months, agents were writing 75% of merged PRs.
That graph changed everything about how we think about software engineering. Seemingly overnight, agents write the majority of our code. But our organizational and technical infrastructure hasn’t kept pace. The bottleneck moved.
It used to be: How do we write code faster?
Now it is: How do we prove the code works without a human reading every line?
This chapter is about the bet I’m taking for my team. Not “how to use AI more” but something harder: how to build the verification system that makes autonomous delivery possible. If you’re leading an engineering team where agents are writing significant code, this is the playbook.
## The five levels of coding autonomy
A useful mental model from Dan Shapiro’s “Five Levels” framework. Level 0 is autocomplete. Level 5 is the factory: agents write the code, the tests, the reviews, deploy, and monitor.
Most teams are at Level 3 or 4. Engineers use Claude Code daily. Agents write most of the code. But humans are still the primary reviewers, the primary deployers, the primary on-call responders. Every step between here and Level 5 involves the same question:
> What capability must exist in the harness before we can transition this step to agent-driven?
That question - not “how do we use AI more?” - is the strategic frame for every technical bet.
Code implementation is already agent-heavy. But look at the rest: PR review, deploy, on-call, remediation - all human-gated. Each one transitions to agent-driven only when the harness provides sufficient verification. The harness is the unlock for everything downstream.
## What is the harness
The harness is the combined system that proves agent output meets expectations without a human reading every line. It has four components:
Context - everything agents need to understand the system: architecture maps, runbooks, API docs, coding rules, domain knowledge. If it lives in a Slack thread or someone’s head, it doesn’t exist. If something matters, it lives in the repo.
Constraints - mechanical boundaries agents cannot cross: linters, type systems, permission gates, architectural rules. Not guidelines that agents might ignore, but hard walls enforced by tooling.
Verification - proving the output works: end-to-end scenarios, integration tests, holdout test suites that agents can’t trivially overfit, performance gates, satisfaction scoring for fuzzy outcomes.
Feedback loops - the system gets smarter over time: incidents become scenario tests, code review comments become linter rules, production monitors feed back into the harness. An incident that doesn’t become a test is a missed compounding opportunity.
The bottleneck is always the harness. As OpenAI’s team put it: “Our most difficult challenges now center on designing environments, feedback loops, and control systems.” Generation is solved. Verification is the hard problem.
## Six principles for decision-making
Every technical investment, every tool choice, every process change gets evaluated against these principles. They’re ordered by priority.
1. Verification over generation
We can already generate code quickly. The constraint is proving it works and ensuring each feature justifies its ongoing maintenance cost. Every investment should be evaluated by how much it increases confidence in agent output, not how much faster agents write code.
2. What agents can’t see doesn’t exist
Slack threads, design decisions in someone’s head, tribal knowledge from three years ago - none of it exists to an agent. If something matters, it lives in the repo as documentation. Think of repo docs as onboarding material for your agent workforce.
3. Encode judgment into the harness
Human judgment about code quality, architecture, and correctness is expensive to apply at scale. When you identify a pattern, a constraint, or a quality standard, encode it: linter rules, test scenarios, architectural boundaries, performance SLOs. Apply it once mechanically instead of catching it in every PR review.
4. Behavior over code
Don’t rely on humans reading diffs as the primary quality gate. Rely on externally observable behavior proven by the harness - end-to-end scenarios that exercise the real system. Humans still intervene when risk is high or harness coverage is low, but the default is automated verification.
5. Close the loop
Every system that generates information - incidents, alerts, test failures, customer feedback - should feed back into the harness. An incident that doesn’t become a scenario test is a missed compounding opportunity. A code review comment that doesn’t become a linter rule will be repeated forever.
6. Legacy code is a specification
Your legacy monolith isn’t an obstacle to autonomous delivery. It’s a living specification. Given a comprehensive enough verification harness, agents could rewrite subsystems from scratch. The harness defines the contract. The implementation is fungible.
## What changes, what stays the same
Some things we currently do exist to serve humans and become less relevant:
Code style debates become irrelevant when agents rewrite freely. Architectural boundaries matter. Formatting preferences don’t.
PR review as quality gate gives way to automated verification. Human review shifts to reviewing specs, scenarios, and harness coverage - not diffs.
Coordination overhead decreases when agents batch PRs, manage feature flags, stack changes, and work 24 hours a day.
Some things become more important:
Specification quality. The spec is the primary human artifact now. If the spec is wrong, the code will be wrong at speed. A vague spec with human engineers meant slow, messy code. A vague spec with agents means fast, confidently wrong code.
Scenario coverage. The breadth and realism of your end-to-end scenarios determines your confidence ceiling. This is now the most important technical investment a team can make.
Architectural constraints. Rigid boundaries, enforced mechanically, are what allow agents to move fast safely. The more you can encode into the system, the less you need humans in the loop.
Observability. Agents need logs, metrics, and traces - but programmatically accessible, not through dashboards. If your monitoring requires a human staring at Grafana, agents can’t use it for automated remediation.
## The risks worth tracking
Cognitive debt
The gap between system complexity and human understanding. As agents change code faster than humans can comprehend it, you accumulate a compounding comprehension gap. Mitigations: understand the system through specs, scenarios, and architecture docs, not code. The code is allowed to be opaque, just like compiled binaries. Keep the behavior layer current. Use agents to generate documentation - they can read the entire codebase each time.
Security and correctness
Agent-written code can introduce bugs, security vulnerabilities, performance degradation, and data loss. Mitigations: encode security rules that get picked up by every agent, create dedicated review agents with security-specific prompts, start with low-risk internal tools and expand to external-facing code as you gain confidence.
Over-speccing
When anyone in the organization can ask agents to add features, you risk a software explosion. Each new requirement is simple in isolation, but complexity compounds. Even with free code generation, every feature has an ongoing maintenance and verification cost. The highest quality, fastest software is no software at all.
Token costs
Agent-heavy teams carry a significant AI budget across Claude Code, Codex, and other inference providers. As adoption grows, this scales. But with three competing frontier providers and market forces, prices stay competitive. And open-source models, though they lag frontier performance, exist at a fraction of the cost for many subtasks like compaction, validation, and code search.
## The job description is changing
The engineer’s job shifts from writing and reviewing code to:
1. Specifying intent - defining what should change, clearly enough that agents can converge without clarification
2. Building verification - scenarios, holdouts, satisfaction scoring, performance gates
3. Designing feedback loops - incident to scenario to prevention, automatically
4. Supervising execution - orchestrating agents, escalating when harness coverage is low
We have decades of experience building software where humans write the code. Most of our engineering best practices, tools, and organizational structures assume that. Agents now write the majority of code, and that percentage will approach 100%.
The companies that build the best harnesses will ship faster and with higher quality. The harness is the product. Let’s build.
## References
- The Five Levels: from Spicy Autocomplete to the Dark Factory - Dan Shapiro’s framework for coding autonomy levels
- Harness Engineering: leveraging Codex in an agent-first world - OpenAI’s take on building verification systems for agent output
- StrongDM Software Factory - StrongDM’s implementation of the software factory pattern
- How StrongDM’s AI team builds serious software without even looking at the code - Simon Willison’s deep dive into the factory approach
- The Future of Software Engineering - ThoughtWorks retreat key takeaways on where the industry is heading
## 相关链接
- [Sarthak Dalabehera](https://x.com/s7rthaks)
- [@s7rthaks](https://x.com/s7rthaks)
- [2.5K](https://x.com/s7rthaks/status/2085605033316647408/analytics)
- [The Five Levels: from Spicy Autocomplete to the Dark Factory](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/)
- [Harness Engineering: leveraging Codex in an agent-first world](https://openai.com/index/harness-engineering/)
- [StrongDM Software Factory](https://factory.strongdm.ai/)
- [How StrongDM’s AI team builds serious software without even looking at the code](https://simonwillison.net/2026/Feb/7/software-factory/)
- [The Future of Software Engineering](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:52 PM · Aug 7, 2026](https://x.com/s7rthaks/status/2085605033316647408)
- [2,549 Views](https://x.com/s7rthaks/status/2085605033316647408/analytics)
---
*导出时间: 2026/8/8 11:14:49*
---
## 中文翻译
# 测试验证框架即产品
**作者**: Sarthak Dalabehera
**日期**: 2026-08-07T05:52:43.000Z
**来源**: [https://x.com/s7rthaks/status/2085605033316647408](https://x.com/s7rthaks/status/2085605033316647408)
---
2025年2月，我们合并的拉取请求（PR）中，由 AI 生成的比例还是 0%。到了11月，智能体已经能生成 15%。随后，Opus 4.5 和 GPT 5.2 迎来了引爆点。我们为每位工程师都部署了 Claude Code 和 Codex 企业版。短短三个月内，由智能体撰写的合并 PR 比例飙升到了 75%。
那张图表彻底改变了我们对软件工程的认知。仿佛就在一夜之间，大多数代码都变成了由智能体编写。但我们的组织和技术基础设施却未能跟上这一步伐。瓶颈发生了转移。
过去的问题是：我们如何更快地编写代码？
现在的问题则是：在人类无需逐行审查代码的情况下，我们如何证明这些代码是行之有效的？
本章主要讲述我正在为自己的团队进行的一场押注。这不是探讨“如何更多地使用 AI”，而是 tackling 一项更艰难的挑战：如何构建一个让自主交付成为可能的验证系统。如果你正在带领一个由智能体编写大量代码的工程团队，这就是你的实操指南。
## 编码自动化的五个等级
这是源自 Dan Shapiro “五个等级”框架的一个实用心智模型。等级 0 是自动补全。等级 5 则是超级工厂：由智能体编写代码、测试、进行代码审查、部署并实施监控。
大多数团队目前处于等级 3 或 4。工程师们每天都在使用 Claude Code。智能体编写了大部分代码。但人类依然是主要的审查者、主要的部署者以及主要的值班响应人员。从现阶段迈向等级 5 的每一步，都绕不开同一个问题：
> 在我们将某个环节交由智能体驱动之前，测试框架中必须具备什么样的能力？
这个问题——而不是“我们该如何更多地使用 AI？”——才是所有技术决策的战略框架。
代码实现环节已经大量依赖智能体了。但看看其他环节：PR 审查、部署、值班响应、故障修复——依然全都被人类卡着关口。只有当测试框架提供了充分的验证能力时，这些环节才能向智能体驱动过渡。测试框架是解锁所有下游环节的万能钥匙。
## 什么是测试框架
测试框架是一个综合系统，它能在无需人类逐行审查的情况下，证明智能体的输出符合预期。它包含四个组成部分：
**上下文** —— 智能体理解系统所需的一切信息：架构图、操作手册、API 文档、编码规范、领域知识。如果这些信息只存在于 Slack 消息或某个人的脑子里，那对智能体来说就等于不存在。但凡重要信息，都必须存在于代码仓库中。
**约束** —— 智能体无法逾越的机械边界：代码检查工具、类型系统、权限网关、架构规则。这不是智能体可能会忽视的指导原则，而是由工具强制执行的坚硬壁垒。
**验证** —— 证明输出结果有效：端到端场景、集成测试、智能体无法轻易过拟合的保留测试套件、性能关卡，以及针对模糊结果的满意度评分。
**反馈闭环** —— 系统随着时间推移变得越来越聪明：将事故转化为场景测试，将代码审查意见转化为代码检查规则，将生产环境的监控结果反馈到测试框架中。一个未能转化为测试的故障，就是一次错失的复利机会。
瓶颈永远在于测试框架。正如 OpenAI 团队所言：“我们现在最棘手的挑战，集中在如何设计环境、反馈闭环和控制系统上。”生成问题已经解决。验证才是真正的难题。
## 决策的六大原则
每一项技术投资、每一次工具选择、每一个流程变更，都要根据这些原则进行评估。它们按优先级排序。
1. 验证重于生成
我们已经能够快速生成代码了。现在的制约因素在于证明代码有效，并确保每个功能都物有所值，能覆盖其持续的维护成本。评估任何一项投资的标准，不应是它能让智能体写代码的速度提升多少，而应是它能多大程度上提升人们对智能体输出结果的信心。
2. 智能体看不见的就不存在
Slack 讨论串、某人脑海中的设计决策、三年前流传下来的部落知识——这些对智能体来说统统不存在。但凡重要的东西，就必须作为文档存放在代码仓库中。请把代码仓库里的文档，当成是为你的智能体劳动力准备的入职培训材料。
3. 将判断力编码到测试框架中
在大规模应用中，人类对代码质量、架构和正确性的判断是非常昂贵的。一旦你识别出某种模式、约束或质量标准，就把它编码进去：代码检查规则、测试场景、架构边界、性能 SLO（服务级别目标）。与其在每次 PR 审查中去揪出这些问题，不如用机械化的方式一次性解决。
4. 行为重于代码
不要把人类审查代码差异作为主要的质量关卡。应当依赖那些由测试框架证明过的、外部可观察的行为——也就是能够演练真实系统的端到端场景。当风险较高或测试框架覆盖率较低时，人类仍然需要介入，但默认机制必须是自动化验证。
5. 闭环管理
每一个能生成信息的系统——如故障、警报、测试失败、客户反馈——都应该将信息反馈回测试框架中。一个没有转化为场景测试的故障，是一个错失的复利机会。一条没有转化为检查规则的代码审查意见，将会被永远重复触发。
6. 遗留代码即规范
你那陈旧的单体架构并不是实现自主交付的绊脚石。它是一份活生生的规范说明。只要有足够全面的验证测试框架，智能体完全可以从头重写各个子系统。测试框架定义了契约，而具体的代码实现是可以替换的。
## 什么变了，什么没变
我们现在所做的一些事情是为了服务于人类，将来它们的相关性会逐渐降低：
当智能体可以自由重写代码时，关于代码风格的争论就变得毫无意义。架构边界很重要。格式偏好则不然。
作为质量关卡的 PR 审查将让位于自动化验证。人类的审查重点将转向审查规范、场景和测试框架的覆盖率——而不是代码差异。
当智能体能够批量处理 PR、管理功能开关、堆叠更改并全天候 24 小时工作时，协调的开销就会随之减少。
而有些事情会变得更加重要：
规范的质量。如今，规范文档是最核心的人工产物。如果规范本身是错的，代码就会以极快的速度出错。对人类工程师来说，一份含糊其辞的规范意味着代码写得既缓慢又糟糕。但对于智能体来说，一份含糊其辞的规范意味着代码会以飞快的速度，满怀“自信”地出错。
场景覆盖率。端到端场景的广度和真实性决定了你能达到的信心上限。这已成为目前一个团队所能做出的最重要的技术投资。
架构约束。由机械手段强制执行的刚性边界，是允许智能体安全快速运作的保障。你编码到系统中的规则越多，你所需的人类干预就越少。
可观测性。智能体需要日志、指标和链路追踪——但前提是它们必须能通过编程方式获取，而不是仅仅通过仪表盘展示。如果你的监控系统还需要人死死盯着 Grafana 才能发现问题，那智能体就无法利用它来进行自动化的故障修复。
## 值得关注的风险
认知负债
系统复杂性与人类理解之间的鸿沟。随着智能体修改代码的速度超过人类的理解速度，你会不断积累出一个呈复利增长的认知鸿沟。应对措施：通过规范、场景和架构文档（而非代码本身）来理解系统。代码可以是不透明的，就像编译后的二进制文件一样。保持行为层的不断更新。利用智能体来生成文档——它们每次都能通读整个代码库。
安全性与正确性
智能体编写的代码可能会引入 Bug、安全漏洞、性能退化以及数据丢失。应对措施：将安全规则编码其中，确保每个智能体都能检测到；创建使用特定安全提示词的专用审查智能体；从低风险的内部工具做起，随着信心的积累，再逐步扩展到面向外部的代码。
过度规范化
当组织中的任何人都能随意要求智能体添加新功能时，你就有面临软件膨胀的风险。孤立地看，每个新需求都很简单，但复杂性是会叠加的。即使代码生成是免费的，每个功能都会带来持续的维护和验证成本。质量最高、速度最快的软件，是压根就不存在的软件。
Token 成本
重度依赖智能体的团队，在 Claude Code、Codex 以及其他推理模型提供商那里，都背负着庞大的 AI 预算。随着应用的深入，这笔费用会水涨船高。不过，得益于三大前沿模型提供商的竞争以及市场规律的调节，价格依然保持在合理区间。而且，尽管开源模型在绝对性能上稍逊一筹，但对于许多诸如上下文压缩、验证和代码搜索等子任务来说，其成本只有前者的九牛一毛。
## 职位描述正在改变
工程师的工作重心，正在从编写和审查代码，转变为：
1. **表达意图** —— 清晰地定义需要修改哪些内容，确保智能体能够在无需进一步澄清的情况下直接执行。
2. **构建验证体系** —— 场景设计、保留测试套件、满意度评分、性能关卡设置。
3. **设计反馈闭环** —— 自动实现从“事故”到“场景测试”再到“预防机制”的闭环。
4. **监督执行** —— 统筹调度智能体，在测试框架覆盖率不足时及时进行人工干预升级。
在过去的几十年里，我们在构建软件时一直遵循着由人类编写代码的既定模式。我们的大多数工程最佳实践、工具以及组织架构都是建立在这一假设之上的。而现在，智能体已经编写了大部分代码，且这个比例将无限趋近于 100%。
谁构建了最好的测试框架，谁就能以更快的速度交付更高质量的产品。测试框架即产品。让我们开始构建吧。
## 参考资料
- The Five Levels: from Spicy Autocomplete to the Dark Factory - Dan Shapiro 关于编码自动化等级的框架
- Harness Engineering: leveraging Codex in an agent-first world - OpenAI 关于为智能体输出构建验证系统的见解
- StrongDM Software Factory - StrongDM 对软件工厂模式的实践落地
- How StrongDM’s AI team builds serious software without even looking at the code - Simon Willison 对该工厂模式的深入剖析
- The Future of Software Engineering - ThoughtWorks 闭门研讨会关于行业发展趋势的核心要点
## 相关链接
- [Sarthak Dalabehera](https://x.com/s7rthaks)
- [@s7rthaks](https://x.com/s7rthaks)
- [2.5K](https://x.com/s7rthaks/status/2085605033316647408/analytics)
- [The Five Levels: from Spicy Autocomplete to the Dark Factory](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/)
- [Harness Engineering: leveraging Codex in an agent-first world](https://openai.com/index/harness-engineering/)
- [StrongDM Software Factory](https://factory.strongdm.ai/)
- [How StrongDM’s AI team builds serious software without even looking at the code](https://simonwillison.net/2026/Feb/7/software-factory/)
- [The Future of Software Engineering](https://www.thoughtworks.com/content/dam/thoughtworks/documents/report/tw_future%20_of_software_development_retreat_%20key_takeaways.pdf)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:52 PM · Aug 7, 2026](https://x.com/s7rthaks/status/2085605033316647408)
- [2,549 Views](https://x.com/s7rthaks/status/2085605033316647408/analytics)
---
*导出时间: 2026/8/8 11:14:49* 
