# The AI Agent Complexity Ratchet: Why 90% Test Coverage Is Required
**作者**: Garry Tan
**日期**: 2026-05-12T05:03:37.000Z
**来源**: [https://x.com/garrytan/status/2054064931515855118](https://x.com/garrytan/status/2054064931515855118)
---
I've been coding with AI for the past year. Not just prompting -- building real software. Two open-source projects: GStack, which makes AI coding agents better, and GBrain, which turns everything you read and write into a searchable knowledge base your AI can use. Between them, about 970,000 lines of code and 665 test files. Pretty much all written by Claude Code and Codex at my direction (15 simultaneous Conductor sessions most of the time).
Last week I merged fourteen pull requests in 72 hours. Nearly 29,000 lines of new code. Each release was better tested than the one before it.
That's supposed to be impossible. Speed and quality are supposed to trade off. Ship fast, break things. Move slow, ship right. Pick one.
You don't have to pick anymore. The unlock is 90% test coverage -- and AI agents made it free to get there. For fifty years, that level of verification cost too much human willpower to sustain. Now the agent writes the tests alongside the code. The result is what I call the complexity ratchet: a system that can only get better, never worse.
(This is the seventh in a series about building with AI: 1 2 3 4 5 6)
## Software used to be brittle
For fifty years, the whole discipline of software engineering was organized around one idea: prevent errors, because errors are catastrophic.
You had to get the code right the first time. Miss one edge case and you crash in production. Ship a bad database migration and you lose customer data. Write a function that does something subtle, and when the one person who understands it quits, nobody knows why it works. The whole system depended on humans being careful, and humans are not careful. So we built elaborate processes -- code reviews, staging environments, QA teams, release trains -- all designed to catch mistakes before they reached users.
It kind of worked. But it was slow. And it meant that the complexity of any software system had a hard ceiling: the number of things one team could hold in their heads simultaneously.
## Now software is squishy
I don't mean sloppy. I mean resilient in a way that wasn't possible before.
When I say "the models are here," I mean that AI coding agents -- Claude, GPT, Codex, and the ecosystem growing around them -- can now read code, understand context, diagnose errors, and write fixes. Not perfectly. But well enough that the error model for software has changed.
The migration breaks? The agent reads the error message, understands the database schema history across 45 versions, writes the fix, writes the test. The file sync hangs on a million symlinks? Agent diagnoses the parser timeout, bounds it at 30 seconds, ships the fix with tests. An extraction pipeline has an attribution bug? A cross-model evaluation catches it, the prompt gets iterated, enforcement gets added at the database layer.
For most code-level errors -- logic bugs, parsing failures, broken edge cases -- agents can now diagnose and fix them in the next turn. That's genuinely new. The errors that remain catastrophic are the ones that destroy state: bad migrations on production data, security holes exploited before detection, privacy leaks that can't be un-leaked. The ratchet helps here too (good tests catch most of these before production) but the real shift is that the vast majority of errors in a codebase are the fixable kind.
This is a phase change for how software gets built. But it only works if you have the ratchet.
## The Agent Complexity Ratchet
A ratchet is a mechanism that allows motion in one direction only. A socket wrench turns a bolt forward and prevents it from turning back. That's the metaphor.
In agent-coded software, every coding session with an AI agent adds three things to the codebase:
1. Tests that encode what "correct" means -- automated checks that run every time someone changes the code, and fail loudly if the change breaks something
2. Documentation that records why decisions were made -- not just what the code does, but the reasoning and tradeoffs behind it
3. Evaluation results that establish quality thresholds -- structured assessments of output quality with scores, so you know if the next version is better or worse
The next time an agent works on the codebase, it loads all three into its context window (the text the AI can see and reason about). It can't regress below the test suite -- the tests would fail. It can't ignore the documentation -- it's right there in context. It can't ship quality below the evaluation baseline -- the scores are recorded.
The quality floor goes up with every turn. Forward-only motion. That's the ratchet.
## What this looks like in practice
I'll make this concrete. GBrain is a knowledge system I'm building -- it gives AI agents long-term memory by storing, indexing, and searching through a person's notes, meetings, conversations, and research. Think of it as a second brain that your AI assistant can actually read.
One of its features is epistemological extraction: it reads through thousands of pages and extracts who believes what, with what confidence, over time. "Garry thinks Bitcoin will hit $300K (confidence: 0.45)." "Jared thinks this startup has strong retention (confidence: 0.80)." Like that, but across 28,000 pages.
The first extraction run pulled 100,720 claims. I used a cross-model evaluation to grade the quality -- I had GPT-5.5 and Claude both independently score the output. Overall: 6.8 out of 10.
The biggest problem? Something I call holder confusion. Take the claim "AI will replace 80% of software engineers by 2027." Who holds that belief? Is it the person who wrote it? Is it someone they're quoting? Or is it the system's analysis engine, which inferred it from a podcast transcript? Version 1 got this distinction wrong 35% of the time. That matters -- if you're building a system that tracks what people believe, you need to know WHO believes it.
So the evaluation results got documented. Six specific failure modes got identified. The version 2 prompt addressed all six. Weight rounding (the confidence scores) got enforced at the database layer -- no more false precision like 0.74 when 0.75 is the honest answer. Seventeen tests locked in the contract.
Now no future version of the extraction can ship without those 17 tests passing. Nobody has to remember why weight rounding matters or what holder confusion is. The tests remember.
The quality floor went up permanently. That's one turn of the ratchet.
## Why most vibecoded projects die
"Vibecoding" is Andrej Karpathy's term for coding with AI by describing what you want in natural language and letting the model generate the code. It's powerful and it's how I build. But from what I've seen across YC applications and open-source repos, most vibecoded projects that skip tests start falling apart once they reach moderate complexity -- a few thousand lines, a handful of interacting features.
They skip the ratchet. No tests, no docs, no evals. The agent adds complexity but nothing prevents regression. Every new feature has a chance of breaking an old one, and without tests, you don't find out until a user reports it. By version 0.5 the codebase is a haunted house where every change breaks something unexpected. Then the developer writes a blog post about how AI coding doesn't work.
AI coding works fine. They just didn't build the ratchet.
You could argue that the kind of person who writes tests is also the kind who writes good architecture in the first place. Fair. But the ratchet mechanism isn't about the person -- it's about what happens on the next turn. When a new contributor opens a PR, or when a model version changes, or when you're coding at 2am and your judgment is impaired, the tests catch regressions regardless of who wrote them. The ratchet works even when the human isn't at their best. That's the point.
Without tests, improvement is a noisy process -- agents try to make things better, but without regression signals, good changes and bad changes are equally invisible. With a dense test suite, you get a ratchet on the tested surface: quality can only go up for the behaviors you've encoded. That's most of the system, not all of it. But it's enough to sustain forward motion at speed.
## Tests as institutional memory
In traditional software companies, institutional memory lives in humans. The senior engineer who knows why that caching layer exists. The architect who remembers the migration that almost destroyed the database. The tech lead who can explain the weird edge case in the billing system.
Humans leave. They retire, they get poached, they burn out. When they leave, the knowledge goes with them. Every software company has had the experience of opening a critical file and finding a comment that says // DO NOT CHANGE THIS -- ask Dave and Dave left three years ago.
The agent's context window doesn't quit. It doesn't get poached. It doesn't forget. When the test suite encodes "weight rounding must use 0.05 increments" and the documentation explains "because cross-modal eval showed false precision degrades trust in the confidence scores," that knowledge is durable. Any agent, any model, any time can load that context and understand the constraint.
Tests are institutional memory that survives employee turnover. For a one-person project, they're even more critical -- they're the only institutional memory you have.
## Everything harnessable is testable
The ratchet doesn't just work for traditional code. It works for anything a computer can observe.
Think about the layers of a modern system. The OS gives you process trees, file system state, network sockets, cron schedules. The terminal gives you every keystroke, every line of output, every interactive prompt. The browser gives you rendered pages, button states, navigation events. APIs give you structured responses you can parse and validate. And AI agents give you observable behavior -- what they say, what tools they call, what order they do things in, whether they ask before acting.
All of these are harnessable. And if you can harness it, you can observe it. If you can observe it, you can assert on it. If you can assert on it, you can ratchet it.
This is a much bigger surface area than traditional unit tests. Let me show you.
GStack is my open-source coding agent framework -- 93,000 GitHub stars, 701,000 lines of code, 46 skills. One of its core features is interactive plan review: you ask it to review your architecture, and it walks through the plan section by section, asking questions, probing edge cases, challenging your assumptions. Like having an engineering manager who actually reads the code.
The problem: Claude Code would sometimes skip the whole interactive part. It would read the plan file, dump every finding in one shot, and exit -- without asking the user a single question. The entire point of the review is the back-and-forth dialogue. Skipping it defeats the purpose.
How do you even test that? You can't unit test "did the AI have a conversation." No traditional testing framework covers this.
So I used Bun's TTY functionality to build a test harness (PR #1354) that literally spawns Claude Code inside a pseudo-terminal, feeds it a specific repo scenario, triggers the review skill, and watches the terminal output in real time. The test observes whether the agent fires an interactive question before finishing. If it dumps findings and exits without asking anything, the test fails.
That's not testing code. That's testing whether an AI agent follows a behavioral contract. At the TTY level. By literally watching it work.
The ratchet response was three layers:
1. STOP gates in the skill instructions -- explicit rules that say "you MUST ask the user before proceeding to the next section," with anti-rationalization clauses that name the specific failure mode so the model can't talk itself into skipping
2. Anti-shortcut clause -- "the plan file is the OUTPUT of the interactive review, not a substitute for it." One sentence that closes the exact loophole the model kept exploiting.
3. Gate-tier floor tests -- the TTY harness tests that spawn Claude Code in controlled scenarios and fail if the agent doesn't ask at least one interactive question
Now when Anthropic ships a new model version, or when I change a skill prompt, the test suite catches any regression in the interactive contract. The agent can't silently stop asking questions. The test watches the terminal and checks.
Or take PR #880, which shipped a new OpenClaw plugin. The test doesn't just check that the code compiles. It builds the plugin from source, spawns a real OpenClaw instance in an isolated profile, installs the plugin via the CLI, runs plugins inspect to verify the runtime loaded it, sets the config slot, validates the config, and runs plugins doctor to confirm zero diagnostics. A full end-to-end round trip across two separate programs. 359 lines of test code. The kind of test a human would almost never write by hand because the setup is too tedious. Claude wrote it in about five minutes. That's the effort wall disappearing in real time.
The principle generalizes. You can test at the OS level: did the migration create the right tables, did the cron job fire, is the process still alive? At the browser level: did the page render, did the agent fill in the form correctly. At the API level: did the model return valid JSON with the right schema. At the behavioral level: did the agent follow the protocol, did it ask before deleting, did it stop when told to stop.
The whole stack is testable. The ratchet applies to all of it. Most people haven't realized this yet because they're still thinking about test coverage as "did my function return the right number." The real test surface is everything the computer can see.
## The 90% number
So what does 90% test coverage actually buy you?
Capers Jones studied over 10,000 software projects and measured defect removal efficiency (DRE) -- the percentage of bugs caught before they reach users. His data from Applied Software Measurement shows a nonlinear curve: below 70% coverage, DRE sits around 65-75%. At 85-95% coverage, DRE jumps to 92-97%. The relationship isn't linear. There's a knee in the curve around 85% where defect escapes drop sharply.
The avionics industry figured this out decades ago. DO-178C, the FAA standard for flight-critical software, requires modified condition/decision coverage (MC/DC) for Level A systems -- the ones where a bug means a plane crash. Branch coverage alone misses 10-20% of faults. MC/DC, which is stricter than line coverage, achieves >99% DRE. They don't mandate this because bureaucrats like paperwork. They mandate it because the data showed that below certain coverage thresholds, critical defects escape at rates incompatible with not killing people.
The reliability engineering parallel is clean. Factories use a system called Six Sigma to measure quality. The idea: count how many defects you get per million units produced, then express that as a "sigma level" -- higher sigma means fewer defects. A 3-sigma process produces about 67,000 defects per million (pretty bad). A 4-sigma process produces about 6,200 (ten times better). A 5-sigma process produces 233 (another 27x better). The jump from 4 to 5 sigma is not incremental improvement. It's a phase change.
Test coverage follows the same curve. Going from 70% to 90% coverage isn't 30% better. It's an order of magnitude fewer escapes. The defects that slip through at 70% are hiding in the 30% of untested code. At 90%, the hiding places shrink to 10% and most of the dangerous paths are locked down.
Now, I should be honest about what the research also shows. Mockus, Nagappan, and Dinh-Trong studied Windows Vista and found that while coverage correlates with fewer post-release defects, the effort to reach 90%+ rises sharply. The last 20% of coverage takes disproportionately more work than the first 70%. This has been true for decades. It's why most teams stop at 70-80% and call it good enough.
But something changed: AI coding agents don't experience effort.
They don't get bored writing the fourteenth edge-case test. They don't cut corners at 5pm on a Friday. They don't look at a gnarly integration test and think "I'll come back to this later." The effort curve that stopped human teams at 70% doesn't apply to agents. You can ask Claude to write tests for every edge case in a module and it will do it cheerfully, thoroughly, at 2am, without complaining. The brutal last 20% that made 90% coverage impractical for human teams is exactly the kind of work AI agents are best at.
This is the real unlock. It's not that AI lets you write code faster. Plenty of people have noticed that. It's that AI lets you verify at a level that was previously too expensive to sustain. The 90% threshold that the data says is magical? It used to cost too much human willpower to reach. Now it's free.
That's the key distinction. The ratchet isn't about line coverage as a vanity metric. It's about tests that encode behavioral contracts -- the holder confusion test, the weight rounding test, the interactive review gate. Each test locks in a specific lesson learned. Coverage is the proxy that tells you how much of the system's behavior is under contract. At 90%, nearly every behavior change triggers a test signal. The agent either passes (safe to ship) or breaks a test (caught immediately).
The remaining 10% is integration points, infrastructure plumbing, and edge cases that are genuinely hard to test. That's fine. The 90% is what turns chaos into a ratchet.
Getting to 90% used to be a heroic effort. Now it's a Tuesday. That's the game change.
## Proof of concept
I started both projects alone. They're not solo anymore.
GStack now has 37 contributors. v1.30 incorporated 21 community PRs in a single release. GBrain has 25 contributors. v0.31.1.1 landed 22 community fixes in one PR -- auth flow, schema bootstrapping, sync, privacy.
The ratchet is what makes this safe. Every external PR has to pass the existing test suite. A new contributor doesn't need to understand the whole system. They need to make the tests pass.
Last week's GBrain releases tell the story:
- v0.31.0: a new facts table for real-time memory, plus a dream consolidation phase that promotes short-term memories into long-term knowledge
- v0.31.1: fixed 25 CLI commands that were silently routing to an empty local database instead of the user's actual brain
- v0.31.1.1: twenty-two community-reported fixes in one PR
- v0.31.2: fixed a code sync that hung forever on large repos with symlinks by adding a 30-second timeout
Each release shipped with more tests than the last. The agent writes the tests alongside the code. The coverage doesn't slip because the effort to maintain it is no longer a human burden.
## The new complexity ceiling
The complexity ceiling for software just got a lot higher.
It used to be bounded by one team's ability to hold the system in their heads. Now it's bounded by one person plus agents who can load the full codebase, schema history, test suite, and documentation into context.
That's a much bigger number. And it keeps growing as context windows get larger and models get better at reasoning about code.
Every software company that doesn't adopt this model -- agents plus taste plus a test suite that only goes up -- is already shipping slower and with less quality than one person who has.
The tools are here. The code is open. Tests are the ratchet. 90% coverage, every PR, no exceptions.
For fifty years, 90% coverage was a luxury reserved for avionics and medical devices -- teams with the budget to throw human hours at the effort wall. AI agents demolished that wall. The coverage threshold that makes software reliable is no longer expensive. It's just a setting. The question isn't whether you can afford 90%. It's whether you can afford not to.
The ratchet, the skills, and the whole knowledge system are open source and free on GitHub. Go build.
My MIT-licensed open source projects:
- GStack -- makes Claude Code dramatically better. 93K stars. Free.
- GBrain -- your second brain for AI agents. 14K stars. Free.
The AI Explainer series:
1. Fat Skills, Fat Code, Thin Harness -- the architecture
2. Resolvers -- the routing table for intelligence
3. The LOC Controversy -- what 600K lines actually produced
4. Naked Models Are Stupider -- the model is the engine, not the car
5. The Skillify Manifesto -- every workflow becomes a testable skill
6. Meta-Meta-Prompting -- compounding skills produce emergent capabilities
7. The Agent Complexity Ratchet -- you are here
## 相关链接
- [Garry Tan](https://x.com/garrytan)
- [@garrytan](https://x.com/garrytan)
- [72K](https://x.com/garrytan/status/2054064931515855118/analytics)
- [GStack](https://github.com/garrytan/gstack)
- [GBrain](https://github.com/garrytan/gbrain)
- [merged fourteen pull requests in 72 hours](https://x.com/garrytan/status/2054055071017538028)
- [GBrain](https://github.com/garrytan/gbrain)
- [Vibecoding](https://x.com/karpathy/status/1886192184808149383)
- [GStack](https://github.com/garrytan/gstack)
- [build a test harness](https://github.com/garrytan/gstack/pull/1354)
- [PR #880](https://github.com/garrytan/gbrain/pull/880)
- [Applied Software Measurement](https://www.amazon.com/Applied-Software-Measurement-Analysis-Productivity/dp/0071502440)
- [DO-178C](https://en.wikipedia.org/wiki/DO-178C)
- [Mockus, Nagappan, and Dinh-Trong](https://ieeexplore.ieee.org/document/4343756)
- [GStack](https://github.com/garrytan/gstack)
- [GBrain](https://github.com/garrytan/gbrain)
- [GStack](https://github.com/garrytan/gstack)
- [GBrain](https://github.com/garrytan/gbrain)
- [Fat Skills, Fat Code, Thin Harness](https://x.com/garrytan/status/2042925773300908103)
- [Resolvers](https://x.com/garrytan/status/2044479509874020852)
- [The LOC Controversy](https://x.com/garrytan/status/2045404377226285538)
- [Naked Models Are Stupider](https://x.com/garrytan/status/2045798603059548364)
- [The Skillify Manifesto](https://x.com/garrytan/status/2046876981711769720)
- [Meta-Meta-Prompting](https://x.com/garrytan/status/2053127519872614419)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:03 PM · May 12, 2026](https://x.com/garrytan/status/2054064931515855118)
- [72.9K Views](https://x.com/garrytan/status/2054064931515855118/analytics)
- [View quotes](https://x.com/garrytan/status/2054064931515855118/quotes)
---
*导出时间: 2026/5/12 21:13:50*
---
## 中文翻译
# AI Agent 的复杂性棘轮：为什么需要 90% 的测试覆盖率
**作者**：Garry Tan
**日期**：2026-05-12T05:03:37.000Z
**来源**：[https://x.com/garrytan/status/2054064931515855118](https://x.com/garrytan/status/2054064931515855118)
---
过去一年我一直在用 AI 编程。不只是写提示词——而是构建真正的软件。两个开源项目：GStack，用于让 AI 编程 agent 变得更好；GBrain，将你读过的和写过的所有内容转化为 AI 可以使用的可搜索知识库。两者加起来大约有 97 万行代码和 665 个测试文件。几乎全部由 Claude Code 和 Codex 在我的指导下编写（大多数时候同时开着 15 个 Conductor 会话）。
上周，我在 72 小时内合并了 14 个拉取请求。近 2.9 万行新代码。每次发布版本的测试质量都比上一次更好。
这本应该是不可能的。速度和质量本应该互相取舍。快速发布，就会出问题；慢慢推进，才能发布正确的产品。二选一。
现在你不必再二选一了。突破口在于 90% 的测试覆盖率——而 AI agent 让达到这个目标变得零成本。五十年来，这种级别的验证成本消耗了太多人类意志力，难以持续。现在，agent 会在编写代码的同时编写测试。结果就是我所说的复杂性棘轮：一个只会变得更好、绝不会变差的系统。
（这是关于用 AI 构建软件系列文章的第七篇：1 2 3 4 5 6）
## 软件曾经是脆弱的
五十年来，整个软件工程学科都围绕着一个理念来组织：预防错误，因为错误是灾难性的。
你必须第一次就把代码写对。漏掉一个边界情况，就会在生产环境崩溃。发布一个有问题的数据库迁移，就会丢失客户数据。写一个行为隐晦的函数，等唯一理解它的人离职后，没人知道它为什么能正常工作。整个系统依赖于人类的谨慎，而人类并不谨慎。所以我们建立了复杂的流程——代码审查、预发布环境、QA 团队、发布节奏——全都是为了在错误到达用户之前将其捕获。
这多少有点用。但它很慢。而且它意味着任何软件系统的复杂性都有一个硬性上限：一个团队的大脑能同时容纳的事物数量。
## 现在软件是柔韧的
我不是说草率。我的意思是它拥有了一种以前不可能具备的弹性。
当我说“模型已经到来”时，我的意思是 AI 编程 agent——Claude、GPT、Codex 以及围绕它们生长的生态系统——现在能够阅读代码、理解上下文、诊断错误并编写修复。并不完美。但已经足够好，以至于软件的错误模型已经改变了。
迁移出问题了？agent 读取错误信息，理解跨越 45 个版本的数据库模式历史，编写修复方案，编写测试。文件同步在百万个符号链接上卡住了？agent 诊断出解析器超时，将其限制在 30 秒内，带着测试发布修复方案。提取管道出现了归因错误？跨模型评估发现了它，提示词得到迭代，数据库层添加了强制约束。
对于大多数代码层面的错误——逻辑 bug、解析失败、损坏的边界情况——agent 现在可以在下一轮对话中诊断并修复它们。这是真正的新事物。仍然灾难性的错误是那些破坏状态的错误：对生产数据的错误迁移、在被检测到之前就被利用的安全漏洞、无法撤销的隐私泄露。棘轮在这方面也有帮助（好的测试会在生产之前捕获大多数这类错误），但真正的转变在于：代码库中绝大多数错误都属于可修复的类型。
这是软件开发方式的相变。但只有拥有棘轮，它才能奏效。
## Agent 复杂性棘轮
棘轮是一种只允许单方向运动的机制。套筒扳手把螺栓向前拧，并防止它往回转。这就是那个比喻。
在由 agent 编写的软件中，与 AI agent 的每一次编码会话都会向代码库添加三样东西：
1. 编码了“正确”含义的测试——每次有人修改代码时都会运行的自动化检查，如果更改破坏了什么就会大声报错
2. 记录了决策原因的文档——不仅是代码做什么，还有背后的推理和权衡
3. 确立质量阈值的评估结果——带分数的结构化输出质量评估，让你知道下一个版本是变好还是变差
下一次 agent 在这个代码库上工作时，它会把这三样东西全部加载到它的上下文窗口（AI 能看到并能推理的文本）中。它不能低于测试套件的水平——测试会失败。它不能忽视文档——它就在上下文里。它不能发布低于评估基线的质量——分数都被记录着。
质量下限随着每一轮不断提升。只进不退。这就是棘轮。
## 实践中是什么样子
我来把这个说得更具体些。GBrain 是我正在构建的一个知识系统——它通过存储、索引和搜索一个人的笔记、会议、对话和研究资料，为 AI agent 提供长期记忆。可以把它想象成一个 AI 助手真正能够阅读的第二大脑。
它的一个功能是认识论提取：它阅读数千页内容，提取出谁相信什么、置信度如何、随时间如何变化。“Garry 认为比特币会涨到 30 万美元（置信度：0.45）。”“Jared 认为这家初创公司的用户留存很强（置信度：0.80）。”类似这样，但涵盖 2.8 万页内容。
第一次提取运行提取了 100,720 条论断。我用跨模型评估来给质量打分——我让 GPT-5.5 和 Claude 各自独立对输出打分。总体得分：10 分中得 6.8 分。
最大的问题是什么？是我称之为“持有者混淆”的问题。拿“AI 将在 2027 年取代 80% 的软件工程师”这个说法来说，谁持有这个信念？是写下这句话的人吗？是他们引用的某个人吗？还是系统的分析引擎——它从一段播客文字记录中推断出这个结论？版本 1 在区分这个问题上的错误率是 35%。这很重要——如果你在构建一个追踪人们信念的系统，你必须知道是谁相信它。
于是评估结果被记录了下来。六个具体的失败模式被识别出来。版本 2 的提示词解决了全部六个问题。权重舍入（置信度分数）在数据库层被强制执行——不再出现像 0.74 这样虚假的精确度，诚实的答案应该是 0.75。十七个测试把契约固定了下来。
现在，任何未来的提取版本，如果这 17 个测试没有通过，就不能发布。没有人需要记住为什么权重舍入很重要，或者什么是持有者混淆。测试会记住。
质量下限被永久地提高了。这就是棘轮的一次转动。
## 为什么大多数 vibecoded 项目会死掉
“Vibecoding”（氛围编程）是 Andrej Karpathy 提出的术语，指通过用自然语言描述你想要的东西、让模型生成代码的方式来用 AI 编程。它很强大，我也正是这样构建软件的。但根据我在 YC 申请材料和开源仓库中所见，大多数跳过测试的 vibecoded 项目一旦达到中等复杂度——几千行代码、若干相互交互的功能——就开始分崩离析。
它们跳过了棘轮。没有测试，没有文档，没有评估。智能体不断增加复杂度，却没有任何机制防止回归。每个新功能都有可能破坏旧功能，而没有测试，你要等到用户报告才能发现问题。到了 0.5 版本，代码库就变成了一座鬼屋，每一次改动都会破坏一些意料之外的东西。然后这位开发者写了一篇博客，说 AI 编程不管用。
AI 编程其实很好用。只是他们没有建造棘轮。
你可能会说，会写测试的那种人本来也就是会写出好架构的那种人。有道理。但棘轮机制的关键不在于人本身，而在于下一回合会发生什么。当新的贡献者提交 PR，或者模型版本变更，或者你在凌晨两点写代码、判断力下降时，无论代码是谁写的，测试都会捕获回归。棘轮即使在人不处于最佳状态时也能发挥作用。这正是它的意义所在。
没有测试，改进是一个充满噪音的过程——智能体试图把事情变好，但没有回归信号，好的改动和坏的改动同样不可见。有了密集的测试套件，你就得到了一个作用于被测表面的棘轮：对于你已编码的行为，质量只会上升。这涵盖了系统的大部分，虽然不是全部。但这已经足以支撑高速前进。
## 测试作为机构记忆
在传统软件公司里，机构记忆存在于人的大脑中。那位知道为什么存在那个缓存层的高级工程师。那位记得那次差点毁掉数据库的迁移的架构师。那位能解释计费系统中奇怪边界情况的技术主管。
人会离开。他们退休，被挖走，或者燃尽。他们离开时，知识也随之而去。每家软件公司都经历过这样的场景：打开一个关键文件，发现一条注释写着 // 不要修改这里——去问 Dave，而 Dave 三年前就离职了。
智能体的上下文窗口不会辞职，不会被挖走，也不会遗忘。当测试套件编码了“权重舍入必须使用 0.05 的增量”，文档解释了“因为跨模态评估显示虚假精确度会损害置信度分数的可信度”，这些知识就是持久的。任何智能体、任何模型、任何时间都可以加载这些上下文并理解这个约束。
测试是能在人员流动中幸存下来的机构记忆。对于单人项目来说，它们甚至更加关键——那是你拥有的唯一的机构记忆。
## 一切可观测的皆可测试
棘轮不仅适用于传统代码。它适用于计算机能观测到的一切。
想想一个现代系统的各个层次。操作系统给你进程树、文件系统状态、网络套接字、cron 调度。终端给你每一次按键、每一行输出、每一个交互提示。浏览器给你渲染后的页面、按钮状态、导航事件。API 给你可以解析和验证的结构化响应。而 AI 智能体给你可观测的行为——它们说了什么、调用了哪些工具、以什么顺序做事、是否在行动前先询问。
所有这些都是可观测的。如果你能观测它，你就能对它断言。如果你能对它断言，你就能为它装上棘轮。
这比传统的单元测试覆盖面大得多。让我给你展示。
GStack 是我的开源编程智能体框架——93,000 个 GitHub 星标，701,000 行代码，46 个技能。它的一个核心功能是交互式计划评审：你让它评审你的架构，它会逐节走查计划，提出问题，探究边界情况，挑战你的假设。就像有一个真正会读代码的工程经理。
问题是：Claude Code 有时会跳过整个交互环节。它会读取计划文件，一次性倾倒所有发现，然后退出——一个问题都不问用户。评审的全部意义就在于这种来回对话。跳过它就等于扼杀了目的。
这该怎么测试？你没法对“AI 是否进行了对话”做单元测试。没有任何传统测试框架能覆盖这种情况。
因此，我利用 Bun 的 TTY 功能构建了一个测试工具（PR #1354），它会在伪终端中真正启动 Claude Code，向其输入特定的仓库场景，触发 review 技能，并实时监视终端输出。测试观察智能体是否在结束前触发了交互式提问。如果它直接输出发现结果然后退出而不问任何问题，测试就会失败。
这可不是在测试代码。这是在测试 AI 智能体是否遵循行为契约。在 TTY 层面上。通过真正观察它工作来实现。
棘轮机制的响应分为三层：
1. 技能指令中的 STOP 门控——明确规则写着"你必须在进入下一部分之前询问用户"，并带有反合理化条款，点名具体的失败模式，让模型无法说服自己跳过
2. 反捷径条款——"计划文件是交互式评审的产出，而不是它的替代品。"一句话就堵住了模型一直钻的那个漏洞。
3. 门控级底线测试——TTY 测试工具在受控场景中启动 Claude Code，如果智能体没有至少提出一个交互式问题，测试就会失败
现在，当 Anthropic 发布新的模型版本，或者当我修改技能提示词时，测试套件能捕捉到交互契约中的任何回归。智能体无法悄悄停止提问。测试会监视终端并进行检查。
再比如 PR #880，它发布了一个新的 OpenClaw 插件。测试不只是检查代码能否编译。它从源码构建插件，在隔离配置文件中启动一个真实的 OpenClaw 实例，通过 CLI 安装插件，运行 plugins inspect 验证运行时已加载它，设置配置槽位，验证配置，再运行 plugins doctor 确认零诊断问题。跨越两个独立程序的完整端到端往返测试。359 行测试代码。这种测试人类几乎绝不会手工编写，因为搭建过程太繁琐了。Claude 用大约五分钟就写好了。这就是努力之墙实时消失的样子。
这个原则可以推广。你可以在操作系统层面测试：迁移是否创建了正确的表，cron 任务是否触发了，进程是否还活着？在浏览器层面：页面是否渲染了，智能体是否正确填写了表单。在 API 层面：模型是否返回了符合正确 schema 的有效 JSON。在行为层面：智能体是否遵循了协议，删除前是否先询问了，被告知停止时是否停止了。
整个技术栈都是可测试的。棘轮机制适用于所有这些层面。大多数人还没意识到这一点，因为他们仍然把测试覆盖率理解为"我的函数是否返回了正确的数字"。真正的测试面是计算机能看到的一切。
## 90% 这个数字
那么，90% 的测试覆盖率究竟能给你带来什么？
Capers Jones 研究了一万多个软件项目，并测量了缺陷移除效率（DRE）——即缺陷在到达用户之前被捕获的百分比。他发表在《Applied Software Measurement》中的数据显示出一条非线性曲线：覆盖率低于 70% 时，DRE 约在 65-75% 之间。覆盖率在 85-95% 时，DRE 跃升至 92-97%。这种关系不是线性的。曲线在 85% 附近有一个拐点，缺陷逃逸率急剧下降。
航空电子行业几十年前就明白了这一点。DO-178C 是 FAA 针对飞行关键软件的标准，要求 A 级系统——即那些出 bug 就意味着坠机的系统——达到修正条件/判定覆盖（MC/DC）。仅靠分支覆盖会遗漏 10-20% 的故障。而比行覆盖更严格的 MC/DC 能达到 >99% 的 DRE。他们强制要求这个标准不是因为官僚喜欢文书工作，而是因为数据显示，低于特定覆盖率阈值时，严重缺陷的逃逸率会达到与"不害死人"这一目标不相容的水平。
可靠性工程的类比也很清晰。工厂使用一套名为"六西格玛"的系统来衡量质量。其理念是：统计每百万件产品中的缺陷数量，然后将其表示为一个"西格玛等级"——西格玛等级越高，缺陷越少。3 西格玛流程每百万件产生约 67,000 个缺陷（相当糟糕）。4 西格玛流程产生约 6,200 个（好十倍）。5 西格玛流程产生 233 个（又好 27 倍）。从 4 西格玛到 5 西格玛的跨越不是渐进式改进，而是一次相变。
测试覆盖率遵循同样的曲线。从 70% 提升到 90% 的覆盖率不是好了 30%，而是缺陷逃逸少了一个数量级。在 70% 时溜走的缺陷，正藏在那 30% 未测试的代码里。到了 90%，藏身之处缩小到 10%，大部分危险路径都被锁死了。
不过，我也应该诚实地谈谈研究的另一面。Mockus、Nagappan 和 Dinh-Trong 研究了 Windows Vista，发现虽然覆盖率与发布后缺陷减少相关，但达到 90% 以上的成本会急剧上升。最后 20% 的覆盖率所需的工作量与最初的 70% 完全不成比例。这种情况已经持续了几十年。这就是为什么大多数团队停在 70-80%，然后宣称"够好了"。
但有什么东西变了：AI 编程智能体不会感受到努力。
它们不会在编写第十四个边界用例测试时感到厌烦。它们不会在周五下午五点偷工减料。它们不会看着一个复杂的集成测试想着"我回头再处理这个"。那种让人类团队止步于 70% 的努力曲线对智能体不适用。你可以让 Claude 为一个模块中的每个边界用例编写测试，它会欣然、彻底地完成，哪怕是在凌晨两点，毫无怨言。正是那些让人类团队难以实现 90% 覆盖率的残酷的最后 20%，恰恰是 AI 智能体最擅长的工作。
这才是真正的解锁点。不是 AI 让你写代码更快——很多人已经注意到了这一点。而是 AI 让你能够以前贵得无法承受的成本进行验证。数据显示具有魔力的那个 90% 阈值？过去它需要太多的人类意志力才能达到。现在它是免费的。
这就是关键区别。棘轮机制不是把行覆盖率当作虚荣指标，而是关于那些将行为契约编码下来的测试——持有人混淆测试、重量舍入测试、交互式审查关卡。每个测试都锁定了一条具体的经验教训。覆盖率是一个代理指标，告诉你系统的行为有多少处于契约保护之下。达到 90% 时，几乎每一次行为变更都会触发测试信号：要么通过（可以安全发布），要么破坏了某个测试（立即被捕获）。
剩下的 10% 是集成点、基础设施管道，以及确实难以测试的边缘情况。这没关系。正是这 90% 将混乱转变成了棘轮。
过去，达到 90% 需要英雄般的努力。现在，这只是一个普通的星期二。这就是游戏规则的改变。
## 概念验证
两个项目都是我一个人启动的。现在它们不再是单人项目了。
GStack 现在拥有 37 位贡献者。v1.30 在一次发布中合并了 21 个社区 PR。GBrain 有 25 位贡献者。v0.31.1.1 在一个 PR 中落地了 22 个社区修复——认证流程、模式引导、同步、隐私。
棘轮机制正是让这一切变得安全的原因。每个外部 PR 都必须通过现有的测试套件。新贡献者不需要理解整个系统，只需要让测试通过即可。
上周 GBrain 的发布说明了这一点：
- v0.31.0：为实时记忆新增事实表，外加一个梦境巩固阶段，将短期记忆提升为长期知识
- v0.31.1：修复了 25 个静默路由到空本地数据库而非用户实际大脑的 CLI 命令
- v0.31.1.1：一个 PR 中包含二十二个社区报告的修复
- v0.31.2：通过添加 30 秒超时，修复了在带符号链接的大型仓库上代码同步永久挂起的问题
每次发布附带的测试都比上一次更多。智能体在编写代码的同时编写测试。覆盖率不会下滑，因为维护它的工作不再是一种人类负担。
## 新的复杂度天花板
软件的复杂度天花板刚刚被大幅抬高。
过去，它受限于一个团队在头脑中掌握整个系统的能力。现在，它受限于一个人加上能够将完整代码库、模式历史、测试套件和文档加载到上下文中的智能体。
这是一个大得多的数字。而且随着上下文窗口变大、模型对代码的推理能力增强，这个数字还在持续增长。
每一家不采用这种模式——智能体加上品味加上只升不降的测试套件——的软件公司，已经在发布速度和质量上落后于一个采用了这种模式的人。
工具已经就绪。代码已经开源。测试就是棘轮。90% 覆盖率，每个 PR，无一例外。
五十年来，90% 的覆盖率一直是航空电子和医疗设备领域才享有的奢侈品——那些有预算往努力之墙上堆砌人力工时的团队。AI 智能体推倒了那堵墙。让软件变得可靠的覆盖率门槛不再昂贵。它只是一个设置而已。问题不在于你是否负担得起 90%，而在于你是否负担得起不用 90%。
棘轮、技能和整个知识系统都在 GitHub 上开源且免费。去构建吧。
我的 MIT 许可开源项目：
- GStack——让 Claude Code 变得极其更好。93K 星。免费。
- GBrain——你的 AI 智能体第二大脑。14K 星。免费。
《AI 解读》系列：
1. Fat Skills, Fat Code, Thin Harness——架构
2. Resolvers——智能的路由表
3. The LOC Controversy——60 万行代码实际产出了什么
4. Naked Models Are Stupider——模型是引擎，不是汽车
5. The Skillify Manifesto——每个工作流都变成可测试的技能
6. Meta-Meta-Prompting——技能的复利产生涌现能力
7. The Agent Complexity Ratchet——你在这里
## 相关链接
- [Garry Tan](https://x.com/garrytan)
- [@garrytan](https://x.com/garrytan)
- [72K](https://x.com/garrytan/status/2054064931515855118/analytics)
- [GStack](https://github.com/garrytan/gstack)
- [GBrain](https://github.com/garrytan/gbrain)
- [72 小时内合并了 14 个拉取请求](https://x.com/garrytan/status/2054055071017538028)
- [GBrain](https://github.com/garrytan/gbrain)
- [Vibecoding](https://x.com/karpathy/status/1886192184808149383)
- [GStack](https://github.com/garrytan/gstack)
- [构建测试工具](https://github.com/garrytan/gstack/pull/1354)
- [PR #880](https://github.com/garrytan/gbrain/pull/880)
- [Applied Software Measurement](https://www.amazon.com/Applied-Software-Measurement-Analysis-Productivity/dp/0071502440)
- [DO-178C](https://en.wikipedia.org/wiki/DO-178C)
- [Mockus、Nagappan 和 Dinh-Trong](https://ieeexplore.ieee.org/document/4343756)
- [GStack](https://github.com/garrytan/gstack)
- [GBrain](https://github.com/garrytan/gbrain)
- [GStack](https://github.com/garrytan/gstack)
- [GBrain](https://github.com/garrytan/gbrain)
- [Fat Skills, Fat Code, Thin Harness](https://x.com/garrytan/status/2042925773300908103)
- [Resolvers](https://x.com/garrytan/status/2044479509874020852)
- [The LOC Controversy](https://x.com/garrytan/status/2045404377226285538)
- [Naked Models Are Stupider](https://x.com/garrytan/status/2045798603059548364)
- [The Skillify Manifesto](https://x.com/garrytan/status/2046876981711769720)
- [Meta-Meta-Prompting](https://x.com/garrytan/status/2053127519872614419)
- [升级至 Premium](https://x.com/i/premium_sign_up)
- [2026年5月12日 下午1:03](https://x.com/garrytan/status/2054064931515855118)
- [72.9K 次浏览](https://x.com/garrytan/status/2054064931515855118/analytics)
- [查看引用](https://x.com/garrytan/status/2054064931515855118/quotes)
---
*导出时间: 2026/5/12 21:13:50*
