Why Software Factories Fail ✍ dex🕐 2026-07-25📦 26.7 KB 🟢 已读 𝕏 文章列表 文章探讨了AI驱动的软件工厂模式面临的挑战。作者指出，尽管AI编码工具看似能提高效率，但实际应用中往往导致代码质量下降、Bug增多和事故频发。问题不在于技术技能不足，而是模型训练的根本性限制。 AI软件工厂代码质量模型训练技术挑战自动化工程实践 # Why Software Factories Fail **作者**: dex **日期**: 2026-07-24T16:51:27.000Z **来源**: [https://x.com/dexhorthy/status/2080697380379427275](https://x.com/dexhorthy/status/2080697380379427275) ---  or: the harness is not enough Update - the talk version of this post is live on youtube: https://www.youtube.com/watch?v=Ib5GBkD555M ## i guess we doin loops now We're all racing to put AI coding into production. A lot has been said about loop engineering, and the prevailing wisdom is that we should probably write more loops.  StrongDM wrote about their lights-off software factory where no human reads code and no human writes code. The narrative goes something like this: 1. You are the bottleneck. 2. The models are good enough. 3. Code is free. 4. Just ship more stuff. Ryan Lopopolo of OpenAI wrote about this in February and gave a talk in April about OpenAI's software factory, Symphony. These people are all really dang smart and I have a ton of respect for them. But the most cynical take here would be to call this yet another excuse to pump more VC money into the slop cannon. ## it's uh...it's going Our friend Mario got up at AI Engineer Europe and begged us to slow down -- because companies that have no business having outages due to coding-agent mishaps, are, well... having outages due to coding-agent mishaps. As Matt Pocock put it, codebases are falling apart faster than they ever have before. I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. edit - there is some conversation with the team on hacker news on July 23 - sounds like we might get a more formal update soon! The folks at Faros AI put out a report: since we2 all picked up these AI coding tools back in January and February, pull-request review quality is way down. - More comments, longer comments, and tons of PRs getting merged with no review at all. - Incidents are way up. - Bugs per developer are way up.  This report is more of a correlation signal than a verifiable smoking gun (yes i chose that word on purpose, don't get me started on claude prose), and the whole point of this post is to be wary of slop data, but it feels directionally valid based on what I've seen. ## "You're holding it wrong" (you're not) A lot of people will tell you that this is a skill issue -- that if you're not getting good results, that's your fault. But however you're choosing to...erhm...hold it, I guarantee you're being told that if token-maxxing isn't working for you, it's a skill issue. You just need to spend more tokens. Let go of reading the code. And if you're just getting there, I promise it's part of the progression. I thought this way last summer too. Unfortunately for my ego, some dumb stuff I decided to say about "how to hold it better" got recorded and now has about a million cumulative views on YouTube. I am not trying to brag here, I share this only to establish that I've been going deep on the best ways to use coding agents for a long time now, and have discovered some things that many others have found genuinely useful. - Advanced Context Engineering for Coding Agents - No Vibes Allowed -- Solving Hard Problems in Complex Codebases - Everything We Got Wrong About RPI Anyhow, The promise of all this online "just token harder" yapping we've been forced to endure is, succinctly: with enough harness engineering, we can get the best of both worlds: - 10 to 100x faster, - high quality, and - nobody ever has to do that thing we all hate called code review All we have to do is configure more linters and sprinkle some magic words like "adversarial review" onto enough PR review bots, and our software will happily build itself without incident. ## This is not a skill issue What I'm gonna try to convince you is that no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue. To grapple with this, I had to dig into how coding models are actually trained and evaluated - with respect to both the RLVR and the benchmark side of things. In this post I'm gonna run through: 1. Software factories date back to 1968, how have they evolved, and how has AI changed them 2. Why models can generate mountains of slop despite ace-ing benchmarks (even the brand new "frontier" benchmarks) 3. In spite of this, you can move pretty fast without setting your codebase on fire I'm gonna try to cut through the hype of every daily-emerging skills plugin and the ai-psychosis-tokenmaxxing advice pandemic, and talk in general terms about the types of things that work without referencing any particular skill or framework. Video Version: this post is based on (and expands upon) my keynote at AI Engineer World's Fair 2026. Thanks to @addyosmani, @CyrusNewDay, @HamelHusain, @zeeg, @dillon_mulroy, @nayshins, and @jeffreyhuber for feedback on this post. ## An aside: this has nothing to do with vibe coding Addy Osmani detangled this thing that is worth highlighting: > A developer vibe-coding a side project a dozen people will ever run, and a team keeping a ten-year-old enterprise system alive for another quarter, share almost no constraints worth naming, and most of the advice in circulation is really one of those two people telling the other how to live. If you love vibe coding, please, go on vibing. I still vibe code lots of things, I just also maintain lots of production software (and through HumanLayer, help 1000s of other engineers do the same), so the rest of this is aimed at folks solving hard problems in complex codebases. I hear the word brownfield a lot to talk about this split. Historically that meant some ten-year-old Java thing, but at the pace we can ship now, it feels like an agent-built codebase starts to struggle after maybe three to six months -- you start to slow down, and the way you approach adding new things has to change. ## A brief history of the software factory I've been building and studying software factories my whole career, but I only learned this recently: the term traces all the way back to a NATO conference in 1968 -- the same one that gave us "software engineering." The only other bit I find super interesting since then is that the US Department of Defense wrote a 31-page pdf about how the DoD needs to start using jenkins better or something. ## The 2022 software factory Let's ground our "software factory" definition around 2022, right before AI. In a typical software factory: - People decide what to build -- engineers, PMs, leadership driving the vision - It goes in a tracker -- Linear, Jira, whatever: a state machine of what needs to happen - Someone grabs a ticket and builds it -- probably does some manual/automated testing while they're at it - Pull request -- automated checks, a human reviews the code, maybe someone pulls it down to test - Anything wrong? Loop back to "someone builds the thing" - Ship to prod -- and it makes contact with users - Add monitoring -- there's an entire industry built around paging an engineer at 3am when something breaks - Users complain -- ask for things, find bugs, file feature requests → back to the team to add to the tracker  And on and on. We haven't even hit AI yet, and there are already several loops in this picture. ## front-loading alignment There's a thing teams figured out decades ago: building takes hours or days, and so does review.  So we front-load the work -- planning, architecture proposals, sprint planning -- together, as a team. That means: - less rework, because we aligned before anyone wrote code - less time reviewing every line, if you've ever read a long-but-well-done PR, you know how fast the review goes when it's close-to-perfect  We'll come back to this later - let's look at what happens when you bring agentic coding into the picture. ## The agentic software factory Now every company and their mother -- - Ramp - Stripe - WorkOS - Brex has spent the better part of this year explaining how they built an agent factory that ships on the order of 75% of their code. The agentic factory looks mostly like swapping "someone builds the thing" → "an agent builds the thing" -- there's some stuff here like orchestration, a harness, a sandbox, a model, computer use, etc. I won't go in depth on those details because quite frankly I'm sick of reading about it and I'm sure you are too.  When the agent builds the thing: - Building drops from hours or days to minutes or hours. - Review still takes hours or days. A human still has to read the code and test the change. So review is now the bottleneck.  So you speed review up too: - Agentic code review, to catch style, bugs, security. - Agentic regression testing, to poke it from the outside with browsers and computer use and maybe send you a cute little video when it's done  Review is faster now, but it's also probably still the bottleneck. But we can do more loops. Next you might route incidents into the factory. Instead of paging someone at 3am, they wake up to a PR that maybe already fixes it.  We can also route user feedback into the factory. People ask for stuff, it gets built.  At which point the job is two questions: how much can you stuff into the queue, and how fast can you review and test what comes out?  Which brings us to the lights-off software factory. ## The lights-off software factory Dan Shapiro coined this term and Simon Willison wrote about StrongDM's implementation of it -- where we no longer read the code. You look at your beautiful software factory. It's ruined by that annoying little code review step and you say: you know what, that thing where a human reads every change? No thanks.  So you drop it, and you put the effort somewhere else: - Invest in testing and letting the agent test its own work - Invest in sandboxes and orchestration - Invest in automated review - Invest in monitoring - Invest in rollout - Invest in collecting feedback signals from users  And now the job really is just one question: how much stuff can we ask the agent to build? How much of the ocean do we want to boil? ## This is going to go great (its not)  I'm going to posit something potentially controversial: the lights off factory does not work. Let's get into why software factories fail. ## We tried this In July 2025 we went full lights-off. Just read the specs and the tickets, background agents for all the small/medium stuff, the whole thing. If you've tried this seriously for a few months, you already know how it ends. You find at least one issue gnarly enough that the agent can't solve it -- even with your most advanced prompting and workflows. - You do deep context-aware research, collating all the right parts into the smart zone for the model to analyze - You have the agent try to reproduce in 10 different ways Eventually you have to suck it up and go dig into the codebase you stopped reading three months ago, trying to figure out what's broken. And in the meantime: - Your site was down. - Your users were pissed. - And you, if you're anything like me, were miserable -- reading all the slop code you let slip into your system. The first time this happened to us, I shook it off. Even though I'd just spent the better part of two weeks digging through claude spaghetti, "the downside risk was worth the velocity". By the ~third time in november, we decided it would be easier to rewrite from scratch, and my cofounder spent two whole weeks in VS Code (not even cursor) plumbing out all the patterns by hand. ## models degrade codebase quality over time What I want to get to is this: models have a shortcoming. They can't maintain and improve codebase quality over time -- not without a decent amount of human steering.4 When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part. This is Martin Fowler's shotgun surgery. I'm not going to say much more about maintainability. There are a bunch of books you can go read about it - John Ousterhout's A Philosophy of Software Design - Robert C. Martin's Clean Code - Martin Fowler's Refactoring So, why can't models do software maintainability? ## "But surely the models have gotten better since then" At this point you might be dying to say: but Dex, surely the models have gotten much better since July They have -- in some ways. In others they're about the same. - Solving one-off problems, or vibe-coding a new marketing site? Yes. Way better. - Improving codebase quality over time? Not much better, as far as I can tell.  I can't prove this. You can't prove it either. There are no good benchmarks for a model's ability to maintain codebase quality. (More on where that's going later.) > THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality But if you've worked with coding agents for a while -- and a lot of people are posting about exactly this -- you probably have the vibe already: they tend to make things worse over time, and make the codebase harder to work in. So to figure out why this happens, I want to zoom out to the first great coding agent. ## Claude Code won because of Reinforcement Learning inside the harness Claude Code went from nothing to ~$4B -- now something like ~$9B -- in revenue in under a year.  Which is a little wild, because there were already great CLI agents. aider, cline, codebuff -- all predated Claude Code, all with genuinely great context engineering built in, all with the same tool set you might attribute to claude code: read, write, edit, grep, bash. I used them. They were good. But also, tool use would just... fail sometimes -- you'd watch it flail at the same edit three times and open your editor back up to do it yourself. The SWE-Agent paper from 2024 outlines how small changes in tool shape make noticeable differences, e.g. including line numbers in ReadFile results, or changing an Edit tool from find/replace to line-range edits.  Then Claude Code launched and went vertical pretty quickly. You can hand-wave this as distribution, but the canonically-accepted explanation is that claude code won because it was better, and that it was better because Anthropic RL'd the model inside the harness -- the first time a lab trained a model against the exact tools they were going to ship it with. And it got really, really good at calling those tools in an agentic loop. It's one thing to fiddle with tool definitions and evals until you find the shape the model likes best -- I've burned weeks doing this for various use cases. It's a different game when you own the weights and can modify the model itself to be better at a particular set of tools. The OpenAI team gave a talk in November that put this pretty well: if you build a harness but you don't own the weights and can't RL the model inside it, you'll always be at a disadvantage to a team that owns both. ## Coding Agent RL in 60 seconds I did a bunch of research on this topic and cooked up a bunch of visualizations to try to explain the parts that matter, but I found that Calvin French-Owen (MTS on the codex team, founder of Segment) did a talk at AI Council that did a much better and cleaner job, so I'm just gonna drop this animation here inspired by his slides:  To make a model better at coding, you're gonna: 1. generate some coding agent traces to solve a problem (e.g. fix my tests) 2. score the traces based on some criteria (verifier) 3. update the model weights to make the good traces more likely, and the bad traces less likely And then you do this millions of times over the course of weeks or months. The "scoring" part of these things can tend to be whimsically one-dimensional though. ## There's no penalty for bad design Take SWE-bench Multilingual. The tasks are small -- about fifteen minutes of work apiece -- scraped out of open-source repos like Redis, jq, and Django. The reward is one or zero based on: - FAIL_TO_PASS - did you fix the thing you were asked to fix? - PASS_TO_PASS - did you do it without breaking anything else? Here's a real one, fastlane__fastlane-19304, from fastlane -- a Ruby project. Its zip action grabs two optional params and calls .empty? on them straight away, so the moment you leave include and exclude off, it falls over:  The human fix that closed this particular issue is two lines (default nils to empty arrays):  During the evaluation, the model 1. starts from a base commit -- the repo checked out to the moment right before that fix landed 2. the bug report - in this case 'zip_command': undefined method 'empty?' for nil:NilClass The agent goes off and writes some code based on the issue. It doesn't see the golden patch or the test patch that serves as the grader:  Then: 1. We keep whatever patch it produced, then 2. Throw away any edits it made to the test files (we've caught a model quietly commenting out the failing test or splicing in a mock that makes the test useless) 3. Apply the benchmark's test patch on top, and 4. Run the whole suite: the existing zip tests (PASS_TO_PASS) plus the new one (FAIL_TO_PASS) to see if they both pass  Aside - Benchmarks are not verifiers - in fact they have to be held out from each other (don't train on test, yada yada) - I primarily mean this to convey the shape of "judging the quality of a coding agent trace" and its limitations. How the model got to a correct answer doesn't matter. If the tests pass, we win, but there is no penalty for eroding codebase maintainability. there is no penalty for eroding codebase maintainability That's how you get try catches around everything:  ## Verifying quality is orders of magnitude harder than "did the tests pass" Running the tests gets you a clean pass or fail in ~seconds. That's why RL can run millions of loops to optimize each model generation. But the cost function of bad architecture is measured in weeks, months, maybe even years. It happens the first time someone opens that file for a one-line change and realizes they can't make it in one line -- that someone vibed this a little too hard, and now we have to make the same edit in eleven places and hope nothing quietly breaks three files over.  Tests give you feedback in seconds, but the cost function of bad architecture is measured in weeks, months, maybe even years Bad design is the one thing today's benchmarks can't evaluate. And I know, I know, RL != Benchmarks, but if this was solved in RL, I'm pretty sure it would start to show up in how our benchmarks are designed too. In any case, I personally don't trust any improvements on today's benchmarks as an indicator that the models are suddenly good at not slopping up your codebase. ## The frontier is getting better, slowly Of course lots of smart folks are working on this. My point is not that it can't be done, it's that the hype is outrunning the discipline. A few efforts I think are pointed the right way: - SWE-Marathon (Abundant AI): ~400-hour tasks like "clone all of Excel, every feature" -- with a compound reward channel instead of a single pass/fail bit - DeepSWE (Datacurve): big tasks on OSS repos that were never actually built in the real world, so by construction they can't already be sitting in the training set (solves contamination, but not quality) - Frontier Code (Cognition): multi-PR tasks, and a clever move that evaluates quality deterministically -- it penalizes the model for writing tests that don't fail on the pre-patch code (if you've never heard about mutation testing you are in for a fun ride5). It also runs a judge model over the diff checking code-quality rules.  But a model judging quality can only go so far. In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with. RL needs a fast+reliable oracle, and we don't yet have one for maintainability if a model could reliably tell good code from bad, it might have written the good version to begin with, but maintainability has no fast oracle, so we can't reward for it during RL Of course, more review agents and more tokens do help -- they raise the floor, catching the dumb stuff. But they don't move the ceiling, because the ceiling is whatever we managed to teach the model in RL, and good design is the thing we still don't know how to teach it. So I still wouldn't bet my codebase on any of these. But they're the first evals I've seen even trying to score maintainability instead of stopping at pass/fail. Aside Maybe a future model just gets this and we can stop. If you want to yolo prompts until GPT-7 ships and find out, be my guest -- but bitter lesson be damned, we've got problems to solve now, and I'm gonna walk through how we do that. ## Turning the lights back on Today I learned that Twitter Articles have a "media limit" which means the rest of this is going into a part II post - stay tuned ## 相关链接 - [dex](https://x.com/dexhorthy) - [@dexhorthy](https://x.com/dexhorthy) - [43K](https://x.com/dexhorthy/status/2080697380379427275/analytics) - [https://www.youtube.com/watch?v=Ib5GBkD555M](https://www.youtube.com/watch?v=Ib5GBkD555M) - [StrongDM wrote about their lights-off software factory](https://factory.strongdm.ai/) - [Ryan Lopopolo](https://x.com/_lopopolo) - [wrote about this in February](https://openai.com/index/harness-engineering/) - [gave a talk in April](https://www.youtube.com/watch?v=am_oeAoUhew) - [Mario](https://x.com/badlogicgames) - [begged us to slow down](https://www.youtube.com/watch?v=RjfbvDXpFls) - [Matt Pocock](https://x.com/mattpocockuk) - [codebases are falling apart faster than they ever have before](https://www.youtube.com/watch?v=3MP8D-mdheA) - [weather-report](https://factory.strongdm.ai/weather-report) - [there is some conversation with the team on hacker news on July 23](https://news.ycombinator.com/item?id=49026625) - [The folks at Faros AI](https://www.faros.ai/research/ai-acceleration-whiplash) - [I thought this way last summer too](https://hlyr.dev/ace) - [Advanced Context Engineering for Coding Agents](https://hlyr.dev/ace) - [No Vibes Allowed -- Solving Hard Problems in Complex Codebases](https://hlyr.dev/nva) - [Everything We Got Wrong About RPI](https://hlyr.dev/qrspi-mlops) - [RLVR](https://github.com/opendilab/awesome-RLVR) - [my keynote at AI Engineer World's Fair 2026](https://www.youtube.com/watch?v=Ib5GBkD555M) - [@addyosmani](https://x.com/addyosmani) - [@CyrusNewDay](https://x.com/CyrusNewDay) - [@HamelHusain](https://x.com/HamelHusain) - [@zeeg](https://x.com/zeeg) - [@dillon_mulroy](https://x.com/dillon_mulroy) - [@nayshins](https://x.com/nayshins) - [@jeffreyhuber](https://x.com/jeffreyhuber) - [Addy Osmani](https://x.com/addyosmani/status/2066595308629594363) - [NATO conference in 1968](http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1968.PDF) - [US Department of Defense wrote a 31-page pdf about how the DoD needs to start using jenkins better or something](https://dodcio.defense.gov/Portals/0/Documents/Library/DevSecOpsReferenceDesign.pdf) - [Ramp](https://infoq.com/news/2026/01/ramp-coding-agent-platform/) - [Stripe](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents) - [WorkOS](https://workos.com/blog/project-horizon) - [Brex](https://www.latent.space/p/brex) - [Dan Shapiro coined this term](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/) - [Simon Willison wrote about StrongDM's implementation of it](https://simonwillison.net/2026/Feb/7/software-factory/) - [ocean do we want to boil](https://garryslist.org/posts/boil-the-ocean) - [Martin Fowler's shotgun surgery](https://refactoring.guru/smells/shotgun-surgery) - [John Ousterhout's A Philosophy of Software Design](https://web.stanford.edu/~ouster/cgi-bin/aposd.php) - [Robert C. Martin's Clean Code](https://www.oreilly.com/library/view/clean-code-a/9780136083238/) - [Martin Fowler's Refactoring](https://martinfowler.com/books/refactoring.html) - [aider](https://aider.chat/) - [cline](https://cline.bot/) - [codebuff](https://codebuff.com/) - [The SWE-Agent paper from 2024](https://arxiv.org/abs/2405.15793) - [gave a talk in November](https://www.youtube.com/watch?v=wVl6ZjELpBk) - [Calvin French-Owen](https://x.com/calvinfo) - [AI Council](https://www.youtube.com/watch?v=q-ntX4DLW_c) - [SWE-bench Multilingual](https://huggingface.co/datasets/SWE-bench/SWE-bench_Multilingual) - [fastlane](https://github.com/fastlane/fastlane) - [the hype is outrunning the discipline](https://www.youtube.com/watch?v=c35YoMdnI78) - [SWE-Marathon](https://www.swe-marathon.org/) - [DeepSWE](https://deepswe.datacurve.ai/blog/deepswe) - [Frontier Code](https://cognition.com/blog/frontier-code) - [mutation testing](https://en.wikipedia.org/wiki/Mutation_testing) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [12:51 AM · Jul 25, 2026](https://x.com/dexhorthy/status/2080697380379427275) - [43K Views](https://x.com/dexhorthy/status/2080697380379427275/analytics) - [View quotes](https://x.com/dexhorthy/status/2080697380379427275/quotes) --- *导出时间: 2026/7/25 11:35:00* --- ## 中文翻译 # 为什么软件工厂会失败 **作者**: dex **日期**: 2026-07-24T16:51:27.000Z **来源**: [https://x.com/dexhorthy/status/2080697380379427275](https://x.com/dexhorthy/status/2080697380379427275) ---  或者：光有 harness 是不够的 更新 - 这篇文章的演讲版本已在 YouTube 上发布：https://www.youtube.com/watch?v=Ib5GBkD555M ## 看来我们现在都在做循环（loops）了 我们都在竞相将 AI 编码投入生产。关于循环工程已经有很多讨论，主流观点是我们可能应该编写更多的循环。  StrongDM 写到了他们的“熄灯”软件工厂，那里没有人阅读代码，也没有人编写代码。 这种叙事大致如下： 1. 你就是瓶颈。 2. 模型已经足够好了。 3. 代码是免费的。 4. 只管交付更多东西。 OpenAI 的 Ryan Lopopolo 在二月份写到了这一点，并在四月份做了一个关于 OpenAI 软件工厂 Symphony 的演讲。 这些人真的非常聪明，我对他们充满敬意。但对此最愤世嫉俗的看法是，这只是又一个借口，将更多 VC 资金注入到“垃圾加农炮”中。 ## 这……进展吧 我们的朋友 Mario 在 AI Engineer Europe 大会上站起来请求我们慢下来——因为那些根本不该因为编码代理事故而导致中断的公司，确实正在因为编码代理事故而发生中断。 正如 Matt Pocock 所说，代码库崩溃的速度比以往任何时候都要快。 我还没能找到 StrongDM 关于那个全暗工厂是如何运行的任何确凿数据/发现。Weather-report 在今年二月到六月之间有一些零星的更新。编辑 - 7 月 23 日在 Hacker News 上有一些与团队的对话——听起来我们可能很快会得到一个更正式的更新！ Faros AI 的人发布了一份报告：自从我们都在今年一月和二月开始使用这些 AI 编码工具以来，拉取请求（PR）的审查质量大幅下降。 - 评论更多了，评论更长了，而且大量的 PR 在完全没有审查的情况下被合并。 - 事故大幅增加。 - 每个开发者的 Bug 数量大幅增加。  这份报告更多的是一个相关性信号，而不是可验证的确凿证据（是的，我是故意选这个词的，别让我开始吐槽 Claude 的文风），这篇文章的重点也是要对垃圾数据保持警惕，但根据我的所见所闻，它在方向上是合理的。 ## “是你拿的方式不对”（不，不是你） 很多人会告诉你，这是技能问题——如果你没有得到好的结果，那是你的错。 但是，无论你选择如何……额……拿住它，我保证你会被告知，如果 Token 最大化对你不起作用，那就是技能问题。你只需要消耗更多的 Token。放弃阅读代码。如果你才刚刚开始，我保证这是进步过程的一部分。去年夏天我也是这么想的。 不幸的是，为了我的自尊，我决定说的一些关于“如何更好地拿住它”的蠢话被记录下来，现在在 YouTube 上累计获得了大约一百万次观看。我不是在炫耀，我分享这些只是为了确立我已经在深入研究使用编码代理的最佳方式很长时间了，并且发现了一些许多其他人觉得真正有用的东西。 - 面向编码代理的高级上下文工程 - 不许来虚的 —— 在复杂的代码库中解决难题 - 我们在 RPI 上犯的所有错误 总之，我们要被迫忍受的所有这些在线上“只要更努力地用 Token”的废话，其简短的承诺是：通过足够的 harness 工程，我们可以两全其美： - 速度快 10 到 100 倍， - 高质量，并且 - 没有人必须做我们都讨厌的那件事，叫做代码审查 我们要做的就是配置更多的 linter，并在足够的 PR 审查机器人上撒一些像“对抗性审查”这样的神奇词汇，我们的软件就会开心地自我构建而不会发生事故。 ## 这不是技能问题 我要试图说服你的是，没有任何数量的 harness 工程或循环最大化可以解决从根本上说是模型训练的问题。 为了解决这个问题，我必须深入研究编码模型实际上是如何训练和评估的——既涉及 RLVR，也涉及基准测试方面。 在这篇文章中，我将通过以下几个方面： 1. 软件工厂可以追溯到 1968 年，它们是如何演变的，以及 AI 如何改变了它们 2. 为什么模型尽管能在基准测试中拿高分（即使是全新的“前沿”基准测试），却仍然能产生海量的垃圾代码 3. 尽管如此，你可以在不烧毁代码库的情况下保持相当快的速度 我将试图透过每天涌现的技能插件的热潮，以及 AI 精神病-Token 最大化建议的流行病，从一般角度谈论那些有效的事情，而不涉及任何特定的技能或框架。 视频版本：这篇文章基于（并扩展了）我在 2026 年 AI Engineer World's Fair 上的主题演讲。 感谢 @addyosmani, @CyrusNewDay, @HamelHusain, @zeeg, @dillon_mulroy, @nayshins, 和 @jeffreyhuber 对这篇文章的反馈。 ## 题外话：这与 Vibe Coding（氛围编程）无关 Addy Osmani 梳理了一件值得注意的事情： > 一个开发者在为最多十几个人会运行的副项目进行氛围编程，和一个团队为了让一个十年的企业系统再存活一个季度而维护它，这两者几乎没有值得一提的共同约束，而且目前流传的大部分建议实际上都是这两个人中的一个在告诉另一个该如何生活。 如果你喜欢氛围编程，请继续。我仍然会对很多东西进行氛围编程，只是我也维护大量的生产软件（并且通过 HumanLayer，帮助 1000 多名其他工程师做同样的事情），所以接下来的内容是针对那些在复杂的代码库中解决难题的人。 我经常听到“棕地”这个词来谈论这种分裂。从历史上看，这意味着某种十年的 Java 东西，但以我们现在的交付速度，感觉代理构建的代码库在大概三到六个月后就开始挣扎——你开始慢下来，你添加新东西的方式不得不改变。 ## 软件工厂简史 我在整个职业生涯中一直在构建和研究软件工厂，但我最近才知道：这个词可以追溯到 1968 年的一次北约会议——正是那次会议给了我们“软件工程”这个词。 从那以后，我觉得唯一另一个超级有趣的一点是，美国国防部写了一份 31 页的 PDF，关于国防部需要如何开始更好地使用 Jenkins 之类的东西。 ## 2022 年的软件工厂 让我们在 2022 年左右，就在 AI 出现之前，确定我们的“软件工厂”定义。在一个典型的软件工厂中： - 人们决定构建什么——工程师、产品经理（PM）、领导层推动愿景 - 它进入一个追踪器——Linear, Jira, 任何东西：一个关于需要发生什么的状态机 - 某人拿一个工单并构建它——可能顺便做一些手动/自动测试 - 拉取请求——自动检查，一个人审查代码，也许某人把它拉下来测试 - 有什么问题吗？循环回到“某人构建这个东西” - 交付到生产环境——并接触到用户 - 添加监控——围绕着在凌晨 3 点东西坏掉时给工程师发寻呼，这已经形成了一个完整的行业 - 用户抱怨——要求东西，发现 bug，提交功能请求 → 回到团队添加到追踪器  以此类推。我们甚至还没有触及 AI，这张图中已经出现了几个循环。 ## 前置对齐 这是几十年前团队就发现的一件事：构建需要几个小时或几天，审查也是如此。  所以我们前置工作——计划、架构提案、冲刺计划——一起，作为一个团队。这意味着： - 更少的返工，因为我们在任何人写代码之前就已经对齐了 - 减少审查每一行的时间，如果你曾经读过一篇长但写得很好的 PR，你知道当它接近完美时审查会有多快  我们稍后会回到这一点——让我们看看当你把代理编码引入画面时会发生什么。 ## 代理软件工厂 现在每家公司及其母公司—— - Ramp - Stripe - WorkOS - Brex 都花了今年大部分的时间来解释他们如何构建了一个代理工厂，可以交付他们大约 75% 的代码。 代理工厂看起来大多像是把“某人构建这个东西”换成“一个代理构建这个东西”——这里有一些东西，比如编排、harness、沙箱、模型、计算机使用等。我不会深入讨论这些细节，因为坦率说我厌倦了阅读它，我相信你也是。  当代理构建这个东西时： - 构建从几小时或几天缩短到几分钟或几小时。 - 审查仍然需要几小时或几天。人类仍然必须阅读代码并测试更改。所以审查现在是瓶颈。  所以你也加快了审查速度： - 代理代码审查，以捕获风格、bug、安全问题。 - 代理回归测试，从外部用浏览器和计算机使用来戳它，也许在完成后给你发送一个可爱的小视频  审查现在更快了，但它可能仍然是瓶颈。但我们可以做更多的循环。 接下来，你可能会将事故引入工厂。与其在凌晨 3 点给某人打电话，他们醒来时可能会看到一个已经修复了它的 PR。  我们也可以将用户反馈引入工厂。人们要求东西，它就被构建出来。  此时，工作变成了两个问题：你可以在队列中塞多少东西，以及你可以多快审查和测试产出的东西？  这把我们带到了熄灯软件工厂。 ## 熄灯软件工厂 Dan Shapiro 创造了这个术语，Simon Willison 写了关于 StrongDM 对它的实现——在那里我们不再阅读代码。 你看着你漂亮的软件工厂。它被那个恼人的小代码审查步骤毁了，你说：你知道吗，那个人类阅读每一个更改的事情？不用了，谢谢。  所以你把它去掉，并把精力放在别处： - 投资于测试并让代理测试它自己的工作 - 投资于沙箱和编排 - 投资于自动审查 - 投资于监控 - 投资于发布 - 投资于收集来自用户的反馈信号  现在工作真的只是一个问题：我们可以要求代理构建多少东西？我们要煮沸多少海洋？ ## 这会很顺利的（其实不会）  我要提出一个可能有争议的观点：熄灯工厂行不通。 让我们进入为什么软件工厂会失败的原因。 ## 我们试过这个 2025 年 7 月，我们完全熄灯了。只阅读规格和工单，所有小/中型的东西都由后台代理处理，整个都是。 如果你认真地尝试了几个月，你已经知道结局了。你至少会发现一个足够棘手的问题，代理无法解决——即使你最先进的提示和工作流程。 - 你进行深度的上下文感知研究，将所有正确的部分整理到智能区域供模型分析 - 你让代理尝试以 10 种不同的方式重现 最终你不得不硬着头皮去挖掘你三个月前停止阅读的代码库，试图弄清楚是什么坏了。 与此同时： - 你的网站宕机了。 - 你的用户很生气。 - 而你，如果你像我一样，也很痛苦——阅读所有你让它溜进系统的垃圾代码。 这第一次发生在我们身上时，我把它甩在一边。虽然我刚刚花了整整两周的时间挖掘 Claude 的意大利面条代码，“下行风险是值得速度的”。到了十一月的第三次左右，我们决定从头重写会更容易，我的联合创始人在 VS Code（甚至不是 Cursor）里花了整整两周手动理清所有模式。 ## 模型会随着时间的推移降低代码库质量 我想说的是：模型有一个缺陷。它们无法随着时间的推移维持和提高代码库质量——如果没有相当数量的人类引导的话。 当我谈到可维护性时，我指的是那种特定的东西，即如果不破坏另一部分，就真的很难更改代码库的一部分。这就是 Martin Fowler 的“霰弹式手术”。 我不会过多谈论可维护性。你可以去读一堆关于它的书 - John Ousterhout 的《软件设计哲学》 - Robert C. Martin 的《代码整洁之道》 - Martin Fowler 的《重构》 那么，为什么模型不能做到软件可维护性？ ## “但模型从那时起肯定变好了” 在这一点上，你可能迫不及待地说：但是 Dex，模型从七月开始肯定变得好多了 它们在某些方面确实变好了。在其他方面，它们差不多一样。 - 解决一次性问题，或者氛围编程一个新的营销网站？是的。好多了。 - 随着时间的推移提高代码库质量？据我所知，并没有好多少。  我无法证明这一点。你也无法证明。没有好的基准来测试模型维持代码库质量的能力。（稍后会详细说明这方面的进展。） > 没有好的基准来测试模型维持代码库质量的能力 但是如果你与编码代理一起工作了一段时间——而且很多人正在发布关于这方面的帖子——你可能已经有这种感觉了：它们倾向于让事情随着时间的推移变得更糟，并使代码库更难工作。 所以为了弄清楚为什么会发生这种情况，我想放大到第一个伟大的编码代理。 ## Claude Code 之所以获胜，是因为在 harness 内部进行了强化学习 Claude Code 在不到一年的时间里从零增长到约 40 亿美元——现在大约 90 亿美元——的收入。  这有点疯狂，因为已经有一些很棒的 CLI 代理了。aider, cline, codebuff——都比 Claude Code 早，所有这些都有真正伟大的上下文工程，都有你可能归因于 Claude Code 的相同工具集：read, write, edit, grep, bash。我用了它们。它们很好。但是，工具使用有时就是会失败——你会看着它在同一个编辑上挣扎三次，然后打开你的编辑器自己动手。 2024 年的 SWE-Agent 论文概述了工具形状的微小变化如何产生明显的差异，例如在 ReadFile 结果中包含行号，或者将 Edit 工具从查找/替换更改为行范围编辑。  然后 Claude Code 发布并很快垂直上升。你可以把这归结为分发，但公认的解释是 Claude Code 之所以获胜是因为它更好，而它之所以更好是因为 Anthropic 在 harness 内部对模型进行了 RL——这是第一次实验室针对他们将要发布的 exact tools 训练模型。它在代理循环中调用这些工具变得真的非常好。 摆弄工具定义和评估直到找到模型最喜欢的形状是一回事——我为了各种用例花了好几周做这个。当你拥有权重并可以修改模型本身使其对一组特定工具更擅长时，那是另一回事。 OpenAI 团队在十一月做了一个演讲，很好地表达了这一点：如果你构建了一个 harness 但你不拥有权重，不能在它内部对模型进行 RL，你将永远处于比拥有两者的团队不利的地位。 ## 60 秒看懂编码代理 RL 我对这个主题做了很多研究，并制作了一堆可视化来试图解释重要的部分，但我发现 Calvin French-Owen（codex 团队的 MTS，Segment 的创始人）在 AI Council 做的演讲更好、更简洁，所以我只是在这里受到他的幻灯片启发放这个动画：  为了让模型更擅长编码，你要： 1. 生成一些编码代理轨迹来解决问题（例如，修复我的测试） 2. 根据某些标准（验证器）对轨迹进行评分 3. 更新模型权重，使好的轨迹更有可能，坏的轨迹更不可能 然后在几周或几个月的时间里做这件事数百万次。 这些东西的“评分”部分往往倾向于 whimsically 一维。 ## 对糟糕的设计没有惩罚 以 SWE-bench Multilingual 为例。任务很小——每个大约十五分钟的工作——从 Redis, jq 和 Django 等开源代码库中抓取。奖励是基于以下的一项或零： - FAIL_TO_PASS - 你修复了你被要求修复的东西吗？ - PASS_TO_PASS - 你在不破坏任何其他东西的情况下做到了吗？ 这里有一个真实的，来自 Ruby 项目 fastlane 的 fastlane__fastlane-19304。它的 zip 操作获取两个可选参数并立即对它们调用 .empty?，所以一旦你把 include 和 exclude 留空，它就会崩溃：  关闭这个特定问题的人工修复是两行（将 nil 默认为空数组）：  在评估过程中，模型 1. 从一个基本提交开始——代码库签出到该修复落地之前的时刻 2. bug 报告——在这个情况下 'zip_command': undefined method 'empty?' for nil:NilClass 代理根据问题去写一些代码。它看不到 golden patch 或作为评分者的测试补丁：  然后： 1. 我们保留它产生的任何补丁，然后 2. 丢弃它对测试文件所做的任何编辑（我们抓住了模型悄悄注释掉失败的测试或拼接一个使测试无用的 mock） 3. 在上面应用基准的测试补丁，以及 4. 运行整个套件：现有的 zip 测试（PASS_TO_PASS）加上新的（FAIL_TO_PASS）来看看它们是否都通过  题外话 - 基准不是验证器——事实上它们必须彼此保持独立（不要在测试上训练，等等）——我主要是想传达“判断编码代理轨迹质量”的形状及其局限性。 模型如何得到正确答案并不重要。如果测试通过，我们就赢了，但对于侵蚀代码库可维护性没有惩罚。 对于侵蚀代码库可维护性没有惩罚 这就是你会到处都是 try catch 的原因：  ## 验证质量比“测试通过”要难几个数量级 运行测试可以在 ~秒内给你一个干净的通过或失败。这就是 RL 可以运行数百万次循环来优化每个模型生成的原因。 但是糟糕架构的成本函数是以周、月甚至年为单位来衡量的。它发生在第一次有人打开那个文件进行一行更改并意识到他们无法在一行内完成它——某人过度“氛围编程”了，现在我们必须在十一个地方进行相同的编辑，并希望没有什么东西悄悄地破坏三个文件之外。  测试在几秒钟内给你反馈，但糟糕架构的成本函数是以周、月甚至年为单位来衡量的 糟糕的设计是今天的基准无法评估的一件事。我知道，我知道，RL != 基准，但如果这在 RL 中解决了，我很确定它也会开始反映在我们的基准是如何设计的。 无论如何，我个人不相信今天基准上的任何改进作为模型突然不把你的代码库搞砸的指标。 ## 前沿正在慢慢变好 当然，很多聪明人在为此努力。我的观点不是说这做不到，而是炒作跑跑在纪律前面。 我认为指向正确方向的一些努力： - SWE-Marathon (Abundant AI)：约 400 小时的任务，如“克隆所有 Excel，每个功能”——具有复合奖励通道而不是单个通过/失败位 - DeepSWE (Datacurve)：OSS 代码库上的大任务，这些任务从未在现实世界中真正构建过，所以从构造上讲它们不可能已经坐在训练集中（解决了污染，但没解决质量） - Frontier Code (Cognition)：多 PR 任务，以及一个确定性评估质量的巧妙举措——它惩罚模型编写在补丁前代码上不失败的测试（如果你从未听说过变异测试，你会很开心）。它还在 diff 上运行一个 judge 模型来检查代码质量规则。  但是一个判断质量的模型只能走这么远。 事实上，不难想象，如果一个模型能可靠地区分好代码和坏代码，它可能一开始就写出了好版本。RL 需要一个快速+可靠的预言机，而对于可维护性我们还没有 如果一个模型能可靠地区分好代码和坏代码，它可能一开始就写出了好版本，但可维护性没有快速预言机，所以我们无法在 RL 期间对此进行奖励 当然，更多的审查代理和更多的 Token 确实有帮助——它们提高了底线，捕获愚蠢的东西。 但它们没有提高上限，因为上限是我们在 RL 中设法教给模型的任何东西，而好的设计是我们仍然不知道如何教给它的东西。 所以我仍然不会把我的代码库押注在其中任何一个上。但它们是我看到的第一批甚至尝试给可维护性打分而不是停在通过/失败的评估。 题外话 也许未来的模型只是懂了，我们可以停下来。如果你想乱试提示直到 GPT-7 出来并发现，请便——但苦涩的教训见鬼去吧，我们现在要解决问题，我将通过我们如何做到这一点。 ## 重新打开灯 今天我了解到 Twitter 文章有一个“媒体限制”，这意味着剩下的内容将进入第二部分——敬请期待
T The guide to software factories 本文介绍了软件开发从交互式编码代理向云端软件工厂的转变。软件工厂通过自动化SDLC（软件开发生命周期）的各个环节，减少人为差异，最大化软件输出，并提升安全性与合规性。文章阐述了软件工厂的概念、工作流程及其核心组件，包括云运行时、沙箱等，旨在帮助工程领导理解如何部署这一系统以解决当前交互式代理的成本和治理问题。 技术 › DevOps ✍ Zach Lloyd🕐 2026-07-15 软件工厂SDLCAI自动化云开发DevOpsClaudeCodex工程效能
装 装完 Codex 不知道干什么？这 6 个 GitHub Skills 让你做视频搞钱 文章介绍了 6 个适用于 Codex 的 GitHub Skills，涵盖动效生成、视频剪辑、批量制作、AI 生成及中文剪辑等工具，帮助用户构建自动化视频工作流以提升效率。 技术 › Codex ✍ Kay🕐 2026-07-30 Codex视频制作AgentSkill自动化HyperFramesRemotionAI剪辑工作流
P Pragmatic Leverage in the Software Factory 文章探讨了在软件开发中如何通过AI实现杠杆效应。仅用AI写代码只能加速部分流程，而结合规划与对齐才能获得更大效率提升。作者通过“预期痛苦”公式分析前期投入与返工成本的关系，强调在多层次规划中保持实用主义，以最大化减少返工概率。 技术 › 工具与效率 ✍ dex🕐 2026-07-30 AI软件工厂杠杆效应规划效率DevOps方法论
如 如何构建你的第一个智能体工厂 文章探讨了如何从构建单个智能体转向构建智能体工厂，以解决人工审核瓶颈问题。介绍了Sage决策模型在自动化质量控制和输出门控中的应用，提供了具体的实现思路和GitHub资源。 技术 › Agent ✍ Avid🕐 2026-07-30 智能体工厂AI自动化质量控制Sage模型LLM
如 如何构建能在你睡觉时运行的公司大脑 本文介绍如何超越个人知识库，构建一个“公司大脑”。通过将文件系统与 AI 智能体结合，设定文件夹作用域、定时任务和自动汇报，让智能体自动处理营销、文件归档等工作，实现无需人工干预的全自动企业级知识管理。 技术 › Agent ✍ Hila Shmuel🕐 2026-07-29 AI知识库自动化工作流LLMCabinet
我 我是如何用 Orca 做 Graph Engineering 作者分享了使用 Orca 进行 Graph Engineering 的实践经验，通过拆分任务给 Agent 执行，结合编排调度和真机验收，解决了长任务拆解、动态调整依赖和防止假成功的问题，强调了 Graph、Eval 和权限控制三者结合的重要性。 技术 › Agent ✍ Aiden🕐 2026-07-28 Graph EngineeringOrcaAgent编排调度真机验收动态依赖假成功自动化移动端开发工程实践
C Codex + Hyperframes + HeyGen +声音克隆：零基础自媒体变现全开源 作者分享了利用 Codex、Hyperframes、HeyGen 和 IndexTTS2 等工具构建的自动化视频工作流，实现了零基础、低成本的短视频制作与变现。文章详细介绍了各工具的选型理由、工作流架构及成本账单，旨在帮助读者通过 AI 技术解决内容生产的效率问题。 技术 › 工具与效率 ✍ 雪踏乌云🕐 2026-07-27 AI自媒体视频制作HeyGen声音克隆CodexHyperFrames自动化变现工作流
如 如何利用AI构建和扩展单人企业 文章介绍了利用AI构建和扩展单人企业的完整蓝图。通过使用AI员工（如Viktor）在内容、项目、拓展、财务和广告五个领域实现自动化，只需保留决策环节。文章探讨了两种构建方式：快速路径和自定义构建，强调业务知识库和操作规则的重要性。 技术 › Agent ✍ Machina🕐 2026-07-26 AI单人企业自动化Agent生产力Viktor商业模式知识库LLM效率
管 管理 AI 员工团队：Ryan Carson 的高效工作系统 Ryan Carson 分享了如何作为唯一员工，通过管理云端 AI Agent 团队日处理 40 个 Pull Request 的经验。文章详细介绍了将工作迁移至云端、建立工作节奏、将重复检查自动化以及控制 Token 成本四个关键步骤，强调在 AI 时代，优秀的工程管理能力比以往任何时候都重要。 技术 › Agent ✍ The Startup Ideas Podcast (SIP)🕐 2026-07-25 AIAgentDevin工程管理自动化云端开发成本控制DevOpsLLM
我 我用 WorkBuddy 跑了 3 个活儿，发现它不是助手，是数字员工 文章通过三个真实任务评测了 WorkBuddy 的实际应用：视频 Skill 管理工具开发、电商数据清洗分析及竞品监控自动化流程。作者指出 WorkBuddy 不仅是聊天工具，更是集专家判断、专业 Skill 和自动化执行于一体的 AI 工作台，能有效提升处理固定流程任务的效率。 技术 › 工具与效率 ✍ 行者AI视频🕐 2026-07-24 WorkBuddyAI自动化数据分析实战评测效率提升Agent专家模式技能管理竞品监控
用 用飞书多维表格打造你的数字分身 文章介绍了如何利用飞书多维表格结合 AI 技术打造知识库“数字分身”。通过三级进化：将知识结构化存库、利用 AI 字段自动化处理知识、接入飞书机器人实现对话交互，解决专业知识沉淀与复用难题，提升业务效率。 技术 › 工具与效率 ✍ 黄小木🕐 2026-07-24 飞书多维表格AI知识库数字分身自动化Agent低代码
如 如何教授 AI 模型 文章通过人类学习的类比，通俗易懂地解释了 AI 模型如何学习新技能和行为。它涵盖了从预训练、监督微调到强化学习的三个阶段，以及通过基准测试评估模型和通过模型规范对齐模型行为的重要性，最后指出了当前模型在持续学习方面的局限性。 技术 › LLM ✍ Lee Robinson🕐 2026-07-24 AI模型训练机器学习RLSFT模型对齐