# Verifying Agentic Development at Scale
**作者**: Ido Pesok
**日期**: 2026-05-29T17:41:24.000Z
**来源**: [https://x.com/ido_pesok/status/2060416230641881336](https://x.com/ido_pesok/status/2060416230641881336)
---
What we’ve learned building end-to-end testing capabilities in Devin’s virtual machine.
3 months ago, I joined Cognition to help build the future of software engineering. Devin has come a long way since launching as the first AI software engineer, and I’ve been blown away watching the team behind it actually use Devin every day.
One thing that stood out was how Devin uses its computer to verify work autonomously in the cloud. From validating our Slack integration to testing complex Windsurf features, the team always has an army of Devins in test mode. In this post, I’ll share why we are so focused on end-to-end cloud agent verification and how we’re approaching building it.
## The shift to async software engineering
At Cognition, we recently reached a new milestone. For the first time, more Devins are being triggered asynchronously, via events, automations, schedules, and other Devins. We expect this to continue to accelerate with our recent launch of Auto-Triage.
As we transition to this async world, it is crucial developers are able to come back to verified results that are ready to be merged. Earlier this year we launched Devin Review, a code review tool that scales human understanding of complex code diffs. It doesn’t just flag bugs - Devin closes the loop by fixing each finding until the diff comes back clean. But a clean review alone often isn’t enough - engineers want to see the change tested end to end, the same way they would test it themselves.
It’s a great feeling when Devin puts up a PR fixing a user complaint before you had the chance to even see the message in the bug channel. What makes it magical is when that PR comes with proof that the fix actually works. And this magic may soon become a necessity - as more PRs come from the rise of proactive agents, unverified changes will quickly become unmanageable.
For the first time, more Devin sessions are triggered asynchronously than interactively.
## From the Beginning
Since Devin launched, it has always been able to demonstrate its work on a cloud virtual machine. Around 6 months ago, we expanded Devin’s computer use capabilities. What this means in practice is that we added tools to Devin’s harness to take screenshots, move the mouse, click, drag, type, press keys, scroll, wait, zoom, and start/stop recording. Computer use has been around for a while now, but we felt the latest cohort of models from the frontier labs has started to really get good at utilizing these tools.
The computer use unlocked some fun new capabilities for Devin, like building and playing a desktop game, or using its browser to order products on Amazon. But the real unlock we noticed was Devin’s ability to test its own work. Devin will spin up the app, click through it, and confirm its changes actually work, the same way an engineer would. Everything runs in the cloud and can scale out in parallel. This really hit me when I saw engineers running 10 to 20 Devins in parallel, each with its own dev server, working through changes – this is something you simply can’t do on a single laptop. Automated cloud testing started saving us an enormous amount of time, since we no longer have to run and verify the code locally.
To be honest, getting here wasn’t smooth. We hit plenty of failure modes along the way that each taught us something about what it takes to make this system more reliable.
## Increasing Reliability
In early versions, it was very common for Devin to go off track during testing. It happened in all sorts of ways: over-testing unrelated parts of the product, getting lost in setup before reaching the feature, or simply missing the core behavior the PR was actually meant to change.
To address this, when Devin enters test mode we have it first write out a test plan detailing a clear target about what to test. This plan must be grounded in source, not assumptions. Without grounding in code, we found the models like to assume they can go down paths in the app that don’t exist. Furthermore, the test plan greatly increases the complexity of changes Devin can successfully test. Some of our most ambitious requests included features that required multiple services running, specific admin settings configured, and the right flags enabled before the behavior was even reachable. When reading the code upfront, Devin is much more likely to set up the environment correctly instead of discovering something missing halfway through the test. The test plan acts as a form of pre-alignment and makes Devin less likely to drift when actively testing.
As Devin works through the plan, it adds its own annotations into the timeline. These include things like setup notes, the start of each named test, and assertions marked as passed, failed, or untested. We found that Devin will lie less about its findings if it annotates its expected behavior right before performing an action - much like test-driven development, if you commit to the expectation upfront it makes it much harder to rationalize an unexpected result as a pass.
Some parts of the testing flow repeat in almost every run. Logging in is the classic example: driving a login form through computer use often means typing an email, completing SSO, clicking through redirects, and waiting on every page load, screenshot by screenshot. This can be costly both in time and tokens. To improve the reliability and cost for these actions, Devin extracted the work to a deterministic script that lives in a testing skill in our repo. This way, Devin can run the script and get an authenticated browser session in seconds and jump into the core part of the test. The deterministic nature of these scripts helped decrease flakiness dramatically. We updated Devin to also close this loop itself. When it figures out a setup step the hard way, Devin can suggest saving that knowledge as a testing skill in the repo and propose the fix back to the user as a one-click PR.
A login skill gets Devin an authenticated session in seconds, so the run spends its time in the click / screenshot / assert loop that proves the change.
We’re also experimenting with routing the testing phase to different models. Since testing leans on different strengths than writing code, like reading screenshots, tracking UI state, and deciding the next browser action, some models are simply better at this than the typical one you’d pick for editing code.
## Using Autonomous Testing in Devin Today
Devin currently enters test mode in two ways: an explicit ask to test a change, or, after Devin creates a PR it will offer to test the change if applicable. From there, it will create the test plan and start getting to work.
Oftentimes when you are first starting to use Devin’s testing capabilities it will need your assistance. A good example is if it requires secrets when running your app locally. To make this process smoother, Devin is able to ask you in the session for any credentials or other information that may be missing. For more difficult cases, you can take over Devin’s computer and enter things like OTP codes. The good news is once Devin is done setting up your repo, it is able to save a declarative configuration in the form of a YAML blueprint that produces a snapshot for every future session to boot from.
## What you get back
When Devin finishes testing, it doesn’t just tell you whether the app worked. A raw screen recording is useful but we felt it wasn’t enough on its own - you need to understand what you’re looking at, why Devin took each action, and which parts of the test passed or failed.
For a quick review, Devin will return a test report with labeled screenshots from key moments in the run so you can quickly see what Devin tested and what the app looked like along the way.
If you want a deeper review, Devin also produces a test video with a rich player UI that has chapters to let you jump between testing sections, scrub through the full run, and inspect the assertions that passed or failed in a chronological list view. In post processing, the dead time between actions is compressed while the moments around actions play back at normal speed. This makes it so a long run condenses into a recording you can actually watch. These artifacts are available in our web interface and also distributed to Slack if Devin was started from there.
Devin runs the app, exercises the flow, and returns a recording with chapters and pass/fail assertions.
## Hard edges
Computer use still has hard edges. One example is timing - if Devin is testing a toast notification, a screenshot taken too early or too late can miss the toast entirely and the models can get confused about whether the expected behavior actually happened.
Another failure mode is cheating. Left to their own devices, the models may sometimes lean too heavily on executing JavaScript in the browser to trigger states programmatically instead of clicking through the UI. This can be helpful to test functionality, but users will often want to see Devin exercise the app the way a real user would.
We’re actively working on these issues through improved evals, tighter guardrails in the harness, and each new generation of models that gets better at computer use.
## The future of async development is verified
In the past couple of months, test runs approved per day on Devin have more than doubled. That growth reflects something simple: async agents are only useful if developers can trust what they come back with. Often that trust can’t come from code alone: for many changes, you want to know that the app was actually run, the important flows were exercised, and the result was captured in a way you can easily inspect.
That’s what autonomous testing in Devin is designed to provide. Devin plans the test, operates the app, records and annotates what happened, and finally returns artifacts that make the result reviewable. There’s still a lot to improve, but we think this is the right shape of the future: agents that don’t just complete work asynchronously, but come back with proof.
We’re constantly surprised by how much time Devin saves us by testing its own work, and feel that many customers still underutilize Devin’s automated testing feature. To support experimentation, we are currently billing at 1/5th the normal usage cost while in test mode.
Try our work at devin.ai or windsurf.com. And if working on problems like these sounds fun, reach out to ido [at] cognition.ai
## 相关链接
- [Ido Pesok](https://x.com/ido_pesok)
- [@ido_pesok](https://x.com/ido_pesok)
- [108K](https://x.com/ido_pesok/status/2060416230641881336/analytics)
- [validating our Slack integration](https://x.com/walden_yan/status/2012351454699532502?s=46)
- [Auto-Triage](https://cognition.ai/blog/auto-triage)
- [we launched Devin Review](https://cognition.ai/blog/devin-review)
- [Devin closes the loop](https://cognition.ai/blog/closing-the-agent-loop-devin-autofixes-review-comments)
- [we expanded Devin’s computer use](https://x.com/cognition/status/1983983151157563762?s=46)
- [ask you in the session for any credentials](https://docs.devin.ai/product-guides/secrets#giving-devin-credentials)
- [declarative configuration](https://docs.devin.ai/onboard-devin/environment/blueprints)
- [devin.ai](http://app.devin.ai/)
- [windsurf.com](http://windsurf.com/)
- [cognition.ai](https://cognition.ai/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:41 AM · May 30, 2026](https://x.com/ido_pesok/status/2060416230641881336)
- [108.9K Views](https://x.com/ido_pesok/status/2060416230641881336/analytics)
- [View quotes](https://x.com/ido_pesok/status/2060416230641881336/quotes)
---
*导出时间: 2026/5/30 11:00:51*
---
## 中文翻译
# 大规模验证代理开发
**作者**: Ido Pesok
**日期**: 2026-05-29T17:41:24.000Z
**来源**: [https://x.com/ido_pesok/status/2060416230641881336](https://x.com/ido_pesok/status/2060416230641881336)
---
我们在 Devin 的虚拟机中构建端到端测试能力时学到的经验。
3 个月前，我加入了 Cognition，致力于构建软件工程的未来。自作为首个 AI 软件工程师发布以来，Devin 已经取得了长足的进步，看着幕后的团队每天都在实际使用 Devin，我深感震撼。
其中最引人注目的一点是 Devin 如何利用其计算机在云端自主验证工作。从验证我们的 Slack 集成到测试复杂的 Windsurf 功能，团队总是拥有一支处于测试模式的 Devin 大军。在这篇文章中，我将分享为什么我们要如此专注于端到端的云代理验证，以及我们是如何构建它的。
## 向异步软件工程转变
在 Cognition，我们最近达到了一个新的里程碑。首次有更多的 Devin 会话是通过事件、自动化、计划以及其他 Devin 异步触发的。随着我们最近推出 Auto-Triage，我们预计这一趋势将继续加速。
当我们过渡到这个异步世界时，开发人员能够回到准备合并的、经过验证的结果这一点至关重要。今年早些时候我们推出了 Devin Review，这是一款扩展人类对复杂代码差异理解的代码审查工具。它不仅会标记 Bug —— Devin 还会通过修复每一个发现点来闭环，直到差异变干净为止。但仅有一个干净的审查往往是不够的 —— 工程师希望看到变更经过端到端的测试，就像他们自己测试那样。
当 Devin 在你甚至还没来得及在 Bug 频道看到消息之前，就提交了一个修复用户投诉的 PR，这种感觉非常棒。而让它真正神奇的是，该 PR 附带了修复确实有效的证明。这种魔力可能很快就会成为必需品 —— 随着 proactive agents（主动代理）的兴起带来更多的 PR，未经验证的变更将很快变得难以管理。
首次，通过异步触发的 Devin 会话多于交互式触发的。
## 从开始做起
自 Devin 发布以来，它一直能够在云端虚拟机上演示其工作。大约 6 个月前，我们扩展了 Devin 的计算机使用能力。这在实践中的意思是，我们在 Devin 的 harness（工具集）中添加了工具来进行截图、移动鼠标、点击、拖动、输入、按键、滚动、等待、缩放以及开始/停止录制。计算机使用已经存在一段时间了，但我们感觉 Frontier Labs 推出的最新一代模型已经开始真正擅长利用这些工具了。
计算机使用为 Devin 解锁了一些有趣的新功能，比如构建并玩一款桌面游戏，或者使用浏览器在亚马逊上订购产品。但我们注意到的真正解锁之处在于 Devin 测试自己工作的能力。Devin 会启动应用程序，点击浏览，并确认其更改确实有效，就像工程师一样。一切都在云端运行，并且可以并行扩展。当我看到工程师并行运行 10 到 20 个 Devin，每个都有自己的开发服务器，正在处理变更时，这真的触动了我 —— 这是你无法在单台笔记本电脑上做到的。自动化云端测试开始为我们节省大量时间，因为我们不再需要在本地运行和验证代码。
老实说，达到这一步并不顺利。一路上我们遇到了许多失败模式，每一个都教会了我们一些关于如何让这个系统更可靠的知识。
## 提高可靠性
在早期版本中，Devin 在测试过程中偏离轨道是非常常见的。它以各种方式发生：过度测试产品中不相关的部分、在到达功能之前迷失在设置中，或者仅仅是错过了 PR 实际要更改的核心行为。
为了解决这个问题，当 Devin 进入测试模式时，我们首先让它写出一个测试计划，详细说明关于测试什么的明确目标。该计划必须基于源代码，而不是假设。如果没有代码作为基础，我们发现模型喜欢假设它们可以沿着应用程序中不存在的路径走。此外，测试计划大大增加了 Devin 可以成功测试的变更的复杂性。我们的一些最具挑战性的请求包括需要运行多个服务、配置特定的管理员设置以及启用正确的标志才能达到行为的功能。当提前阅读代码时，Devin 更有可能正确设置环境，而不是在测试中途才发现缺少某些东西。测试计划充当一种预先对齐的形式，使 Devin 在主动测试时不太可能漂移。
当 Devin 执行计划时，它会在时间线中添加自己的注释。这些包括设置说明、每个命名测试的开始，以及标记为通过、失败或未测试的断言。我们发现，如果 Devin 在执行操作之前立即注释其预期行为，它对其发现的谎报就会减少 —— 就像测试驱动开发一样，如果你提前承诺预期，就很难将意外结果合理化为通过。
测试流程的某些部分几乎在每次运行中都会重复。登录就是一个典型的例子：通过计算机使用驱动登录表单通常意味着输入电子邮件、完成 SSO、点击重定向、等待每个页面加载，以及逐个截图。这在时间和 Token 方本都可能代价高昂。为了提高这些操作的可靠性和成本，Devin 将工作提取为一个确定性脚本，该脚本存在于我们仓库的一个 testing skill 中。这样，Devin 可以在几秒钟内运行该脚本并获得经过身份验证的浏览器会话，并跳转到测试的核心部分。这些脚本的确定性性质帮助显著减少了不稳定性。我们更新了 Devin，使其也能自己闭环这个过程。当 Devin 通过困难的方式弄清楚了设置步骤时，它可以建议将该知识作为 testing skill 保存到仓库中，并将修复作为一键 PR 提议给用户。
登录技能让 Devin 在几秒钟内获得经过身份验证的会话，以便运行将时间花在证明变更的点击/截图/断言循环中。
我们还在尝试将测试阶段路由到不同的模型。由于测试依赖于与编写代码不同的优势，例如阅读截图、跟踪 UI 状态和决定下一个浏览器操作，因此某些模型在这方面比典型的用于编辑代码的模型要好。
## 今天在 Devin 中使用自主测试
Devin 目前通过两种方式进入测试模式：明确要求测试变更，或者，在 Devin 创建 PR 后，如果适用，它会提议测试该变更。从那里，它将创建测试计划并开始工作。
通常，当你第一次开始使用 Devin 的测试功能时，它需要你的帮助。一个很好的例子是，如果在本地运行你的应用程序时需要机密信息。为了使这个过程更顺畅，Devin 能够在会话中向你询问任何可能缺失的凭据或其他信息。对于更困难的情况，你可以接管 Devin 的计算机并输入诸如 OTP 码之类的内容。好消息是，一旦 Devin 完成了你的仓库设置，它就能够以 YAML 蓝图的形式保存声明性配置，从而为每个未来的会话生成启动快照。
## 你会得到什么
当 Devin 完成测试时，它不仅会告诉你应用程序是否有效。原始屏幕录制很有用，但我们觉得仅此还不够 —— 你需要理解你在看什么，为什么 Devin 采取每个行动，以及测试的哪些部分通过或失败了。
为了快速审查，Devin 将返回一个测试报告，其中包含来自运行关键时刻的带标签的截图，以便你可以快速看到 Devin 测试了什么以及应用程序沿途的样子。
如果你想进行更深入的审查，Devin 还会生成一个带有丰富播放器 UI 的测试视频，该 UI 具有章节功能，允许你在测试部分之间跳转，浏览完整的运行，并在按时间顺序排列的列表视图中检查通过或失败的断言。在后处理中，操作之间的死时间会被压缩，而操作周围的时刻则以正常速度回放。这使得长时间运行可以浓缩成你实际上可以观看的录制。这些工件在我们的 Web 界面中可用，如果 Devin 是从那里开始的，也会分发到 Slack。
Devin 运行应用程序，执行流程，并返回一个带有章节和通过/失败断言的录制。
## 棘手之处
计算机使用仍然有棘手之处。一个例子是时机 —— 如果 Devin 正在测试 toast 通知，过早或过晚拍摄的截图可能会完全错过 toast，模型可能会对预期行为是否实际发生感到困惑。
另一种失败模式是作弊。如果任其自生自灭，模型有时可能会过于依赖在浏览器中执行 JavaScript 以编程方式触发状态，而不是通过 UI 点击。这有助于测试功能，但用户通常希望看到 Devin 以真实用户的方式使用应用程序。
我们正在通过改进的评估、更严格的工具围栏以及每一代更擅长计算机使用的新模型来积极解决这些问题。
## 异步开发的未来是经过验证的
在过去几个月里，每天在 Devin 上批准的测试运行增加了一倍多。这种增长反映了一件简单的事情：异步代理只有在开发人员可以信任其返回的结果时才有用。通常，这种信任不能仅来自代码：对于许多变更，你想知道应用程序实际上已运行，重要的流程已执行，结果以一种易于检查的方式捕获。
这就是 Devin 中的自主测试旨在提供的。Devin 计划测试，操作应用程序，记录并注释发生的事情，最后返回使结果可审查的工件。仍有很多需要改进的地方，但我们认为这是未来的正确形态：不仅异步完成工作，而且带回证明的代理。
我们对 Devin 通过测试自己的工作为我们节省了多少时间不断感到惊讶，并觉得许多客户仍然利用不足 Devin 的自动化测试功能。为了支持实验，我们目前在测试模式下按正常使用成本的 1/5 计费。
请在 devin.ai 或 windsurf.com 尝试我们的工作。如果解决此类问题听起来很有趣，请联系 ido [at] cognition.ai