# Inside an Applied AI company
**作者**: Adithya Sanjay
**日期**: 2026-08-11T12:54:02.000Z
**来源**: [https://x.com/dittysan/status/2087160612971323723](https://x.com/dittysan/status/2087160612971323723)
---
The first time we run a new agent against a customer's real workflow, it gets just a little over 70% of it right. On a recent premium audit deployment, our agent read a mess of inbound documents, structured and reasoned over the data, and wrote it back to an antiquated policy admin system. Day one, the customer's own experts graded it at ~70% accuracy. The number we promised in our contract was >98%.
Everything interesting about my job lives in that gap. This post is about what closing it takes, and why I think Applied AI is the most interesting seat in the industry right now.
What it means to be an Applied AI company
At its core, an Applied AI company sells completed work. Unlike traditional software, a carrier doesn't buy a claims dashboard from us and staff people on it. They hand us an end-to-end workflow, our agents run it, and we own the outcome.
That difference shows up in how we're organized. Pace has three functions: engineering, GTM, and applied AI. Engineering builds the core platform and handles the sophisticated infrastructure-level work. GTM owns top-of-funnel, marketing, and getting customers to a point of interest. Applied AI is the bridge: we take a warm prospect, define the proof of value, scope the build, and get the agent into production.
Within applied AI, there are two roles: Agent PMs own the account and carry the customer from proof of value through production and beyond. They're technical enough to build sophisticated agents themselves without having to write production code. Forward deployed engineers write code, contribute to the product surface area, and are commercial enough to be in the room during sales conversations. That last part matters more than it sounds: trust with a large insurer comes from being honest about what you can promise, and the person who will actually build the thing is the most credible voice in the room.
Automating the job also enables a tight product feedback cycle. Since Applied AI closely works inside the customer's operation, they see what to build next and, often, can build it themselves. As a result, every deployment sharpens the product and improves customer outcomes. This tight feedback loop from customer → Applied AI → engineering not only shortens our time-to-value but also does some of the discovery work a product team traditionally would.
The harness is the product
A model alone can't run an insurance workflow. General-purpose models now excel at long-horizon tasks, but they struggle with messy multi-modal work - especially work that's out-of-distribution from their training data. Off the shelf, a model can't accurately read an ACORD form, rate a policy, and write the result back to the system of record in one flow - let alone do it thousands of times with an audit trail at 99.5% quality.
I'd break an AI system into three layers: the model, the interface, and the harness. Everything underneath that lets a model do real work inside real systems. The harness is the product. From a carrier's seat, the experience looks less like software and more like a new teammate: we give the harness the same tools a new hire would get. It goes on to read every inbound submission, works inside the same systems the other employees would use, and asks for review when it isn't sure.
Our harness is purpose-built for insurance. In practice, it entails five things:
- Tools: Models can reason but can't act, so we build the functions they invoke. An OCR router that picks a provider based on document complexity, because a handwritten ACORD form routes differently than a structured policy PDF. A calculator, because models still fumble arithmetic on premium calculations. A code sandbox for data transformations. A skills library for things like writing a submission back into a specific carrier's portal, in that carrier's format.
- Integrations: Enterprise insurance systems break every assumption that normal integrations make. Pace’s integrations can easily handle the gnarly enterprise APIs with minimal error handling, and work of the box for common builds. Plus, plenty of insurance systems have no API at all, so we built our own infrastructure to drive [browser-based workflows programmatically](https://withpace.com/news/continual-learning-in-computer-use).
- File systems: Done in isolation, a claim doesn't resolve in one sitting, and a multi-day workflow either persists state or blows up the context window and loses everything on interruption. Our agents [get a structured working directory](https://withpace.com/news/multimodal-agents) that enable them to read inputs, write intermediate outputs, pick up where they left off.
- Orchestration: Not every task goes to the same model. On a policy intake, a fast, cheap model classifies the inbound documents, a more capable one reasons through ambiguous coverage language, and a specialized model handles structured extraction. Above that sits the task graph which decides what work to run in parallel, what waits on validation, and when to spawn a sub-agent for the write-back into the policy admin system.
- Oversight: Insurers can’t hand an agent full autonomy on day one. To start off, low-confidence work goes to a review queue, irreversible actions wait for human approval, and audit trails record what the agent did and why. A policy layer controls what data the agent can touch under GDPR or on-prem constraints.
As models improve, large aspects of this will look different: the calculator probably goes away, routing simplifies, and at some point it makes sense to distill a fine-tuned model of our own. That said, carrier portals will still lack APIs, and regulators will still want to know who approved what. Giving AI the ability to act, persist, and integrate inside insurance workflows isn't going away, and the companies that build that layer tightly around a vertical will be hard to displace.
Evals as a core platform primitive
This is the part that gets the least attention and deserves the most. In an outcomes-based business, evals aren't a nice-to-have. You cannot promise a carrier 98% accuracy on policy intake, and price against the cost of the work, unless you can measure and prove that number, continuously.
That day-one run from beginning of this essay, where the agent works from the customer's SOP and lands at ~60%, is the start of a specific loop. It begins with golden test sets: a corpus of real inputs and correct outputs that the customer's SMEs have signed off on, built by sitting with them and pulling out edge cases. Then, getting to the 98%+ SLAs that gate production requires setting up the right grading loops to hillclimb. Once the agent is live the golden test runs keep going as drift detection, because the agent that hit 98% in UAT has to stay at 98% as documents and processes shift.
The standard we're held to is objective: the claim was either resolved correctly or it wasn't, at the accuracy SLA stated in the contract. More importantly, as our applied AI team works on the n+1st deployment, we actively scope out and build the next set of infrastructure to consistently meet our SLAs as well as reduce the overall deployment cycle time.
Why this problem is worth working on
The obvious question about any applied AI company is what compounds. The first instinct most people have when thinking about AI moats is data. Having spent time on the front lines of enterprise deployments, I'm increasingly skeptical that a traditional data moat exists at the app layer, at least in the way people assume.
In regulated industries, you don't get the data you'd want. No insurance carrier will hand us their underwriting data, no bank will give the labs access to their loan book, no health system will open up patient records, and none of them should. The data that would make a model smarter at the domain stays locked inside the enterprise. That's exactly why these industries persist as valuable verticals: the data is the moat for the customer, which means it can't also be the moat for the vendor.
What's actually hard, and what actually compounds, is everything a deployment builds up. As we concentrate more of an organizations operating knowledge in one place, the tacit knowledge, internal evals, and hardening accumulate to ensure consistency. Almost anyone can build an agent that completes a workflow once. But building one that gets it right 995 times out of 1000 every week, while documents drift and carriers change their portals, takes everything above (the harness, the eval loop).
There's a lot more to cover, including outcomes-based pricing and how a great applied AI org becomes the distribution engine of a vertical AI company. We'll cover those in a future post.
If you are interested, reach out!
## 相关链接
- [Adithya Sanjay](https://x.com/dittysan)
- [@dittysan](https://x.com/dittysan)
- [33K](https://x.com/dittysan/status/2087160612971323723/analytics)
- [browser-based workflows programmatically](https://withpace.com/news/continual-learning-in-computer-use)
- [get a structured working directory](https://withpace.com/news/multimodal-agents)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [8:54 PM · Aug 11, 2026](https://x.com/dittysan/status/2087160612971323723)
- [33.7K Views](https://x.com/dittysan/status/2087160612971323723/analytics)
- [View quotes](https://x.com/dittysan/status/2087160612971323723/quotes)
---
*导出时间: 2026/8/12 10:24:00*
---
## 中文翻译
# 走进一家应用型AI公司
**作者**: Adithya Sanjay
**日期**: 2026-08-11T12:54:02.000Z
**来源**: [https://x.com/dittysan/status/2087160612971323723](https://x.com/dittysan/status/2087160612971323723)
---
我们第一次将新的智能体部署到客户真实工作流中时，它的正确率只有70%多一点。在最近一次高级审计部署中，我们的智能体读取了一堆杂乱无章的入站文档，对数据进行结构化处理和推理，然后将结果写回一个过时的保单管理系统。上线第一天，客户自己的专家给它的准确率打了约70%。而我们在合同中承诺的数字是>98%。
我工作中所有有趣的部分，都存在于这个差距之中。这篇文章要讲的是，弥合这个差距需要什么，以及为什么我认为应用型AI是当下整个行业中最有意思的岗位。
## 应用型AI公司意味着什么
从本质上讲，应用型AI公司出售的是完成的工作。与传统软件不同，保险公司不会从我们这里买一个理赔仪表盘然后派人去操作。他们把端到端的工作流交给我们，由我们的智能体来运行，而我们为最终结果负责。
这种差异体现在我们的组织架构上。Pace有三个职能部门：工程、GTM和应用型AI。工程部门负责构建核心平台，处理复杂的基础设施层面的工作。GTM负责漏斗顶端、市场营销，以及让客户产生兴趣。应用型AI是桥梁：我们接手一个有意向的潜在客户，定义价值验证方案，规划构建范围，并将智能体推进到生产环境。
在应用型AI内部，有两个角色：智能体产品经理负责客户账户，带领客户从价值验证一路走到生产环境及之后。他们具备足够的技术能力，可以自己构建复杂的智能体，而无需编写生产级代码。前向部署工程师编写代码，为产品功能面做贡献，同时具备足够的商业意识，能够参与销售对话。最后这一点比听起来更重要：与大型保险公司的信任，来自于对你能承诺什么保持诚实，而真正负责构建的人，是房间里最可信的声音。
自动化工作还促成了紧密的产品反馈循环。由于应用型AI团队深入客户运营内部，他们能看到下一步该构建什么，而且往往能自己动手构建。因此，每一次部署都会打磨产品，改善客户成果。这种从客户→应用型AI→工程的紧密反馈循环，不仅缩短了我们的价值实现时间，还承担了一部分传统上由产品团队完成的探索工作。
## 框架（Harness）就是产品
单靠模型无法运行保险工作流。通用模型现在擅长长周期任务，但在处理杂乱的多模态工作时表现不佳——尤其是那些偏离其训练数据分布的工作。开箱即用的模型无法在一个流程中准确读取ACORD表格、对保单进行费率计算、并将结果写回记录系统——更不用说以99.5%的质量水平、带着审计追踪记录执行数千次了。
我会把AI系统分为三层：模型、界面和框架。框架是让模型在真实系统中完成真实工作的所有底层支撑。框架就是产品。从保险公司的视角来看，这种体验不像软件，更像一个新同事：我们给框架的，是一个新员工会得到的同样工具。它读取每一份入站提交，在与其他员工相同的系统中工作，不确定时请求人工审核。
我们的框架是为保险行业量身打造的。在实践中，它包含五个部分：
- **工具**：模型能推理但不能行动，所以我们构建它们调用的函数。一个OCR路由器，根据文档复杂度选择服务提供商，因为手写ACORD表格与结构化保单PDF的路由方式不同。一个计算器，因为模型在保费计算上仍然会算错。一个用于数据转换的代码沙箱。一个技能库，用于将提交内容以特定保险公司的格式写回该公司的门户。
- **集成**：企业保险系统打破了常规集成的所有假设。Pace的集成能够轻松处理那些棘手的、几乎不做错误处理的企业API，并且对常见构建开箱即用。此外，很多保险系统根本没有API，所以我们构建了自己的基础设施，以[编程方式驱动基于浏览器的工作流](https://withpace.com/news/continual-learning-in-computer-use)。
- **文件系统**：单独处理时，一个理赔无法在一次会话中解决，而一个多天的工作流要么持久化状态，要么就会撑爆上下文窗口，在中断时丢失一切。我们的智能体[获得一个结构化的工作目录](https://withpace.com/news/multimodal-agents)，使它们能够读取输入、写入中间输出、从中断处继续。
- **编排**：并非所有任务都交给同一个模型。在保单录入中，一个快速、廉价的模型对入站文档进行分类，一个能力更强的模型推理模糊的承保语言，一个专门的模型处理结构化提取。在这些之上是任务图，它决定哪些工作可以并行运行、哪些需要等待验证、以及何时为写回保单管理系统生成子智能体。
- **监督**：保险公司不可能第一天就给智能体完全自主权。一开始，低置信度的工作进入审核队列，不可逆操作等待人工批准，审计追踪记录智能体做了什么以及为什么这么做。一个策略层控制智能体在GDPR或本地部署约束下可以接触哪些数据。
随着模型的改进，这些方面中的很大一部分会发生变化：计算器可能会消失，路由会简化，在某个时点，提炼一个我们自己的微调模型是有意义的。话虽如此，保险公司的门户仍然不会有API，监管机构仍然想知道谁批准了什么。赋予AI在保险工作流中行动、持久化和集成能力这件事不会消失，而那些将这一层紧密构建在垂直行业之上的公司，将很难被取代。
## 评估（Evals）作为核心平台原语
这是最不受关注、却最值得关注的部分。在以结果为导向的业务中，评估不是可有可无的。你不可能向保险公司承诺保单录入98%的准确率、并按照工作成本定价，除非你能持续衡量并证明这个数字。
这篇文章开头提到的第一天运行——智能体按照客户的SOP工作，准确率约60%——是一个特定循环的起点。它始于黄金测试集：一组由客户领域专家签字确认的真实输入和正确输出的语料库，通过与他们坐下来一起梳理边缘案例来构建。然后，要达到进入生产环境所要求的98%+ SLA，需要建立正确的评分循环来逐步优化。一旦智能体上线，黄金测试会继续运行，作为漂移检测，因为在UAT中达到98%的智能体，必须在文档和流程变化时保持在98%。
我们被要求达到的标准是客观的：理赔要么被正确处理了，要么没有，以合同中规定的准确率SLA为准。更重要的是，当我们的应用型AI团队在推进第n+1个部署时，我们会主动规划并构建下一套基础设施，以持续满足我们的SLA，同时缩短整体部署周期。
## 为什么这个问题值得做
关于任何应用型AI公司，最明显的问题是：什么在复利积累？大多数人想到AI护城河时的第一反应是数据。在企业管理部署的前线待过之后，我越来越怀疑在应用层存在传统意义上的数据护城河——至少不是人们想象的那种方式。
在受监管的行业中，你得不到你想要的数据。没有保险公司会把承保数据交给我们，没有银行会给实验室访问其贷款组合的权限，没有医疗系统会开放患者记录——它们也不应该这么做。那些能让模型在领域内变得更聪明的数据，被锁在企业内部。这正是这些行业作为有价值的垂直领域持续存在的原因：数据是客户的护城河，这意味着它不可能同时成为供应商的护城河。
真正困难的、真正在复利积累的，是每次部署所积累的一切。当我们把组织越来越多的运营知识集中在一个地方时，隐性知识、内部评估和加固都在不断积累，以确保一致性。几乎任何人都能构建一个完成一次工作流的智能体。但要构建一个每周在1000次中做对995次、同时文档在漂移、保险公司在更换门户的智能体，需要上面提到的所有东西（框架、评估循环）。
还有很多内容可以讲，包括基于结果的定价，以及一个优秀的应用型AI组织如何成为垂直AI公司的分销引擎。我们会在后续文章中介绍。
如果你感兴趣，欢迎联系我们！
## 相关链接
- [Adithya Sanjay](https://x.com/dittysan)
- [@dittysan](https://x.com/dittysan)
- [33K](https://x.com/dittysan/status/2087160612971323723/analytics)
- [browser-based workflows programmatically](https://withpace.com/news/continual-learning-in-computer-use)
- [get a structured working directory](https://withpace.com/news/multimodal-agents)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [8:54 PM · Aug 11, 2026](https://x.com/dittysan/status/2087160612971323723)
- [33.7K Views](https://x.com/dittysan/status/2087160612971323723/analytics)
- [View quotes](https://x.com/dittysan/status/2087160612971323723/quotes)
---
*导出时间: 2026/8/12 10:24:00* 
