# How We Rebuilt Playbook Review as a Multi-Agent System
**作者**: Pablo Felgueres
**日期**: 2026-09-02T18:34:26.000Z
**来源**: [https://x.com/pfelgueres/status/2095218809284309307](https://x.com/pfelgueres/status/2095218809284309307)
---
Companies run their businesses with contracts and almost all contracts get negotiated. According to Gartner, legal departments spend [21% of their capacity](https://www.gartner.com/en/webinar/921279/1920428-the-new-ai-augmented-contract-tech-stack) on contracting, which is more than any other activity. Reviewing a contract typically takes between 20 minutes to five hours, depending on contract type and size.
When a company receives a contract from a counterparty, someone in their legal department compares it, clause by clause, against their company’s playbook; an internal rulebook where companies codify which terms are acceptable, negotiable, or dealbreakers. Then they mark up the document with redlines, comments, and send it back to the counterparty for review. This goes on until both parties agree to mutually agreeable terms.
Earlier this year, we rebuilt Harvey's playbook review engine from the ground up to make it even more effective in reducing the toil of contract review for our users. In this post, we walk through how the old system worked, why we made that change, how we evaluate quality of this problem (which is very subjective and relies on legal judgement), and how this new multi-agent system analyzes a document against a playbook, flags risks, and proposes surgical redline suggestions that lawyers can trust.
## Why Contract Review is Harder Than it Looks
Playbook rules are conceptually simple, and include the following elements:
- Standard position: the ideal term a company wants to see in their contracts
- Acceptable deviations: fallback provisions a company is willing to negotiate
- Unacceptable deviations: dealbreakers for the contract
- Guidance: how to interpret and apply the rule given specific deal context
- Optional: whether or not it is ok for a clause to be absent in a contract
But there are also a few reasons why reviewing a real contract is complicated, especially for AI systems:
Contracts include complex conditional logic: Contracts have complex conditional logic and clauses span pages, reference each other, and describe when certain conditions are triggered and not. And even after creating a playbook, a counterparty’s contract will almost never use your playbook’s exact words. Positions are often qualified with conditional logic. An agent reviewing a clause, therefore, requires comprehending meaning and repercussions of these conditional, syntatic vs semantic differences instead of word matching.
Rules interact across a document: A single rule about assignment rights can touch four different sections in the document and at the same time, another rule might propose an edit on the same sentence creating a conflict. This complicates parallelizing review.
The right answer depends on deal context: Lawyers don’t review contracts they have written (First Party Paper) the same way they review counterparty-written contracts (Third Party Paper). The way lawyers negotiate contracts depends on a lot of factors: Which party do you represent? Is the contract on your paper or the counterparty's? Is this round one of the negotiations or round seven? The same clause can be fine in one deal posture and unacceptable in another. If the counterparty rejected the previous suggestion, the same one again isn’t a good idea to propose.
The best edit is usually the smallest one: Every redline you send gets scrutinized by an opposing counsel. If a clause needs three words changed and an agent rewrites the entire clause it will both slow down a deal and create extra work for human reviewers. Lawyers call this taking the "lightest touch," and it's a quality bar that out-of-the-box AI models and agents miss.
Some decisions cannot be delegated: A playbook alone cannot capture the nuance of a deal. If the counterparty is an important customer, a contract reviewer will likely give them more leeway. A good system identifies when a clause requires escalating to a human with the reasoning laid out based on the context of a deal.
Along with the nuances of document understanding in a specific deal context, handling the scale of documents and playbooks, which can be hundreds of pages long, is also a challenge given LLM response quality degrades with more tokens. For the Harvey platform, we need to ensure we maintain the same high quality on real contracts that can span hundreds of pages and still produce edits in a timely fashion.
## Our First System: A Pipeline of Prompts
Like most production LLM systems of its era, Harvey's original playbook review was a workflow: a fixed sequence of model calls, orchestrated in code, each with a carefully engineered prompt.
The system consisted of two phases, classification and redline. Classification ran as a waterfall of status match. For each batch of rules, one model call asked: Does the contract match the rule's standard position? Rules that failed moved to a second call checking acceptable deviations match; rules that failed moved to a third call checking unacceptable deviations. Once classifications were done, a separate retrieval step then located supporting text in the document, and a final call wrote the summary.
Redlines came from another single model call: given one clause, one position, and a list of the contract's defined terms, the model returned a revised version of the clause. We diffed the revision against the original, and that diff became the tracked change.
Previous rule-based approach to redlining contracts
This design had its pros and cons. It was predictable, debuggable, and cost-effective. Every call had one job. But each stage of the pipeline hard-coded an assumption about how contract review decomposes and those assumptions caused the system to struggle in complex playbook reviews:
- Context got lost between LLM calls and subtasks of contract review: Since each LLM call worked on its own, the context of classification got lost during citations and redlines. This would sometimes lead to inconsistencies in final output.
- Edits were scoped by clause: The redline model call got the context of one clause at a time and could only edit or delete it. It couldn't move language to the section where it belonged, add a missing clause, or keep an edit consistent with the rest of the document. Suggested text sometimes landed in the wrong place or related sections drifted out of order.
- Suggestions were made in isolation: Every rule was evaluated independently, so two rules could propose contradictory edits to the same sentence, and there was no rewriting for those edge cases. End users would have to manually fix these mistakes. Rules and documents that have complex dependencies across paragraphs also suffered because a single LLM call with limited budget won’t be able to think through and give accurate output to solve complex documents and playbooks.
With these limitations, reviews could over-redline, miss flags, insert suggestions in the wrong location, and miss whose paper the contract was on. Incremental prompt fixes moved individual metrics, but ultimately we needed a different architecture. And before choosing one, we needed a benchmark to measure improvements.
## Making Legal Judgment Measurable
When we started this revamp, no public benchmark captured this workflow end to end, so we built our own evaluation suite with our in-house legal team.
The core dataset pairs contracts with playbooks for various contract types across hundreds of provisions. The benchmark evaluates different aspects of contract review:
1. Risk classification: Ability to identify and categorize issues by their level of risk.
2. Redline quality: A list of considerations that a revision by a competent lawyer would have, both in substance and style. These rubrics are defined for each example.
Flagging risk is best understood and scored as a classification problem which is a straightforward task. Redline quality is more complex in that it requires judgment on whether an edit is minimal, correctly placed, and legally correct. After sourcing the rubrics with lawyers, we set up an evaluation framework that uses LLM judges to score against these rubrics. We then used a committee of three frontier models to score independently and aggregate the votes.
## The Multi-Agent System
With evals in place, we explored a few architectures of increased complexity and graded them across quality, latency, and complexity:
| Approach | Quality | Latency | Complexity |
| --- | --- | --- | --- |
| Rule-based LLM workflow | Moderate | Excellent | Low |
| Single agent reviewer | Excellent | Unacceptable | Moderate |
| Orchestrator agent with subagents | Excellent | Good | High |
Single Agent Reviewer
Agentic systems are powerful because they allow LLMs to keep solving a problem until certain conditions are met. This works well at scale because an agent can be equipped with tools to search for various key parts of a document, then look up rules and other information needed until it's satisfied with finding a redlined document that will work for a given deal and playbook.
To prove this hypothesis, we built a single agent with context of the review document and playbook and allowed it to iterate on the document using doc editing tools until it met requirements of the playbook. This was the simplest agentic solution and performed well on the benchmark, but was very high latency and would make for a poor user experience.
Orchestrator Agent With Subagents
We proved with a single agent prototype that quality problems can be solved with an agent. Often the best way to improve on latency is to find ways to parallelize work. With that idea in mind, we implemented an orchestrator-worker pattern consisting of a lead agent in charge of distributing work between parallel agents. Each of these agents in turn focus on a specific task, in this case reviewing a rule, and finally the lead agent reconciles the results to ensure final output is of good quality.
We found that this architecture is a good balance between quality, latency, and engineering complexity:
Worker agents work on individual rules: The orchestrator is prompted as the lead lawyer on the review. It spawns a team of subagents to review each playbook rule, running up to dozens in parallel. Each subagent works the way an associate would: it reads its rule, reads or searches the contract, decides whether the contract complies with the playbook, chooses which position to pursue (standard or a specific fallback), and then drafts a list of suggested edits. Crucially, it is an agent, not a function: if a clause references an exhibit, it goes and reads the exhibit. To make it fast and precise for it to reference and search for various parts of a document, we gave each component of the document a unique identifier. This allows agents to cite and edit a specific element unambiguously. Each subagent concludes its work by filing a short memo that explains its classification of a rule, edits to the contract, and rationale for its steps.
Worker agents work on a copy of the document: Dozens of agents editing one document concurrently would be chaos, as they often would need to apply edits to the same paragraph or sentence while addressing different rules. This would result in an inconsistent state in the document as previous work done by one agent would get overwritten by another agent. To solve this, we built a branching mechanism where each subagent works on its own branch of a versioned document model, and every proposed edit is a tracked change on that branch, tagged with the rule that produced it. When subagents finish, a reconciliation step merges branches the same way a version-control system would: non-conflicting edits apply cleanly to the final doc, and collisions of two rules touching the same text are escalated for the lead agent to resolve.
Playbook Review as a Multi-Agent System
The orchestrator ensures everything gets reviewed: The orchestrator has the task to dispatch reviewers, collect their work, and resolve conflicts in the document the way a senior lawyer would by drafting edits that satisfies all conflicting rules. It then takes a final pass to validate the quality of the review as a whole. This ensures that every rule gets classified and has corresponding edits when needed.
The orchestrator and workers share context: The system takes into account the same context that lawyers use while reviewing a contract: which party they represent, whether the contract is on your template or theirs, how strict or permissive this negotiation should be, and any deal-specific instructions. Reviews can also draw on precedent documents the user attaches as additional context. Every agent in the hierarchy sees this context and adjusts their behavior accordingly.
State persistence to ensure continuity: Every agent's state — classifications, chosen positions, edit summaries, reasoning — persists with the review. When a user asks a follow-up question, switches chosen position, or applies some suggestions and wants the rest reconsidered, we reuse the relevant agents rather than re-running the full agentic loop end to end.
## Eval Results
We saw major improvements in our ability to flag and classify risks and redline quality. On the other hand, reviewing a contract with this system takes more time, but it’s a reasonable tradeoff to make as we are optimizing for the quality of output.
| Metric | Before | After | Change |
| --- | --- | --- | --- |
| Risk classification | 59% | 77% | +18% |
| Redline rubric | 53% | 87% | +34% |
| Average latency (minutes) | 2.6 | 3.8 | +47% |
## Making it Scalable
The agentic redesign improved review quality but it also introduced new challenges. As a result of using agents to complete work end to end, tokens and overall system latency increased significantly, especially on long contracts or playbooks with high rule count. Our system performed well in offline evals but to enable it to scale in real production, we built a set of guardrails around the agent team to keep reviews fast, reliable, and within our infrastructure limits:
- Custom timeouts and targeted retries. During our eval runs, some model calls occasionally got stuck in unproductive loops on unusual clauses. The issue would self-resolve usually on retry. To mitigate this, we set timeouts by model, review phase, and document size, then retry only the affected rule. This prevented one bad subagent from derailing the entire review.
- Limiting concurrency. Since every rule in a playbook fans out a subagent, we added concurrency limits and exponential backoffs to avoid overwhelming shared model infrastructure.
- Prompt caching. On long documents and playbooks, input tokens would explode exponentially, resulting in prohibitive cost of doing playbook review. However, each rule reviews much of the same underlying document. We used that fact to redesign our prompts to optimize for prefix cache hits. That helped reduce model cost by a significant amount and improved latency.
- Stream results. Even after all the optimizations, time to first result was high. To mitigate this, we enabled streaming results such that each rule’s result becomes available as soon as its subagent finishes. This allows the end user to begin reviewing outputs within seconds instead of waiting for the final result. Streaming also lets us expose progress and isolate slow or retried rules without blocking the rest of the review.
- Using the best model for each task. Our evaluation showed that no single model performed best across subtasks of contract review. We built the system to switch models across phases of the same review, selecting each one based on its measured quality, latency, and cost for that specific task.
- A multi-model harness. We built our agent harness to normalize the differences between model providers. This helps future-proof the system since we are able to quickly swap to the best performing models with minimal overhead.
## What Agentic Review Unlocks
We are excited about the possibilities that reliable first pass reviews unlock.
Contract reviews can now run in the background. Whenever a new contract arrives over email or a shared file system, Harvey kicks off a review workflow, frontloading the work and giving users a well reasoned draft as a starting point.
Closing the loop with self-improvement becomes possible. Every accepted, edited, or rejected redline helps agents understand a legal team's preferences. This will allow us to build towards a personalized system, from users to accounts, such that every new contract needs less guidance and effort than the last one.
A team’s historical contracts become a new input to inform better reviews. This opens the ability to search across thousands of documents for previous examples to trust agent outputs and quickly understand what a team has approved in the past and why.
This is what we are building towards with [Contract Intelligence](https://www.harvey.ai/blog/introducing-contract-intelligence), so every contract review improves the next and legal teams can stay in control as volume grows.
Authors: [@pfelgueres](https://x.com/@pfelgueres), Maharshi Patel, Zach Huang Acknowledgements
Thank you to the following members of Harvey’s team for their work on Playbook Review: Karl De La Roche, Rina Kim, Scott Werwath, Jianan Zhang, Sakshi Pratap, Kunal Baweja, Will Huang, Andy Pham, and Alex Conrad-Dormoy
## 相关链接
- [Pablo Felgueres](https://x.com/pfelgueres)
- [@pfelgueres](https://x.com/pfelgueres)
- [22K](https://x.com/pfelgueres/status/2095218809284309307/analytics)
- [21% of their capacity](https://www.gartner.com/en/webinar/921279/1920428-the-new-ai-augmented-contract-tech-stack)
- [Contract Intelligence](https://www.harvey.ai/blog/introducing-contract-intelligence)
- [@pfelgueres](https://x.com/@pfelgueres)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [2:34 AM · Sep 3, 2026](https://x.com/pfelgueres/status/2095218809284309307)
- [22.9K Views](https://x.com/pfelgueres/status/2095218809284309307/analytics)
- [View quotes](https://x.com/pfelgueres/status/2095218809284309307/quotes)
---
*导出时间: 2026/9/3 23:30:58*
---
## 中文翻译
# 我们如何将 Playbook 审查重建为多智能体系统
**作者**: Pablo Felgueres
**日期**: 2026-09-02T18:34:26.000Z
**来源**: [https://x.com/pfelgueres/status/2095218809284309307](https://x.com/pfelgueres/status/2095218809284309307)
---
企业依靠合同来运营业务，而几乎所有合同都要经过谈判。根据 Gartner 的数据，法务部门将[21% 的精力](https://www.gartner.com/en/webinar/921279/1920428-the-new-ai-augmented-contract-tech-stack)花在合同工作上，超过其他任何活动。审查一份合同通常需要 20 分钟到 5 个小时不等，具体取决于合同的类型和规模。
当一家公司收到交易对手发来的合同时，其法务部门会将其与公司的 playbook（内部规则手册，企业在此中明确哪些条款可接受、可谈判或不可接受）逐条比对。然后他们对文档进行修订标记、添加批注，并将其发回给交易对手审查。如此往复，直到双方就彼此均可接受的条款达成一致。
今年早些时候，我们从零开始重建了 Harvey 的 playbook 审查引擎，使其能更有效地为用户减少合同审查的负担。在这篇文章中，我们将介绍旧系统的工作原理、我们做出这一改变的原因、我们如何评估这一问题的质量（这个问题非常主观，依赖于法律判断），以及这个新的多智能体系统如何对照 playbook 分析文档、标记风险，并提出律师可以信赖的精准修订建议。
## 为什么合同审查比看起来更难
Playbook 规则在概念上很简单，包含以下要素：
- 标准立场：公司希望在其合同中看到的理想条款
- 可接受的偏差：公司愿意谈判的退让条款
- 不可接受的偏差：合同的“红线”条款
- 指引：在特定交易背景下如何解释和应用该规则
- 可选性：某条款在合同中缺失是否可以接受
但审查真实合同之所以复杂，尤其对 AI 系统而言，还有以下几个原因：
合同包含复杂的条件逻辑：合同具有复杂的条件逻辑，条款跨越多页、相互引用，并描述某些条件何时被触发或不触发。而且即使创建了 playbook，交易对手的合同也几乎不会使用与你 playbook 完全相同的措辞。立场常常带有条件性限定。因此，审查条款的智能体需要理解这些条件性差异、句法与语义差异的含义及影响，而不是进行简单的词语匹配。
规则在整个文档中相互作用：一条关于转让权的规则可能涉及文档中的四个不同部分，同时另一条规则可能对同一句话提出修改，从而产生冲突。这使并行审查变得复杂。
正确的答案取决于交易背景：律师审查自己起草的合同（第一方文件）与审查交易对手起草的合同（第三方文件）的方式不同。律师谈判合同的方式取决于许多因素：你代表哪一方？合同用的是你的模板还是对方的？这是谈判的第一轮还是第七轮？同一条款在一种交易态势下可以接受，在另一种态势下却不可接受。如果交易对手之前拒绝了某项建议，再次提出同样的建议就不是一个好主意。
最佳的修改通常是最小的修改：你发出的每一处修订标记都会被对方律师仔细审查。如果一个条款只需要修改三个词，而智能体重写了整个条款，那既会拖慢交易进度，也会给人工审查者增加额外工作。律师称之为“最轻触”，这是开箱即用的 AI 模型和智能体往往达不到的质量标准。
有些决策不能被委派：仅靠 playbook 无法捕捉一笔交易的全部细微差别。如果交易对手是重要客户，合同审查者可能会给他们更多余地。一个好的系统需要识别出何时某条款需要升级给人工处理，并根据交易背景给出理由。
除了在特定交易背景下理解文档的细微差别之外，处理可能长达数百页的文档和 playbook 的规模也是一大挑战，因为 LLM 的响应质量会随着 token 数量的增加而下降。对于 Harvey 平台而言，我们需要确保在可能长达数百页的真实合同上依然保持同样高的质量，并能及时产出修改建议。
## 我们的第一套系统：提示词流水线
与那个时代的大多数生产级 LLM 系统一样，Harvey 最初的 playbook 审查是一个工作流：代码编排的固定模型调用序列，每个调用都有精心设计的提示词。
该系统由两个阶段组成：分类和修订标记。分类以状态匹配的瀑布方式运行。对于每一批规则，一次模型调用会问：合同是否符合该规则的标准立场？不匹配的规则进入第二次调用，检查是否符合可接受的偏差；再不匹配的规则进入第三次调用，检查是否符合不可接受的偏差。分类完成后，由一个独立的检索步骤在文档中定位支持性文本，最后一次调用生成摘要。
修订标记来自另一次模型调用：给定一个条款、一个立场和合同中定义术语的列表，模型返回该条款的修订版本。我们将修订版与原文进行对比，该差异就成为修订记录。
此前基于规则的合同修订标记方法
这种设计有其优缺点。它可预测、可调试且成本低。每次调用只做一件事。但流水线的每个阶段都对合同审查的分解方式硬编码了假设，这些假设导致系统在复杂的 playbook 审查中表现不佳：
- LLM 调用与合同审查子任务之间丢失上下文：由于每次 LLM 调用独立运行，分类的上下文在引用和修订标记过程中会丢失。这有时会导致最终输出不一致。
- 修改以条款为单位：修订标记的模型调用每次只获得一个条款的上下文，且只能编辑或删除它。它无法将措辞移动到其应在的部分，无法添加缺失的条款，也无法保持修改与文档其余部分一致。建议的文本有时会落在错误的位置，或者相关部分之间失去连贯性。
- 建议孤立生成：每条规则独立评估，因此两条规则可能对同一句话提出相互矛盾的建议，而系统没有针对这些边缘情况的改写机制。最终用户不得不手动修复这些错误。那些在段落间有复杂依赖关系的规则和文档也会受到影响，因为预算有限的单次 LLM 调用无法充分思考并为复杂文档和 playbook 提供准确的输出。
由于这些限制，审查可能过度修订、遗漏风险标记、将建议插入错误位置，或误判合同用的是谁的模板。通过增量式提示词修复虽然能改善个别指标，但归根结底我们需要一种不同的架构。而在选择架构之前，我们需要一个基准来衡量改进。
## 让法律判断可衡量
当我们开始这次重构时，没有公开的基准可以端到端地覆盖这一工作流，因此我们与内部法务团队一起构建了自己的评估套件。
核心数据集将各类合同类型的合同与 playbook 配对，覆盖数百个条款。该基准从不同维度评估合同审查：
1. 风险分类：按风险等级识别和归类问题的能力。
2. 修订标记质量：一名合格律师进行修订时会考虑的一系列因素，涵盖实质和文风。这些评分标准针对每个样例定义。
风险标记最好被理解和评分为一个分类问题，这是比较直接的任务。修订标记质量则更为复杂，因为它需要判断一项修改是否足够精简、位置是否正确、法律上是否正确。在律师提供评分标准后，我们搭建了一个评估框架，使用 LLM 评审员按照这些标准评分。然后我们使用三个前沿模型组成的评审委员会进行独立评分并汇总投票。
## 多智能体系统
有了评估体系，我们探索了几种复杂度递增的架构，并从质量、延迟和复杂度三个维度进行评估：
| 方案 | 质量 | 延迟 | 复杂度 |
| --- | --- | --- | --- |
| 基于规则的 LLM 工作流 | 中等 | 优秀 | 低 |
| 单智能体审查者 | 优秀 | 不可接受 | 中等 |
| 带子智能体的编排者智能体 | 优秀 | 良好 | 高 |
单智能体审查者
智能体系统的强大之处在于它们能让 LLM 持续解决问题，直到满足某些条件。这在大规模场景下效果很好，因为智能体可以配备工具来搜索文档的各个关键部分，然后查阅规则和其他所需信息，直到它找到一份适用于特定交易和 playbook 的修订文档为止。
为了验证这一假设，我们构建了一个拥有审查文档和 playbook 上下文的单一智能体，并允许它使用文档编辑工具对文档进行迭代，直到满足 playbook 的要求。这是最简单的智能体方案，在基准测试中表现出色，但延迟非常高，会带来糟糕的用户体验。
带子智能体的编排者智能体
我们通过单智能体原型证明了质量问题可以通过智能体解决。而改善延迟的最好办法往往是找到并行化的方式。基于这一思路，我们实现了一种编排者-工作者模式，由一个负责在多个并行智能体之间分配工作的主导智能体组成。这些智能体各自专注于特定任务（本例中为审查一条规则），最后由主导智能体汇总结果，确保最终输出质量良好。
我们发现这种架构在质量、延迟和工程复杂度之间取得了很好的平衡：
工作者智能体处理单条规则：编排者被设定为审查中的首席律师角色。它生成一组子智能体来审查每条 playbook 规则，最多可并行运行数十个。每个子智能体的工作方式如同一位初级律师：阅读其规则、阅读或搜索合同、判断合同是否符合 playbook、选择 pursuing 哪种立场（标准立场或某个特定退让立场），然后起草一份建议修改清单。至关重要的是，它是一个智能体，而不是一个函数：如果某个条款引用了一个附件，它会去阅读该附件。为了让它能快速、精准地引用和搜索文档的各个部分，我们为文档的每个组件分配了唯一标识符。这使智能体能够无歧义地引用和编辑特定元素。每个子智能体最后会提交一份简短的备忘录，说明它对该规则的分类、对合同的修改及其步骤的理由。
工作者智能体在文档副本上工作：数十个智能体同时编辑一份文档将是一片混乱，因为它们在处理不同规则时常常需要对同一段落或句子进行修改。这会导致文档处于不一致状态，因为一个智能体先前完成的工作会被另一个智能体覆盖。为解决这一问题，我们构建了分支机制：每个子智能体在版本化文档模型的自己的分支上工作，每项建议的修改都是该分支上的一条修订记录，并带有产生它的规则标签。当子智能体完成工作后，由一个汇总步骤像版本控制系统那样合并分支：无冲突的修改干净地应用到最终文档，而两条规则触及同一文本的冲突则上报给主导智能体解决。
作为多智能体系统的 Playbook 审查
编排者确保一切都被审查：编排者的任务是分派审查者、收集他们的工作，并以资深律师的方式解决文档中的冲突——通过起草满足所有冲突规则的修改。然后它会进行最后一轮审查，验证整体审查质量。这确保每条规则都得到分类，并在需要时有相应的修改。
编排者和工作者共享上下文：系统会考虑律师审查合同时所用的相同上下文：他们代表哪一方、合同用的是你的模板还是对方的、这次谈判应该多严格或多宽松，以及任何针对特定交易的指示。审查还可以利用用户附加的先例文档作为额外上下文。层级中的每个智能体都能看到这些上下文，并相应地调整其行为。
状态持久化以确保连续性：每个智能体的状态——分类结果、所选立场、修改摘要、推理过程——都随审查一起持久保存。当用户提出后续问题、切换所选立场，或应用了部分建议并希望对其余部分重新考虑时，我们会复用相关的智能体，而不是端到端地重新运行整个智能体循环。
## 评估结果
我们在标记和分类风险的能力以及修订标记质量方面看到了显著提升。另一方面，用这套系统审查一份合同需要更多时间，但这是一个合理的权衡，因为我们优化的是输出质量。
| 指标 | 之前 | 之后 | 变化 |
| --- | --- | --- | --- |
| 风险分类 | 59% | 77% | +18% |
| 修订标记评分 | 53% | 87% | +34% |
| 平均延迟（分钟） | 2.6 | 3.8 | +47% |
## 实现可扩展性
智能体化重构提升了审查质量，但也带来了新的挑战。由于使用智能体端到端地完成工作，token 消耗和整体系统延迟显著增加，尤其是在长合同或规则数量较多的 playbook 上。我们的系统在离线评估中表现良好，但要使其在生产环境中规模化，我们围绕智能体团队构建了一套防护机制，以保持审查的快速、可靠，并控制在我们基础设施的承受范围内：
- 自定义超时和针对性重试。在评估运行期间，一些模型调用偶尔会在异常条款上陷入低效循环。这个问题通常在重试时会自行解决。为缓解此问题，我们按模型、审查阶段和文档大小设置了超时，然后仅重试受影响的规则。这防止了一个表现不佳的子智能体拖垮整个审查。
- 限制并发。由于 playbook 中的每条规则都会派生一个子智能体，我们添加了并发限制和指数退避，以避免压垮共享的模型基础设施。
- 提示词缓存。在长文档和长 playbook 上，输入 token 会呈指数级增长，导致 playbook 审查的成本高得令人望而却步。然而，每条规则审查的都是同一底层文档的大部分内容。我们利用这一事实重新设计提示词，以优化前缀缓存命中率。这大幅降低了模型成本并改善了延迟。
- 流式输出结果。即使进行了所有优化，首个结果的产出时间仍然较长。为缓解这一问题，我们启用了结果流式传输，使每条规则的结果在其子智能体完成时立即可用。这让最终用户能在几秒内开始审查输出，而无需等待最终结果。流式传输还让我们能够展示进度，并隔离缓慢或重试的规则，而不阻塞审查的其余部分。
- 为每个任务使用最佳模型。我们的评估显示，没有单一模型在合同审查的所有子任务中都表现最佳。我们构建的系统可以在同一次审查的不同阶段切换模型，并根据每个特定任务所测得的质量、延迟和成本来选择模型。
- 多模型框架。我们构建的智能体框架可以规范化不同模型提供商之间的差异。这有助于系统面向未来，因为我们能够以最小的开销快速切换到性能最佳的模型。
## 智能体审查带来的可能性
我们对可靠的首轮审查所开启的可能性感到兴奋。
合同审查现在可以在后台运行。每当有新合同通过电子邮件或共享文件系统送达时，Harvey 会启动审查工作流，前置处理这些工作，为用户提供一份论证充分的草稿作为起点。
自我改进的闭环成为可能。每一处被接受、编辑或拒绝的修订标记都能帮助智能体理解法务团队的偏好。这将使我们能够构建一个从用户到账户层面的个性化系统，让每一份新合同所需指导和投入都比上一份更少。
团队的历史合同成为改进审查质量的新输入。这打开了在数千份文档中搜索过往案例的能力，从而信任智能体的输出，并快速了解团队过去批准了什么以及为什么批准。
这正是我们通过 [Contract Intelligence](https://www.harvey.ai/blog/introducing-contract-intelligence) 致力实现的目标：让每一次合同审查都能改进下一次，并让法务团队在业务量增长时始终保持掌控。
作者：[@pfelgueres](https://x.com/@pfelgueres)、Maharshi Patel、Zhuang Huang 致谢
感谢以下 Harvey 团队成员在 Playbook 审查方面的工作：Karl De La Roche、Rina Kim、Scott Werwath、Jianan Zhang、Sakshi Pratap、Kunal Baweja、Will Huang、Andy Pham 和 Alex Conrad-Dormoy
## 相关链接
- [Pablo Felgueres](https://x.com/pfelgueres)
- [@pfelgueres](https://x.com/pfelgueres)
- [22K](https://x.com/pfelgueres/status/2095218809284309307/analytics)
- [21% 的精力](https://www.gartner.com/en/webinar/921279/1920428-the-new-ai-augmented-contract-tech-stack)
- [Contract Intelligence](https://www.harvey.ai/blog/introducing-contract-intelligence)
- [@pfelgueres](https://x.com/@pfelgueres)
- [升级到 Premium](https://x.com/i/premium_sign_up)
- [2026年9月3日 凌晨2:34](https://x.com/pfelgueres/status/2095218809284309307)
- [22.9K 次浏览](https://x.com/pfelgueres/status/2095218809284309307/analytics)
- [查看引用](https://x.com/pfelgueres/status/2095218809284309307/quotes)
---
*导出时间: 2026/9/3 23:30:58* 
