# Post-training RLM agents for end-to-end M&A Diligence
**作者**: Niko
**日期**: 2026-09-08T17:01:19.000Z
**来源**: [https://x.com/nikogrupen/status/2097369705791307952](https://x.com/nikogrupen/status/2097369705791307952)
---
Article link: [https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence](https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence)
In our recent [research preview for Harvey Tenet](https://www.harvey.ai/blog/post-training-update-harvey-tenet), we highlighted the importance of model-harness co-optimization for solving complex, end-to-end legal tasks, and previewed initial results on M&A Diligence, where a single task requires traversing up to 80M tokens of document context. Today, we are sharing updated results from our continued experimentation.
Together with Baseten, we built a [recursive language model ](https://arxiv.org/abs/2512.24601)(RLM) harness for M&A Diligence. In our RLM formulation, a complete dataroom is loaded into a Python REPL and a root agent delegates bounded review and analysis to sub-agents that work within their own context windows. On [LAB Diligence](https://www.harvey.ai/blog/legal-agent-bench-m-and-a-due-diligence) tasks, this harness improves rubric criteria pass rate by 39.1 percentage points on average across seven models that we evaluated.
We find that post-training within the RLM harness yields further performance gains and options for navigating quality-efficiency trade-offs. We post-trained a Qwen3.5-122B-A10B orchestrator via reinforcement learning and found that post-training increased rubric criteria pass rate from 29.9% to 63.0% on 50 held-out LAB Diligence datarooms.
Figure 1: Mean criteria pass rate on LAB Diligence for base models in the standard tool-loop harness, coding agents in their own harnesses, and models in our RLM harness. Asterisks mark the matched GLM-5.2 results before and after SFT, evaluated on a separate 20-room hold-out set. All other results use the 50-room hold-out set.
These results suggest that post-training within a task-specific harness is a practical route for document-intensive legal work, like M&A Diligence, and that scaling RL within the RLM harness is a promising direction for future work. For this reason, we are also training GLM-5.3, a large frontier open weight model, as the root model in an ongoing scale-up run.
In the rest of this post, we describe the LAB Diligence environments that we evaluated, our methodology, and post-training experiments in more detail.
## Environments for M&A Diligence
We recently introduced [LAB Diligence](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark), an extension of [LAB](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark) that includes synthetic environments for M&A Diligence. A single data room in LAB Diligence contains up to 5,000 documents, organized across dozens of folders by category, and up to 80M tokens of total context.
Figure 2: An example LAB Diligence dataroom. Aravon Bridge Bank contains 2,270 documents in 82 folders across 14 categories, 31M tokens in total. The agent's diligence memo for this task is graded against 571 rubric criteria.
Working within this dataroom, the agent is tasked with producing a complete diligence memo, including its findings with document citations, quantified exposure where relevant, and recommended next steps for the transaction. An LLM judge scores the diligence memo against an expert rubric with hundreds of pass or fail criteria. The Aravon Bridge Bank example has 571 rubric criteria in total.
Figure 3: Rubric examples for the Aravon Bridge Bank task in LAB Diligence, with one verbatim criterion of each type.
The evidence needed for diligence is spread across the dataroom. Some findings require combining several documents, while others require checking whether supporting evidence is missing. In our baseline runs, agents searched and read selectively, leaving much of the dataroom unexamined. These tasks call for a harness that can distribute review across many bounded contexts and bring the findings together.
## An RLM Harness for M&A Diligence
To establish a baseline, we started with the standard tool-loop harness from[ Legal Agent Bench](https://github.com/harveyai/harvey-labs). In this harness, base models pass 23.3% of rubric criteria on average across the 50 held-out datarooms, and no model passes every criterion on any dataroom.
Figure 4: Mean criteria pass rate in the standard tool-loop harness on the 50 held-out diligence datarooms.
These results suggest a mismatch between task and harness. A LAB Diligence dataroom is too large to fit within a single model context, but much of the initial review can be divided by category. The root can then combine findings across categories to produce the memo. Deal teams within law firms divide diligence work in a similar way.
Figure 5: The RLM harness at depth-1. The dataroom is loaded into a Python REPL as queryable variables. The root agent operates on it in code and delegates bounded reading tasks to subagents, which return their findings as REPL variables. Only the output the root agent prints enters its context window, so the root never holds the full dataroom.
This led us to explore RLMs as a harness for M&A Diligence. In an RLM harness, a root agent is given a Python REPL with the dataroom loaded as queryable variables, which lets it search the corpus programmatically. The root agent plans and scopes the review and dispatches sub-tasks to sub-agents. Each sub-agent receives a bounded slice of the corpus and instructions from the root agent, works within its own context window, and returns its findings as REPL variables. The experiments below use a single layer of sub-agents unless noted, and in practice the root dispatches many calls in parallel.
Figure 6: Mean criteria pass rate for seven models in the standard tool-loop harness and as the root of the RLM harness (depth-1, Qwen3.6-35B-A3B sub-agents), on the 50-room hold-out set.
Across seven models, the RLM harness raises mean pass rate from 23.3% to 62.4%, a gain of 39.1 percentage points. We also ran two general-purpose coding agents with web tools disabled, [Claude Code](https://claude.com/product/claude-code) with Opus-5 and [Codex](https://chatgpt.com/codex/) with GPT-5.6 Sol, using the same minimal instruction as the tool-loop harness. Claude Code passes 24.6% of criteria and Codex 12.0%, below the same models in the tool-loop harness by 17.9 and 4.6 points, respectively. In the traces, both agents stop reading early and write shorter memos, and neither spawns sub-agents despite having the ability to.
Coverage and Cost
The RLM harness substantially increases the amount of useful content that becomes agent context. We estimate coverage with probes that measure the share of dataroom content reaching any model in the harness, root or sub-agent. In the standard tool-loop harness, no run reads more than 1% of the dataroom, and most read between 0.1% and 0.5%. In the RLM harness, nearly every run reads more than 10% of the dataroom and most read close to all of it. Higher coverage is associated with higher rubric criteria pass rates across this range.
Figure 7: Rubric criteria pass rate versus probe-based coverage, one point per run, for seven models in each harness on the 50 held-out data rooms from LAB Diligence. Coverage is on a log scale.
Moving to the RLM harness raises generation cost per dataroom for six of the seven baseline models, as shown in Figure 8. Claude Opus 5 is the exception. In the tool-loop harness it spends ~$18 per data room reading on its own; as an RLM root it spends ~$7 and scores 35 points higher.
Figure 8: Mean criteria pass rate versus generation cost per data room for seven models in each harness, averaged over 50 held-out data rooms. Dotted lines connect the same model across harnesses. Costs are cache-aware estimates at list prices, log scale.
Division of Labor
To examine how much performance depends on the root agent vs. the sub-agents, we paired four root models with three Qwen sub-agent models on 30 held-out data rooms (see Figure 9). Across the configurations tested, changing the root had a larger effect. Holding the sub-agent model fixed while varying the root model, the gap between the highest- and lowest-scoring roots averaged about 38 percentage points. Holding the root fixed while varying the sub-agents, the corresponding gap between sub-agent models averaged about 8 points (Figure 9b).
Figure 9: Division of labor in the RLM harness, aggregated over four root models and three sub-agent models on 30 held-out datarooms. (a) Root share of input and output tokens, averaged over sub-agent choices. (b) The gap between the highest- and lowest-scoring roots, averaged over sub-agent choices, and the corresponding gap between sub-agents, averaged over roots.
This difference is notable because, in the RLM harness, we observe the root accounting for a minority of the agent’s total token usage. With Opus 5 orchestrating, for example, the root accounts for 3.8% of input tokens and 1.1% of output tokens (see Figure 9a). The sub-agents process most of the text, while the root decides how to divide the review and assemble the findings.
Figure 10: Mean criteria pass rate for four root models paired with three sub-agent models, on 30 held-out data rooms.
These results suggest that improving coordination is an important opportunity in this setup, so we focused our initial post-training experiments on the root.
## Post-Training in the RLM Harness
The results above point to the root agent as a target for post-training. In this section, we report results from post-training open-weight root models inside the RLM harness.
Self-Distillation SFT
First, we post-trained a GLM-5.2 root using rejection-sampling self-distillation SFT. This experiment uses a different 20-room hold-out evaluation set from the 50-room evaluations reported elsewhere. Its matched base-model score on this hold-out is 46.1%, below the 65.4% GLM-5.2 result reported in Figure 6 over the larger hold-out set.
Base GLM-5.2 does not stably execute a high-scoring policy out of the box – there are frequent performance collapses where the base model, acting as root agent, gives up early, under-delegates, or fails to convert the subagent outputs into a strong deliverable. Given that a strong policy is already within the model’s distribution but needs to be made more robust, we use rejection-sampling self-distillation to sharpen GLM-5.2’s distribution onto the desired mode. We selected successful GLM-5.2 runs with high dataroom coverage and fine-tuned the root on those trajectories.
Figure 11: GLM-5.2 as the RLM root before and after rejection-sampling self-distillation SFT, evaluated on a separate 20-room hold-out set. These are matched results within the SFT experiment, not the 50-room evaluation used in Figure 6.
On this 20-dataroom hold-out set, the SFT-trained root scores 60.1% against 46.1% for the base model (see Figure 11). A review of the post-trained agent’s traces suggests that training helps the root agent comprehensively review the dataroom and carry a larger percentage of sub-agent findings into the final memo.
Table 1 summarizes other behavioral changes we observe after post-training. Most notably, the correlation between sub-agent call volume and dataroom size rises from 0.17 to 0.84, indicating that the trained root agent scales its delegation to the size of the dataroom. The trained root also learns to begin writing the memo while sub-agents are still reading, similar to what we found in our RL runs.
Table 1: Behavior of the GLM-5.2 root before and after SFT, averaged over the separate 20-room hold-out set. Coverage is a percentage; the final row is a correlation coefficient.
The behaviors that make a strong root diligence agent, like exhaustive delegation, are learnable from a relatively small number of on-policy traces, with no privileged information and no labeled legal knowledge. Here, self-distillation is effective because good behavior already exists in the model’s distribution and training stabilizes it.
This experiment motivated us to explore RL-based post-training as an attempt to push the model beyond what is within its distribution, eliciting new behavior directly through the task’s reward.
Reinforcement learning
For reinforcement learning we started with a smaller root model, Qwen3.5-122B-A10B, to keep the initial RL experiment loop tractable.
We trained the RLM root with GRPO, using the judged rubric criteria pass rate as the reward and holding the sub-agent models fixed (sub-agents were each Qwen3.6-35B-A3B model). Over 40 training steps, we found that the mean rollout pass rate rose from about 23% to about 56%. On the 50-room hold-out set, the final checkpoint scores 63.0% against 29.9% for the base model.
Figure 12: Reinforcement learning on the Qwen3.5-122B-A10B root with fixed Qwen3.6-35B-A3B sub-agents. (a) Mean rollout criteria pass rate at each of 40 training steps. (b) Base model and final checkpoint on the 50-room hold-out set.
After RL, mean data room coverage rises from 62% to 96%, even though coverage is not an explicit reward term. Base runs are spread across the full coverage range, with a cluster below 20% that scores near zero. Every RL-trained run reads more than 60% of the data room and most read all of it.
Figure 13: Criteria pass rate versus probe-based coverage, one point per run, for the Qwen3.5-122B-A10B root before and after RL on the 50-room hold-out set.
The post-trained root makes 64% more sub-agent calls per dataroom. RL also changed delegation volume more than SFT did, 64% against 29%.
Table 2: Behavior of the Qwen3.5-122B-A10B root before and after RL, averaged over the 50-room hold-out set.
Before post-training, the Qwen root agent attempted to write its diligence memo in a single shot and only after all of the sub-agents return. During RL post-training, it learns a more efficient approach, writing the memo incrementally and interleaving section writing with the review of sub-agent work.
Scaling RL with GLM-5.3
To test our RL recipe at scale, we are now training GLM-5.3 as the RLM root with the same GRPO-style configuration: judged criteria pass rate as the reward, Qwen3.6-35B-A3B sub-agents held fixed, LoRA on the root, group size 8 and batch size 24. GLM-5.3 rollouts start well above where the Qwen root started, so there is less headroom. Over steps 0 to 20 (Figure 14), mean rollout pass rate rises from about 51% to about 59%, averaged over the first and last eight steps shown, with the step-to-step noise expected at this batch size.
Figure 14: Early training progress for the ongoing GLM-5.3 RL run, with Qwen3.6-35B-A3B sub-agents held fixed. Mean rollout criteria pass rate is shown over the course of 20 training steps. Grey shows raw per-step values; black shows an exponential moving average. This figure reports training scores, not held-out performance.
## What’s Next
The RLM harness improves performance across the models we tested, and RL produces further gains for the Qwen root within that harness. There is still a substantial gap between these scores and the near-perfect pass rates we are working toward.
The training experiments detailed above are initial explorations. In addition to completing the scaled up GLM-5.3 training run, we will also explore combinations of SFT and RL, and trained sub-agents. The RLM harness separates the root's orchestration from the sub-agents' reading, so each can be trained on its own or jointly, including setups in which sub-agents are trained to delegate in turn.
Finally, the core finding here, that model-harness co-optimization can meaningfully improve agent performance in long-horizon environments, is not specific to diligence. We are exploring ways to map the same harness structure and training approach to other long-context legal work.
## Appendix
RLM Recursion Depth
We also tested whether adding another layer of delegation improves performance. At depth-1, sub-agents return plain LLM completions, with no tools or delegation of their own. At depth-2, sub-agents receive a Python REPL and can delegate further. Within each experiment, we used the same model for the root and every sub-agent.
We evaluated both GLM-5.2 and Qwen3.5-122B-A10B. Many GLM-5.2 depth-2 runs did not finish within the time limit, so we report the Qwen results below. Across 14 datarooms, depth-2 improved scores on four and reduced them on ten, lowering mean criteria pass rate by 19 percentage points (Figure A2). In four of the ten regressions, the depth-2 agent read dataroom content but never produced a report.
Figure A1: Per-room criteria pass rate at depth-1 and depth-2 for 14 datarooms. We held the model fixed and compared plain-completion sub-agents with sub-agents that could use a REPL and delegate.. In four of the ten regressions, the depth-2 run reads the dataroom but never writes the report.
These results motivated our use of depth-1 for the initial post-training experiments. Whether training agents specifically for deeper delegation can make additional layers useful remains an open question.
RL Infrastructure
RL, especially with such long episode lengths, is as much an infrastructure problem as a training signal problem. For this task, single-episode rollouts could take on the order of an hour or more of wall-clock to complete. To optimize the training wall-clock time, we applied methods like asynchronous off-policy inference, continuous inference batching, oversampling, and in-flight weight updates. Selective token masking and async-related caps were applied to control the effect of off-policyness. Given that in the RLM harness most of the wall-clock is spent waiting for subagents to complete, the ability to use small fast subagents helped to greatly speed up the RL training process.
We plot step wall-clock time below. It is dominated by rollout time which grows over the course of RL as the agent becomes more thorough and sends out larger waves of subagents.
Figure A2: Wall-clock minutes per RL training step for the 40-step run, with a 5-step rolling mean. Steps 2–10 average 48 minutes and steps 31–40 average 74 minutes; the increase comes from the trained model dispatching more sub-agents per rollout.
## 相关链接
- [Niko](https://x.com/nikogrupen)
- [@nikogrupen](https://x.com/nikogrupen)
- [101K](https://x.com/nikogrupen/status/2097369705791307952/analytics)
- [https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence](https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence)
- [research preview for Harvey Tenet](https://www.harvey.ai/blog/post-training-update-harvey-tenet)
- [recursive language model](https://arxiv.org/abs/2512.24601)
- [LAB Diligence](https://www.harvey.ai/blog/legal-agent-bench-m-and-a-due-diligence)
- [LAB Diligence](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)
- [LAB](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)
- [Legal Agent Bench](https://github.com/harveyai/harvey-labs)
- [Claude Code](https://claude.com/product/claude-code)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:01 AM · Sep 9, 2026](https://x.com/nikogrupen/status/2097369705791307952)
- [101.3K Views](https://x.com/nikogrupen/status/2097369705791307952/analytics)
- [View quotes](https://x.com/nikogrupen/status/2097369705791307952/quotes)
---
*导出时间: 2026/9/9 09:25:10*
---
## 中文翻译
# 针对端到端并购尽职调查的后训练 RLM 智能体
**作者**: Niko
**日期**: 2026-09-08T17:01:19.000Z
**来源**: [https://x.com/nikogrupen/status/2097369705791307952](https://x.com/nikogrupen/status/2097369705791307952)
---
文章链接：[https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence](https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence)
在我们最近的 [Harvey Tenet 研究预览](https://www.harvey.ai/blog/post-training-update-harvey-tenet)中，我们强调了模型与执行框架协同优化对于解决复杂端到端法律任务的重要性，并预览了并购尽职调查（M&A Diligence）的初步成果——在这类任务中，单一任务需要遍历多达 8000 万 token 的文档上下文。今天，我们将分享持续实验的最新结果。
我们与 Baseten 合作，为并购尽职调查构建了一个[递归语言模型](https://arxiv.org/abs/2512.24601)（RLM）执行框架。在我们的 RLM 设计中，完整的数据室被加载到 Python REPL 中，根智能体将范围受限的审查与分析任务委派给在各自上下文窗口内工作的子智能体。在我们评估的七个模型上，该框架在 [LAB Diligence](https://www.harvey.ai/blog/legal-agent-bench-m-and-a-due-diligence) 任务中将评分标准通过率平均提升了 39.1 个百分点。
我们发现，在 RLM 框架内进行后训练可以带来进一步的性能提升，并为权衡质量与效率提供了更多选择。我们通过强化学习对 Qwen3.5-122B-A10B 编排器进行了后训练，发现在 50 个留出的 LAB Diligence 数据室上，评分标准通过率从 29.9% 提升至 63.0%。
图 1：在标准工具循环框架下运行的基础模型、在各自框架下运行的编程智能体，以及在我们的 RLM 框架下运行的模型在 LAB Diligence 上的平均评分标准通过率。星号标注的是 GLM-5.2 在 SFT 前后的匹配结果，在单独的 20 个数据室留出集上评估。其余结果均使用 50 个数据室的留出集。
这些结果表明，在特定任务框架内进行后训练是处理文档密集型法律工作（如并购尽职调查）的一条可行路径，而在 RLM 框架内扩展强化学习是一个有前景的未来方向。因此，我们也在进行 GLM-5.3（一个大型前沿开源权重模型）的训练，将其作为根模型用于正在进行的规模化运行。
在本文的其余部分，我们将更详细地介绍我们评估的 LAB Diligence 环境、我们的方法论以及后训练实验。
## 并购尽职调查环境
我们最近推出了 [LAB Diligence](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)，它是 [LAB](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark) 的扩展版本，包含用于并购尽职调查的合成环境。LAB Diligence 中的单个数据室包含多达 5,000 份文档，按类别组织在数十个文件夹中，总上下文多达 8000 万 token。
图 2：LAB Diligence 数据室示例。Aravon Bridge Bank 包含 82 个文件夹中 14 个类别下的 2,270 份文档，总计 3100 万 token。智能体为该任务撰写的尽调备忘录将依据 571 条评分标准进行评分。
在该数据室中，智能体的任务是撰写一份完整的尽调备忘录，包括附带文档引用的发现、相关的量化风险敞口，以及针对交易的后续建议步骤。一个 LLM 评审会依据包含数百条通过/不通过标准的专家评分表对尽调备忘录进行评分。Aravon Bridge Bank 示例总共有 571 条评分标准。
图 3：LAB Diligence 中 Aravon Bridge Bank 任务的评分标准示例，每种类型各展示一条原文标准。
尽调所需的证据分散在整个数据室中。有些发现需要综合多份文档，另一些则需要核实支持性证据是否缺失。在我们的基线运行中，智能体只进行选择性搜索和阅读，数据室的大部分内容未被审查。这类任务需要一个能够将审查分配到多个受限上下文中、并将发现汇总起来的执行框架。
## 用于并购尽职调查的 RLM 框架
为了建立基线，我们从 [Legal Agent Bench](https://github.com/harveyai/harvey-labs) 的标准工具循环框架入手。在该框架下，基础模型在 50 个留出数据室上的平均评分标准通过率为 23.3%，没有任何模型能在任何数据室上通过全部标准。
图 4：标准工具循环框架在 50 个留出尽调数据室上的平均评分标准通过率。
这些结果表明任务与框架之间存在不匹配。LAB Diligence 数据室太大，无法装进单一模型上下文，但初始审查的大部分工作可以按类别划分。根智能体随后可以整合各分类的发现以撰写备忘录。律所内部的交易团队也是以类似方式划分尽调工作的。
图 5：深度为 1 的 RLM 框架。数据室作为可查询变量加载到 Python REPL 中。根智能体通过代码对其进行操作，并将范围受限的阅读任务委派给子智能体，子智能体将其发现以 REPL 变量的形式返回。只有根智能体打印的输出会进入其上下文窗口，因此根智能体从不持有完整的数据室。
这促使我们探索将 RLM 作为并购尽职调查的框架。在 RLM 框架中，根智能体获得一个加载了数据室的 Python REPL（作为可查询变量），从而能够以编程方式检索语料库。根智能体负责规划和确定审查范围，并向子智能体分发子任务。每个子智能体接收语料库的一个受限片段以及来自根智能体的指令，在各自的上下文窗口内工作，并将发现以 REPL 变量的形式返回。除非另有说明，以下实验使用单层子智能体，且在实践中根智能体会并行分发大量调用。
图 6：七个模型在标准工具循环框架下以及作为 RLM 框架根模型（深度 1，Qwen3.6-35B-A3B 子智能体）时，在 50 个数据室留出集上的平均评分标准通过率。
在七个模型上，RLM 框架将平均通过率从 23.3% 提高到 62.4%，提升了 39.1 个百分点。我们还在禁用网络工具的情况下运行了两个通用编程智能体：搭载 Opus-5 的 [Claude Code](https://claude.com/product/claude-code) 和搭载 GPT-5.6 Sol 的 [Codex](https://chatgpt.com/codex/)，使用与工具循环框架相同的精简指令。Claude Code 通过了 24.6% 的标准，Codex 通过了 12.0%，分别比相同模型在工具循环框架下的成绩低 17.9 和 4.6 个百分点。从运行轨迹来看，两个智能体都过早停止阅读并撰写了较短的备忘录，而且尽管具备能力，都没有生成子智能体。
覆盖范围与成本
RLM 框架显著增加了进入智能体上下文的有用内容量。我们使用探针来衡量数据室内容到达框架中任一模型（根或子智能体）的比例，以此估算覆盖率。在标准工具循环框架下，没有任何一次运行阅读超过数据室的 1%，大多数运行阅读比例在 0.1% 到 0.5% 之间。在 RLM 框架下，几乎每次运行都阅读超过数据室的 10%，大多数运行接近阅读全部内容。在此范围内，更高的覆盖率与更高的评分标准通过率相关。
图 7：评分标准通过率与基于探针的覆盖率的关系，每次运行一个点，涵盖各框架中七个模型在 LAB Diligence 50 个留出数据室上的表现。覆盖率采用对数刻度。
如图 8 所示，切换到 RLM 框架后，七个基线模型中有六个的每数据室生成成本上升。Claude Opus 5 是例外。在工具循环框架下，它每个数据室花费约 18 美元自行阅读；而作为 RLM 根模型，它只花费约 7 美元，得分还高出 35 分。
图 8：各框架下七个模型的平均评分标准通过率与每数据室生成成本的关系，在 50 个留出数据室上取平均。虚线连接同一模型在不同框架下的结果。成本为按标价计算的缓存感知估算值，对数刻度。
分工
为了考察性能在多大程度上取决于根智能体还是子智能体，我们将四个根模型与三个 Qwen 子智能体模型配对，在 30 个留出数据室上进行测试（见图 9）。在所有测试配置中，更换根模型的影响更大。在固定子智能体模型、改变根模型的情况下，最高分与最低分根模型之间的差距平均约为 38 个百分点。在固定根模型、改变子智能体的情况下，子智能体模型之间的相应差距平均约为 8 分（图 9b）。
图 9：RLM 框架中的分工，汇总四个根模型与三个子智能体模型在 30 个留出数据室上的结果。(a) 根模型的输入和输出 token 占比，对子智能体选择取平均。(b) 最高分与最低分根模型之间的差距（对子智能体选择取平均），以及子智能体之间的相应差距（对根模型取平均）。
这一差异值得注意，因为在 RLM 框架中，我们观察到根模型在智能体总 token 使用量中只占少数。例如，在由 Opus 5 编排时，根模型占输入 token 的 3.8% 和输出 token 的 1.1%（见图 9a）。子智能体处理大部分文本，而根模型决定如何划分审查工作并汇总发现。
图 10：四个根模型与三个子智能体模型配对时，在 30 个留出数据室上的平均评分标准通过率。
这些结果表明，改进协调机制是这一设置中的一个重要机会，因此我们最初的后训练实验聚焦于根模型。
## RLM 框架内的后训练
上述结果指向根智能体是后训练的目标。在本节中，我们报告在 RLM 框架内对开源权重根模型进行后训练的结果。
自蒸馏 SFT
首先，我们使用拒绝采样自蒸馏 SFT 对一个 GLM-5.2 根模型进行了后训练。该实验使用了一个与其它部分报告的 50 个数据室评估不同的 20 个数据室留出评估集。其匹配的基础模型在此留出集上的得分为 46.1%，低于图 6 中报告的 GLM-5.2 在更大留出集上的 65.4%。
基础 GLM-5.2 无法开箱即用地稳定执行高分策略——经常出现性能崩溃：作为根智能体的基础模型会过早放弃、委派不足，或无法将子智能体的输出转化为高质量的交付成果。鉴于强策略已存在于模型分布之内但需要变得更加稳健，我们使用拒绝采样自蒸馏将 GLM-5.2 的分布锐化到目标模式。我们选取了具有高数据室覆盖率的成功 GLM-5.2 运行，并在这些轨迹上对根模型进行微调。
图 11：GLM-5.2 作为 RLM 根模型在拒绝采样自蒸馏 SFT 前后的对比，在单独的 20 个数据室留出集上评估。这些是 SFT 实验内的匹配结果，而非图 6 中使用的 50 个数据室评估。
在这个 20 个数据室的留出集上，SFT 训练后的根模型得分为 60.1%，而基础模型为 46.1%（见图 11）。对后训练智能体运行轨迹的审查表明，训练帮助根智能体更全面地审查数据室，并将更大比例的子智能体发现带入最终备忘录。
表 1 总结了我们在后训练后观察到的其他行为变化。最显著的是，子智能体调用次数与数据室大小之间的相关性从 0.17 上升到 0.84，表明训练后的根智能体能够根据数据室的大小调整其委派规模。训练后的根模型还学会了在子智能体仍在阅读的同时开始撰写备忘录，这与我们在 RL 运行中的发现类似。
表 1：GLM-5.2 根模型在 SFT 前后的行为对比，在单独的 20 个数据室留出集上取平均。覆盖率为百分比；最后一行为相关系数。
造就强大根尽调智能体的行为，如彻底的委派，是可以通过相对较少的在策略轨迹学习的，无需特权信息，也无需标注的法律知识。在这里，自蒸馏之所以有效，是因为良好行为已经存在于模型分布中，而训练使其变得稳定。
这一实验促使我们探索基于 RL 的后训练，试图推动模型超越其分布内的能力，通过任务的奖励直接引出新的行为。
强化学习
对于强化学习，我们从一个较小的根模型 Qwen3.5-122B-A10B 开始，以保持初始 RL 实验循环的可控性。
我们使用 GRPO 训练 RLM 根模型，以评审后的评分标准通过率作为奖励，并固定子智能体模型（子智能体均为 Qwen3.6-35B-A3B 模型）。在 40 个训练步骤中，我们发现平均 rollout 通过率从约 23% 上升到约 56%。在 50 个数据室的留出集上，最终检查点得分为 63.0%，而基础模型为 29.9%。
图 12：对 Qwen3.5-122B-A10B 根模型进行强化学习，固定 Qwen3.6-35B-A3B 子智能体。(a) 40 个训练步骤中每一步的平均 rollout 评分标准通过率。(b) 基础模型与最终检查点在 50 个数据室留出集上的表现。
经过 RL 之后，平均数据室覆盖率从 62% 上升到 96%，尽管覆盖率并不是显式的奖励项。基础模型的运行分布在全部覆盖率范围内，其中有一群低于 20%、得分接近零的运行。每一个经过 RL 训练的运行都阅读了超过 60% 的数据室，大多数阅读了全部内容。
图 13：Qwen3.5-122B-A10B 根模型在 RL 前后于 50 个数据室留出集上的评分标准通过率与基于探针的覆盖率关系，每次运行一个点。
后训练后的根模型每个数据室的子智能体调用次数增加了 64%。RL 对委派规模的改变也大于 SFT，分别为 64% 和 29%。
表 2：Qwen3.5-122B-A10B 根模型在 RL 前后的行为对比，在 50 个数据室留出集上取平均。
在后训练之前，Qwen 根智能体试图在所有子智能体返回后一次性写出尽调备忘录。在 RL 后训练过程中，它学会了一种更高效的方法：增量式撰写备忘录，并在撰写章节与审查子智能体工作之间交替进行。
使用 GLM-5.3 扩展 RL
为了大规模测试我们的 RL 方案，我们目前正在以相同的 GRPO 风格配置训练 GLM-5.3 作为 RLM 根模型：以评审后的评分标准通过率作为奖励，固定 Qwen3.6-35B-A3B 子智能体，根模型使用 LoRA，组大小为 8，批大小为 24。GLM-5.3 的 rollout 起点远高于 Qwen 根模型的起点，因此提升空间较小。在第 0 到 20 步中（图 14），平均 rollout 通过率从约 51% 上升到约 59%（对所示前八步和后八步取平均），且在此批大小下存在预期的逐步噪声。
图 14：正在进行的 GLM-5.3 RL 运行的早期训练进展，固定 Qwen3.6-35B-A3B 子智能体。展示了 20 个训练步骤过程中的平均 rollout 评分标准通过率。灰色为原始每步数值；黑色为指数移动平均。此图报告的是训练得分，而非留出集表现。
## 下一步计划
RLM 框架提升了我们测试的所有模型的性能，而 RL 在该框架内为 Qwen 根模型带来了进一步提升。但这些得分与我们努力追求的接近满分通过率之间仍有很大差距。
上述训练实验只是初步探索。除了完成扩大规模的 GLM-5.3 训练运行外，我们还将探索 SFT 与 RL 的组合，以及经过训练的子智能体。RLM 框架将根模型的编排与子智能体的阅读分离开来，因此可以单独训练或联合训练二者，包括子智能体又被训练去进一步委派的设置。
最后，本文的核心发现——模型与框架协同优化能够显著提升智能体在长时程环境中的表现——并不局限于尽调。我们正在探索将相同的框架结构和训练方法应用于其他长上下文法律工作的途径。
## 附录
RLM 递归深度
我们还测试了增加一层委派是否能提升性能。在深度 1，子智能体返回纯 LLM 补全结果，不使用工具，也不进行自己的委派。在深度 2，子智能体获得 Python REPL，可以进一步委派。在每个实验中，我们为根模型和每个子智能体使用相同的模型。
我们评估了 GLM-5.2 和 Qwen3.5-122B-A10B。许多 GLM-5.2 深度 2 的运行未能在时限内完成，因此我们在下方报告 Qwen 的结果。在 14 个数据室中，深度 2 在四个数据室上提升了得分，在十个数据室上降低了得分，平均评分标准通过率下降 19 个百分点（图 A2）。在十次退步中的四次里，深度 2 智能体阅读了数据室内容，但从未产出报告。
图 A1：14 个数据室在深度 1 和深度 2 下的逐数据室评分标准通过率。我们固定模型，对比纯补全子智能体与可使用 REPL 并委派的子智能体。在十次退步中的四次里，深度 2 运行阅读了数据室，但从未撰写报告。
这些结果促使我们在最初的后训练实验中使用深度 1。专门针对更深层次委派训练智能体是否能使额外层级发挥作用，仍是一个开放问题。
RL 基础设施
RL，尤其是在如此长的回合长度下，既是一个训练信号问题，也同样是一个基础设施问题。对于本任务，单回合 rollout 的完成可能需要一小时或更长的实际时间。为了优化训练实际耗时，我们采用了异步离策略推理、持续推理批处理、过采样和在线权重更新等方法。我们还应用了选择性 token 掩码和异步相关上限来控制离策略的影响。鉴于在 RLM 框架中大部分实际时间都花在等待子智能体完成上，能够使用小型快速的子智能体极大地加快了 RL 训练过程。
我们在下方绘制了每步实际耗时。它主要由 rollout 时间主导，随着 RL 的进行，智能体变得更加彻底并派出更大规模的子智能体，rollout 时间也随之增长。
图 A2：40 步运行中每个 RL 训练步骤的实际耗时（分钟），附 5 步滚动均值。第 2-10 步平均 48 分钟，第 31-40 步平均 74 分钟；增长的原因是训练后的模型在每次 rollout 中派出了更多子智能体。
## 相关链接
- [Niko](https://x.com/nikogrupen)
- [@nikogrupen](https://x.com/nikogrupen)
- [101K](https://x.com/nikogrupen/status/2097369705791307952/analytics)
- [https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence](https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence)
- [research preview for Harvey Tenet](https://www.harvey.ai/blog/post-training-update-harvey-tenet)
- [recursive language model](https://arxiv.org/abs/2512.24601)
- [LAB Diligence](https://www.harvey.ai/blog/legal-agent-bench-m-and-a-due-diligence)
- [LAB Diligence](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)
- [LAB](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)
- [Legal Agent Bench](https://github.com/harveyai/harvey-labs)
- [Claude Code](https://claude.com/product/claude-code)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [1:01 AM · Sep 9, 2026](https://x.com/nikogrupen/status/2097369705791307952)
- [101.3K Views](https://x.com/nikogrupen/status/2097369705791307952/analytics)
- [View quotes](https://x.com/nikogrupen/status/2097369705791307952/quotes)
---
*导出时间： 2026/9/9 09:25:10* 
