顶级AI实验室如何构建RL智能体：从RLHF到RULER的演进

Avi Chawla 2026-04-28 63.5 KB 857.5K 502 𝕏 文章列表

文章探讨了 Anthropic、OpenAI 和 DeepSeek 等顶尖 AI 实验室在 2026 年关于强化学习（RL）的趋同思路，即使用系统提示作为奖励函数。内容详细梳理了 RL 在 LLM 中的应用历程，从 RLHF 到 RULER（Karpathy 提出的系统提示学习概念）的演变。重点介绍了 DeepSeek R1 利用可验证奖励（RLVR）突破传统瓶颈的机制，通过规则编译器直接提供信号，并结合 GRPO 算法移除了复杂的 Critic 和奖励模型，实现了从 15.6% 到 77.9% 的 AIME 数学成绩飞跃。

LLM RL DeepSeek RULER GRPO RLHF AI智能体模型训练强化学习

# How top AI labs are building RL agents in 2026 (using Karpathy's system prompt learning idea) **作者**: Avi Chawla **日期**: 2026-04-28T08:05:36.000Z **来源**: [https://x.com/_avichawla/status/2049037299334472015](https://x.com/_avichawla/status/2049037299334472015) --- ![Image](/doc/1220/image/img_d3c1530df3c7.jpg) How Anthropic, OpenAI, and DeepSeek are converging on a single idea of using the system prompt as the reward function. A full breakdown of the RL evolution from RLHF to RULER, with code. Reinforcement learning, at its core, is straightforward: a system takes an action, the environment rewards it, and the agent updates its behavior to maximize that reward over time. The interaction above works in discrete steps. At each step, three things happen in order: ![Image](/doc/1220/image/img_779bbdfdd3fb.jpg) - The agent observes the current state of the environment (S). A state is a description of the situation the agent is in, enough to decide what to do next. For instance, in chess, the state is the board position, and in a dialogue model, the state is the conversation history so far. - The agent picks an action (A) based on what it sees. The action is the agent's output, the only way it can influence the environment. For instance, in chess, an action is a legal move. For an LLM, an action is the generated response. - The environment then does two things: it transitions to a new state (S'), and it emits a reward (R), a scalar number that evaluates the action. The next step begins, and the loop continues. Stringing these steps together gives a trajectory: ![Image](/doc/1220/image/img_2a1247847fdf.jpg) Reading left to right, this is the entire history of the agent's interaction with the environment. Each (S, A, R, S') quartet is one transition, and much of RL is about learning from these transitions. # Applying RL to LLMs When RL was first applied to LLMs, the environment was human preference. OpenAI's InstructGPT (2022) introduced RLHF (Reinforcement Learning from Human Feedback), where: ![Image](/doc/1220/image/img_4b231a4a19bd.jpg) - humans ranked model outputs - those rankings trained a reward model - and PPO (Proximal Policy Optimization) used that reward model to fine-tune the LLM. ChatGPT was built on this exact pipeline. But humans can't sit in the training loop rating every output in real time. If the model generates 16 responses per prompt across thousands of training steps, that's hundreds of thousands of evaluations. OpenAI solved this by splitting the process into two phases. ![Image](/doc/1220/image/img_0d4279a60472.jpg) - First, the offline phase. Here, humans ranked a relatively small set of model outputs and generated pairwise comparisons. This was the expensive human labor part, but it was a one-time cost. - Second, they trained a reward model on those rankings, which was a separate LLM that learned to predict what humans would prefer. Now you had a neural network that could score any output instantly, without waiting for a human. The reward model was a compressed approximation of human judgment, fast enough to sit inside the training loop. With the reward model in place, PPO could run the actual RL training at GPU speed. The model generated responses, the reward model scored them, and PPO updated the weights, without extensive need for humans. The cost, however, was that PPO required four full-size models in memory simultaneously. ![Image](/doc/1220/image/img_956e82b6177a.jpg) - The policy (the LLM being trained). - The reference policy (a frozen copy of the original, used to prevent training from drifting too far via a KL divergence penalty). - The reward model (the human-preference approximator discussed above to score every output). - And the critic, also called the value model (more about it below). The critic exists to answer one question: > Was this reward good or bad relative to what we'd normally expect for this prompt? We need this because a raw reward of 0.7 means nothing in isolation. For instance, on a simple factual question where most responses score 0.9, a 0.7 is below average. But on a complex open-ended question where most responses score 0.4, a 0.7 is excellent. ![Image](/doc/1220/image/img_4523176f6e22.jpg) The critic learns this baseline by observing thousands of (prompt, reward) pairs during training. PPO's actual training signal is the advantage, which is estimated as the reward minus the critic's predicted baseline. This makes the signal stable across prompts of different difficulty. But the cost involved here is that the critic is a full-size LLM itself, adding another model's worth of memory. For a 7B parameter LLM, that meant roughly 28B parameters in memory at once. # DeepSeek R1 breakthrough using verifiable rewards In January 2025, DeepSeek released R1 with a fundamentally different approach to the reward signal. Instead of training a reward model from human preferences (Phases 1 and 2 of the RLHF pipeline), they used RLVR (Reinforcement Learning with Verifiable Rewards). It's a simple, rule-based verification where the environment itself provides the signal. ![Image](/doc/1220/image/img_12d4c99ab3d6.jpg) For instance: - For math problems, the verifier checked if the model's answer matched the known solution. - For code, a compiler ran the output and returned pass or fail. Binary rewards: 1 for correct, 0 for wrong. There are no human rankings or explicit reward models required since the ground truth was available (or inferable) to be used as the reward. The RL optimizer was GRPO (Group Relative Policy Optimization), which stripped away most of PPO's infrastructure. It removed the critic model entirely. Instead of training a separate model to predict expected reward per prompt, GRPO generated multiple responses to the same prompt (typically 16) and normalized rewards within each group. ![Image](/doc/1220/image/img_2cc0e3b76925.jpg) If 4 out of 16 responses got the math problem right, those 4 received a positive advantage, and the other 12 received a negative advantage. This step cut an entire full-size model from memory. GRPO also removed the need for the learned reward model, since RLVR's verifier handled scoring directly. So the four-model PPO setup (policy + reference + critic + reward model) collapsed to just two, i.e., the policy being trained and a reference copy for KL regularization. In fact, in practice, some implementations fold the reference into the policy checkpoint, bringing it close to a single-model setup. With this setup. DeepSeek R1-Zero, trained with just GRPO and verifiable rewards (no supervised fine-tuning at all), went from 15.6% to 77.9% on AIME 2024 math problems. With majority voting, it hit 86.7%, matching OpenAI's o1. The model developed self-verification, reflection, and chain-of-thought reasoning on its own, purely from the binary correct/incorrect signal, and nobody taught it to reason step by step. The RL training loop discovered that reasoning improved the reward, so the model learned to reason. RLVR with GRPO became the dominant approach for training reasoning models through 2025. Every major lab released a reasoning variant following this recipe. # The problem GRPO itself is general-purpose. It doesn't care whether the reward comes from a math verifier, a code compiler, a human, or a Python script. ![Image](/doc/1220/image/img_bf838316ae83.jpg) It just needs a number for each response, and it normalizes within each group to produce the training signal. But a clear bottleneck here is where these reward comes from. For math and code, this is fine since the environment provides a deterministic signal. But agents that interact with real-world tools and data don't produce outputs you can string-match against a gold answer. A RAG agent retrieves context and generates a response. There's no single correct answer to compare against. A customer support agent drafts a reply. There's no compiler to run it through. A summarization agent condenses a 20-page document. There are many valid summaries, and no string-matching verifier can distinguish a good one from a mediocre one. ![Image](/doc/1220/image/img_09d6c708f919.jpg) In these cases, the environment doesn't hand you a reward signal the way a math problem does. Of course, some agent tasks do have verifiable outcomes, and for these, RLVR works just fine, even with multi-step tool use. The verifiability depends on the task's outcome, not on whether the model is acting as an agent. But for the majority of agent workflows, the outcome is subjective or multi-dimensional. Intuitively, GRPO is still the right fit here because Agents that take multiple steps, call tools, and compose responses would benefit from learning through exploration, trying different approaches, and getting reinforced for what works. So, while the RL framework is the right fit, the missing piece is the scoring function. One solution is to write custom reward functions where Python code scores each output based on hand-defined criteria. ![Image](/doc/1220/image/img_7af3f599b347.jpg) - A RAG reward function might check whether the response used the retrieved context (faithfulness), penalize content that wasn't in the context (hallucination), reward completeness, and handle cases where the context itself is ambiguous. - A tool-use reward function might score partial progress through a multi-step task, penalize unnecessary API calls, and measure whether the agent reached the correct final state. Each criterion returns a partial score, and these get summed or weighted into a final reward. This works, but it introduces its own set of problems. Writing a good reward function takes days of iteration. Researchers need to anticipate edge cases, calibrate the weights between different criteria, and test that the function actually rewards the behavior you want. A reward function that over-weights format compliance and under-weights faithfulness will train an agent that produces beautifully formatted hallucinations. Reward functions are also brittle. If you change the retrieval pipeline, add a new tool, or modify the system prompt, the reward function needs to be rewritten. Debugging is problematic too. When the agent learns bad behavior during training, the cause could be the reward function, the training hyperparameters, the data, or something else entirely. But because the reward function is custom code, you often can't tell whether the function is measuring what you think it's measuring until you've already trained a model on it and evaluated the outputs. This is the primary reason RL has been widely adopted for verifiable tasks (math, code, logic) but not for agent workflows (RAG, customer support, tool use, summarization). RLVR gave reasoning models a general-purpose, automatic reward signal where they could check the answer and return 0 or 1. No such equivalent exists for most agentic workflows. The distinction isn't about the model. The same Qwen 2.5 14B can serve both roles. The distinction is about the task. Can we verify if an Agent is producing an output that can be automatically checked? # How are AI labs approaching this? This isn't a gap that only open-source practitioners are noticing. The major AI labs have been converging on the same problem from different directions. Anthropic demonstrated that you don't need humans in the RL loop at all. Their Constitutional AI work showed that if you write down a set of principles (a "constitution"), an AI can evaluate outputs against those principles and generate preference data for RL training. ![Image](/doc/1220/image/img_d658ff03b0dd.jpg) The AI judged its own outputs against the written principles and used those judgments as the RL signal. This was a significant conceptual shift that a document of rules replaced an army of human evaluators. OpenAI has been working on something similar internally. They are developing "Universal Verifiers," a technique to extend RL beyond math and code into domains like biology, medicine, and general knowledge, where answers can't be checked with a simple string match. The details aren't public, but the direction is clear that we need general-purpose reward signals that work across any domain, not just the ones with deterministic verifiers. Karpathy has been pointing in this direction too. ![Image](/doc/1220/image/img_09e356e51379.jpg) He argued in 2025 that we're missing a major learning paradigm for LLMs, something he tentatively called "system prompt learning." The core idea is that the system prompt carries a richer signal than a scalar reward, and RL training should be finding ways to leverage that signal rather than relying solely on hand-crafted reward functions. # RULER If you want to see this in practice, RULER, built into OpenPipe's ART framework (open-source with 9k+ stars) is a general-purpose reward function that replaces all of that custom scoring code with a single function call. ![Image](/doc/1220/image/img_55de4c3962c0.jpg) It uses an LLM-as-judge to rank multiple trajectories, and it works by exploiting the same property that makes GRPO powerful, i.e., only relative rankings matter. Here's how it works step-by-step: ![Image](/doc/1220/image/img_c491a071b0f1.jpg) - For each training step, you generate N trajectories for the same scenario (typically 4 to 8). - RULER sends all N to a judge LLM (like o3, o4-mini, or even a local Qwen3 32B). - The judge reads the agent's system prompt to understand what the agent was supposed to do, then scores each trajectory from 0 to 1 relative to the others. Two properties make this work: 1) Relative scoring is easier than absolute scoring. LLMs struggle with absolute scoring because there's no shared calibration. But asking "which of these 4 responses best follows the system prompt's instructions" is a comparison task, and LLMs do those consistently well. RULER leans into this by presenting all trajectories together and asking the judge to rank them against each other. 2) GRPO normalizes within each group anyway. Whether the best trajectory scored 0.9 or 0.3 in absolute terms doesn't matter. GRPO takes the scores within a group, computes the mean and standard deviation, and normalizes. The training signal comes from the relative ordering by understanding which trajectories were above average and which were below. RULER's relative rankings map directly onto what GRPO expects. ## A rough walkthrough Before jumping into code, let's trace what happens conceptually. Say you're training a RAG agent. At each training step, GRPO generates multiple responses for the same query: ``` Scenario: "What is the refund policy?" Retrieved context: "Refunds within 30 days. Digital products non-refundable..." (Faithful) Response A: "Refunds within 30 days. Email support@example.com." (hallucinated) Response B: "Refunds within 30 days. Also store credit for 90 days." (ignored context) Response C: "Not sure, check the website." (verbose but accurate) Response D: "The policy states that refunds are available within..." ``` In a traditional setup, you'd write a reward function to score each of these: ``` def reward_function(response, context): score = 0.0 if uses_context(response, context): score += 0.4 if not has_hallucination(response, context): score += 0.3 if is_complete(response, context): score += 0.2 if is_concise(response): score += 0.1 return score ``` Each of those helper functions (uses_context, has_hallucination, is_complete, is_concise) is its own engineering project. You need to define what "uses context" means precisely, decide thresholds, handle edge cases, and test everything. With RULER, you replace all of that with: ``` scored_group = await ruler_score_group(group, "openai/o3") ``` The judge LLM reads the system prompt ("Answer using ONLY the retrieved context. Do not add information not in the context."), reads all four responses and scores them. The system prompt already defines faithfulness, hallucination, and completeness implicitly. The judge applies those criteria without implementing them in Python. ## Trajectories and Groups ART represents each agent response as a Trajectory, and it is a sequence of messages (system, user, assistant) packaged with metadata that GRPO needs for training. Multiple trajectories for the same scenario form a TrajectoryGroup. This is the unit RULER scores and GRPO trains on. ``` # A single trajectory: one complete agent interaction traj = art.Trajectory( messages_and_choices=[ {"role": "system", "content": "You are a RAG support agent..."}, {"role": "user", "content": "What is the refund policy?\n\n[Context]: ..."}, Choice(finish_reason="stop", index=0, message=ChatCompletionMessage(role="assistant", content="...")), ], reward=0.0, # RULER fills this in ) # A group: multiple trajectories for the same scenario group = art.TrajectoryGroup([traj_a, traj_b, traj_c, traj_d]) # Score the entire group relatively judged_group = await ruler_score_group(group, "openai/o3") ``` The reward=0.0 initialization is a placeholder. - After ruler_score_group returns, each trajectory's reward field is updated with the judge's score. - The Choice and ChatCompletionMessage objects are OpenAI's standard types, so if you're already using the OpenAI SDK for inference, these are the same objects you're already working with. # Two concrete examples RULER has two levels of API. 1) The lower-level `ruler` function works with plain message dictionaries, so no ART-specific objects are needed. This is the fastest way to see what RULER actually does. Let's look at a minimal example with three trajectories for the same RAG query. Below, we have defined three message lists: one is faithful, one hallucinates, and one ignores the context entirely. ``` message_lists = [ # Good: uses retrieved context accurately [ {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."}, {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."}, {"role": "assistant", "content": "You can get a refund within 30 days of purchase. However, digital products are non-refundable once downloaded. To start the process, email support@example.com."}, ], # Mediocre: partially uses context, adds hallucinated details [ {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."}, {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."}, {"role": "assistant", "content": "Our refund policy allows returns within 30 days. You can also get store credit for up to 90 days. Just call our hotline at 1-800-REFUND."}, ], # Bad: ignores context entirely [ {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."}, {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."}, {"role": "assistant", "content": "I'm not sure about the refund policy. You should check the website or contact someone from the billing team."}, ], ] ``` Next, we run the scoring: ``` from art.rewards import ruler scores = await ruler(message_lists, "openai/o3") for label, score in zip(["Faithful", "Hallucinated", "Ignored context"], scores): print(label) print("→", score.score) print("→", score.explanation) ``` This produces the following output: ``` Faithful: → 0.97 → Accurately reflects the retrieved policy details, complete and concise. Hallucinated: → 0.45 → Gives correct 30-day refund info but adds unsupported details (90-day credit, hotline), reducing accuracy. Ignored context: → 0.05 → Provides no useful information and ignores available context. ``` Notice that we never wrote a faithfulness checker or coded a hallucination detector. The system prompt mentioned "Use the retrieved context to answer user queries accurately," and the judge applied that as the evaluation criteria. The hallucinated response scored 0.45 (not zero) because it partially used the context. The 30-day refund part was correct. The judge gave partial credit for what it got right and penalized what it invented. That's a nuanced distinction that would take significant engineering to encode in a rule-based reward function. Moreover, the scores are spread across the 0-1 range: 0.97, 0.45, 0.05, unlike binary pass/fail. RULER produces a gradient that reflects relative quality. GRPO can use this gradient to apply proportional updates to strongly reinforce the faithful behavior, mildly suppress the hallucination pattern (since it was partially correct), and strongly suppress the context-ignoring behavior. 2) The `ruler` function above works for understanding and experimentation, but ART's training loop operates on Trajectory and TrajectoryGroup objects. These carry the reward field that GRPO reads, debug logs for inspection, and the structure that model.train() expects. After this, the higher-level ruler_score_group function handles the conversion. Below, let's look at the same RAG scenario structured the way you'd use it in a real training pipeline, now with 4 trajectories instead of 3. ``` # The system prompt defines the agent's goal # RULER uses this as the implicit reward function system_msg = { "role": "system", "content": ( "You are a RAG-based support agent. " "Answer user queries using ONLY the retrieved context. " "Do not add information that is not in the context." ), } user_msg = { "role": "user", "content": ( "What is the refund policy?\n\n" "[Retrieved context]: Refunds are available within 30 days " "of purchase. Digital products are non-refundable after " "download. Contact support@example.com to initiate." ), } responses = [ "You can get a refund within 30 days of purchase. Digital products " "are non-refundable once downloaded. Email support@example.com to start.", "Refunds are available within 30 days. You can also get store credit " "for up to 90 days, and our hotline is 1-800-REFUND.", "I'm not sure about the refund policy. Please check the website or " "contact the billing team for more details.", "Based on the information I have, the refund policy states that " "refunds are available within 30 days of purchase. It is important " "to note that digital products cannot be refunded after they have " "been downloaded. If you wish to initiate a refund, you should " "reach out to support@example.com.", ] ``` Now we have 4 trajectories instead of 3. The fourth is a verbose but accurate response that uses only the retrieved context but wraps it in unnecessary filler words/sentences. Moving on, we define our Trajectories and Groups as we discussed earlier: ``` import art from openai.types.chat.chat_completion import Choice from openai.types.chat import ChatCompletionMessage trajectories = [] for resp in responses: traj = art.Trajectory( messages_and_choices=[ system_msg, user_msg, Choice( finish_reason="stop", index=0, message=ChatCompletionMessage(role="assistant", content=resp), ), ], reward=0.0, ) trajectories.append(traj) group = art.TrajectoryGroup(trajectories) ``` Finally, we run the scoring: ``` from art.rewards import ruler_score_group judged_group = await ruler_score_group(group, "openai/o3", debug=True) ``` With debug=True, RULER prints the judge's raw reasoning with the actual scores. This is the raw reasoning: ``` { "scores": [ { "trajectory_id": "1", "explanation": "Accurately answers the question using only the retrieved context, concisely and completely.", "score": 0.98 }, { "trajectory_id": "2", "explanation": "Includes unsupported details about store credit and a hotline that are not in the retrieved context, so it violates the instruction to use only the context.", "score": 0.2 }, { "trajectory_id": "3", "explanation": "Does not answer the question despite the needed information being present in the retrieved context.", "score": 0.05 }, { "trajectory_id": "4", "explanation": "Accurately and completely answers the question using only the retrieved context, though slightly more verbose than necessary.", "score": 0.96 } ] } ``` And these are the scores (ranked): ``` Rank 1 | Score: 0.980 — Concise, faithful response Rank 2 | Score: 0.960 — Verbose but accurate response Rank 3 | Score: 0.200 — Hallucinated store credit and hotline Rank 4 | Score: 0.050 — Ignored the retrieved context entirely ``` If you notice closely... - The concise faithful response (0.98) scored just above the verbose accurate one (0.96). Both used only the retrieved context, both were correct, but the system prompt said "Answer using ONLY the retrieved context," and the concise version did that more directly. The judge recognized verbosity as a minor quality issue, not a correctness issue. That's a nuanced distinction that would be hard to encode in a scoring function because how do you write a rule that says "technically correct but unnecessarily wordy, penalize by 0.02"? - The hallucinated response dropped from 0.45 in the first experiment to 0.20 here. The difference is the system prompt. The first experiment said "Use the retrieved context to answer accurately." This one says "Do not add information that is not in the context." The stricter instruction produced stricter scoring. The judge adapted automatically. If you tighten your system prompt, RULER tightens its evaluation to match, without you changing any scoring code. - The context-ignoring response scored 0.05 in both experiments. When the answer is right there in the retrieved context, and the agent says "I'm not sure," there's no ambiguity regardless of how the system prompt is worded. These scored trajectories are exactly what `model.train()` expects, so let's look at that ahead. # The full training loop To actually train with these scores, you replace the hardcoded responses with real model inference. ART's `gather_trajectory_groups` handles the orchestration. Essentially, for each scenario, it generates a group of trajectories using the model's current weights, scores them with RULER, and collects the results for GRPO: ``` for step in range(num_steps): groups = await art.gather_trajectory_groups( ( art.TrajectoryGroup( rollout(model, scenario) for _ in range(4) ) for scenario in scenarios ), after_each=lambda g: ruler_score_group( g, "openai/o3"), ) await model.train(groups) # GRPO updates LoRA weights ``` In every step, the model generates 4 responses per scenario using its current weights, RULER ranks them relatively, and GRPO reinforces the high-scoring behavior while suppressing the low-scoring behavior. The agent gets better at following the system prompt's instructions with every iteration. Over multiple steps, the model learns the patterns that score well (faithfulness, conciseness, grounding in context) and unlearns the patterns that score poorly (hallucination, ignoring context, verbosity). And notice that no reward function was defined anywhere in this code. # Custom rubrics For most tasks, the system prompt provides enough signal for RULER to score effectively. But when you need more specific evaluation criteria, RULER supports custom rubrics: ``` custom_rubric = """ - Prioritize responses that are concise and clear - Penalize responses that include emojis or informal language - Reward responses that cite sources """ await ruler_score_group(group, "openai/o3", rubric=custom_rubric) ``` The rubric is natural language, not Python, so iterating on it is fast. You just change a sentence, rerun, and check the scores. Compare this to editing a reward function where a misplaced weight or a buggy condition can silently teach the agent bad behavior that you won't notice until after training. # Application to non-verifiable tasks RULER is general-purpose. It works on any task, not just freeform ones where custom rewards are painful. The practical question is when RULER adds value over simpler alternatives. For purely deterministic tasks (did the SQL query return the right rows?), a binary verifier is cheaper and gives a cleaner signal. For purely subjective tasks (was the summary good?), RULER is the only automatic option. For tasks that sit in between (did the agent find the right answer AND explain it well?), you can combine both: ``` judged_group = await ruler_score_group(group, "openai/o3") for traj in judged_group.trajectories: independent_reward = verify_correctness(traj) # binary 0/1 traj.reward += independent_reward ``` RULER preserves any rewards you assign during rollout under a separate metric, so you can layer LLM-judge scoring on top of deterministic verification without losing either signal. # Practical details Here are some practical insights I have gathered based on using RULER: → You don't need the most expensive model as the judge. Cheaper models like Qwen3 32B often work well. You can also use Claude, local models through Ollama, or any model supported by LiteLLM. The choice is a cost-quality tradeoff, not a hard requirement. → 4 to 8 trajectories per group is the recommended range. Fewer than 4 gives the judge too little to compare against. More than 8 can confuse the judge and increase the cost without proportional benefit. → When all trajectories in a group share the same system prompt and user message (which they usually do), RULER deduplicates the common prefix automatically. The judge only sees the shared context once, followed by the different responses. This cuts token usage significantly for long system prompts or multi-turn conversations. → RULER caches judge responses to disk. If you rerun the same trajectories, it won't hit the API again. This matters during debugging when you're iterating on the system prompt or rubric. The bottleneck in applying RL to agents was never the optimization algorithm. GRPO handles that well. It was always the reward signal. RLVR solved this for verifiable tasks by letting the environment score outputs directly. RULER solves it for every task (verifiable or non-verifiable) by letting an LLM judge score outputs relatively. The full implementation is in the ART repo, along with Colab notebooks that walk through the training loop end to end. Repo: https://github.com/OpenPipe/ART (don't forget to star it ⭐️) ![Image](/doc/1220/image/img_ee791ed5435c.jpg) That's a wrap! If you enjoyed this tutorial: Find me → @_avichawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs. ## 相关链接 - [Avi Chawla](https://x.com/_avichawla) - [@_avichawla](https://x.com/_avichawla) - [6.5K](https://x.com/_avichawla/status/2049037299334472015/analytics) - [OpenPipe's ART framework](https://github.com/OpenPipe/ART) - [https://github.com/OpenPipe/ART](https://github.com/OpenPipe/ART) - [@_avichawla](https://x.com/@_avichawla) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [4:05 PM · Apr 28, 2026](https://x.com/_avichawla/status/2049037299334472015) - [6,529 Views](https://x.com/_avichawla/status/2049037299334472015/analytics) - [View quotes](https://x.com/_avichawla/status/2049037299334472015/quotes) --- *导出时间: 2026/4/28 20:47:47* --- ## 中文翻译 # 2026年顶级AI实验室如何构建RL智能体（基于Karpathy的系统提示词学习思想） **作者**： Avi Chawla **日期**： 2026-04-28T08:05:36.000Z **来源**： [https://x.com/_avichawla/status/2049037299334472015](https://x.com/_avichawla/status/2049037299334472015) --- ![Image](/doc/1220/image/img_d3c1530df3c7.jpg) Anthropic、OpenAI 和 DeepSeek 如何在“将系统提示词作为奖励函数”这一单一理念上达成共识。从 RLHF 到 RULER 的 RL 演进全解析，附代码。强化学习的核心其实很简单：系统采取一个动作，环境给予奖励，智能体随之更新其行为，以随着时间的推移最大化该奖励。上述交互以离散步骤进行。在每个步骤中，依次发生三件事： ![Image](/doc/1220/image/img_779bbdfdd3fb.jpg) - 智能体观察环境的当前状态（S）。状态是对智能体所处情境的描述，足以让它决定下一步做什么。例如，在国际象棋中，状态是棋盘局面；在对话模型中，状态是迄今为止的对话历史。 - 智能体根据所见选择一个动作（A）。动作是智能体的输出，也是它影响环境的唯一方式。例如，在国际象棋中，一个动作就是一步合法走法；对于 LLM 来说，一个动作就是生成的回复。 - 然后环境做两件事：转换到新状态（S'），并发出奖励（R），即一个评估该动作的标量数值。下一步开始，循环继续。将这些步骤串联起来就得到一条轨迹： ![Image](/doc/1220/image/img_2a1247847fdf.jpg) 从左到右阅读，这就是智能体与环境交互的完整历史。每个 (S, A, R, S') 四元组是一次转移，而强化学习的很大一部分就是从这些转移中学习。 # 将 RL 应用于 LLM 当 RL 最初应用于 LLM 时，环境是人类偏好。 OpenAI 的 InstructGPT（2022年）引入了 RLHF（基于人类反馈的强化学习），其中： ![Image](/doc/1220/image/img_4b231a4a19bd.jpg) - 人类对模型输出进行排序 - 这些排序用于训练奖励模型 - 然后 PPO（近端策略优化）利用该奖励模型对 LLM 进行微调。 ChatGPT 正是建立在这条流水线之上的。但人类不可能坐在训练循环中实时对每个输出进行评分。如果模型在每个提示词下生成 16 个回复，且训练有数千步，那就是数十万次评估。 OpenAI 通过将流程分为两个阶段来解决这个问题。 ![Image](/doc/1220/image/img_0d4279a60472.jpg) - 第一，离线阶段。在这个阶段，人类对相对较少的一批模型输出进行排序，生成成对比较。这是昂贵的人力劳动部分，但它是一次性成本。 - 第二，他们基于这些排序训练了一个奖励模型，这是一个独立的 LLM，学会了预测人类会偏好什么。这样你就拥有了一个可以即时给任何输出打分的神经网络，无需等待人类。奖励模型是人类判断的压缩近似，其速度足以嵌入训练循环之中。有了奖励模型，PPO 就能以 GPU 速度运行真正的 RL 训练。模型生成回复，奖励模型为其打分，PPO 更新权重，几乎不再需要人类参与。然而代价是，PPO 需要同时在内存中容纳四个全尺寸模型。 ![Image](/doc/1220/image/img_956e82b6177a.jpg) - 策略模型（正在训练的 LLM）。 - 参考策略（原始模型的冻结副本，通过 KL 散度惩罚防止训练偏离过远）。 - 奖励模型（前面提到的用于给每个输出打分的人类偏好近似器）。 - 以及评论家模型，也称为价值模型（下文详述）。评论家模型的存在是为了回答一个问题： > 相对于我们通常对该提示词的预期，这个奖励是好还是坏？我们需要它，因为 0.7 的原始奖励单独来看毫无意义。例如，在一个简单的事实性问题上，大多数回复得分 0.9，那么 0.7 就低于平均水平。但在一个复杂的开放式问题上，如果大多数回复得分 0.4，那么 0.7 就非常出色。 ![Image](/doc/1220/image/img_4523176f6e22.jpg) 评论家模型通过在训练期间观察成千上万个（提示词，奖励）对来学习这个基线。 PPO 的实际训练信号是优势，其估算方式为奖励减去评论家模型预测的基线。这使得信号在不同难度的提示词之间保持稳定。但代价是评论家模型本身就是一个全尺寸 LLM，又增加了一个模型的内存开销。对于一个 7B 参数的 LLM，这意味着内存中要同时容纳大约 28B 参数。 # DeepSeek R1 使用可验证奖励的突破 2025 年 1 月，DeepSeek 发布了 R1，在奖励信号上采用了一种截然不同的方法。他们没有从人类偏好训练奖励模型（RLHF 流水线的第 1 和第 2 阶段），而是使用了 RLVR（基于可验证奖励的强化学习）。这是一种简单的、基于规则的验证，由环境本身提供信号。 ![Image](/doc/1220/image/img_12d4c99ab3d6.jpg) 例如： - 对于数学问题，验证器检查模型的答案是否与已知解匹配。 - 对于代码，编译器运行输出并返回通过或失败。二值奖励：正确为 1，错误为 0。由于真值可用（或可推断）并直接用作奖励，因此不需要人类排序或显式的奖励模型。 RL 优化器是 GRPO（组相对策略优化），它去掉了 PPO 的大部分基础设施。它完全移除了评论家模型。 GRPO 不再训练一个单独的模型来预测每个提示词的期望奖励，而是对同一提示词生成多个回复（通常为 16 个），并在每组内部对奖励进行归一化。 ![Image](/doc/1220/image/img_2cc0e3b76925.jpg) 如果 16 个回复中有 4 个答对了数学题，那么这 4 个获得正优势，其余 12 个获得负优势。这一步把整个完整尺寸的模型从显存中省去了。 GRPO 还省去了训练奖励模型的需要，因为 RLVR 的验证器直接处理评分。于是 PPO 的四模型架构（策略 + 参考 + 评价者 + 奖励模型）缩减为两个，即正在训练的策略模型和一个用于 KL 正则化的参考副本。实际上，在实践中，一些实现把参考模型折叠进策略检查点，使其更接近单模型设置。在这个设置下，DeepSeek R1-Zero 仅用 GRPO 和可验证奖励训练（完全没有监督微调），在 AIME 2024 数学题上的成绩从 15.6% 提升到 77.9%。采用多数投票后，达到 86.7%，与 OpenAI 的 o1 持平。该模型自主发展出了自我验证、反思和思维链推理，完全来自二元的是否正确信号，没有人教它逐步推理。 RL 训练循环发现推理能提高奖励，于是模型学会了推理。 RLVR 结合 GRPO 成为了 2025 年训练推理模型的主流方法。各大实验室都按照这个配方发布了推理变体模型。 # 问题所在 GRPO 本身是通用的。它不关心奖励来自数学验证器、代码编译器、人类还是 Python 脚本。 ![Image](/doc/1220/image/img_bf838316ae83.jpg) 它只需要每个响应对应一个数字，并在每组内进行归一化以产生训练信号。但这里一个明显的瓶颈是这些奖励的来源。对于数学和代码，这没问题，因为环境提供了确定性的信号。但与真实世界的工具和数据交互的智能体，其输出无法与标准答案进行字符串匹配。一个 RAG 智能体检索上下文并生成响应。没有单一的正确答案可以比较。一个客服智能体起草回复。没有编译器可以运行它。一个摘要智能体压缩一份 20 页的文档。有效的摘要有许多种，没有任何字符串匹配验证器能区分出好的摘要和平庸的摘要。 ![Image](/doc/1220/image/img_09d6c708f919.jpg) 在这些情况下，环境不会像数学题那样直接给你奖励信号。当然，一些智能体任务确实有可验证的结果，对于这些任务，RLVR 完全适用，即使涉及多步工具调用。可验证性取决于任务的结果，而不取决于模型是否以智能体形式行动。但对于大多数智能体工作流来说，结果是主观的或多维的。直观上，GRPO 在这里仍然是合适的选择，因为执行多步操作、调用工具并组织响应的智能体会从探索式学习中受益——尝试不同方法，并对有效的方法进行强化。所以，虽然 RL 框架是合适的选择，但缺失的部分是评分函数。一种解决方案是编写自定义奖励函数，用 Python 代码根据手工定义的标准对每个输出评分。 ![Image](/doc/1220/image/img_7af3f599b347.jpg) - RAG 奖励函数可能检查响应是否使用了检索到的上下文（忠实度），惩罚上下文中不存在的内容（幻觉），奖励完整性，并处理上下文本身模糊的情况。 - 工具调用奖励函数可能对多步任务中的部分进展评分，惩罚不必要的 API 调用，并衡量智能体是否达到了正确的最终状态。每项标准返回一个部分分数，这些分数被求和或加权成最终奖励。这可行，但也带来了一系列自身的问题。编写一个好的奖励函数需要数天的迭代。研究者需要预见到边界情况，校准不同标准之间的权重，并测试该函数确实在奖励你想要的行为。一个过度重视格式合规、低估忠实度的奖励函数，会训练出一个产生格式精美幻觉的智能体。奖励函数也很脆弱。如果你更改了检索流水线、添加了新工具或修改了系统提示词，奖励函数就需要重写。调试也有问题。当智能体在训练中学到坏行为时，原因可能是奖励函数、训练超参数、数据或其他完全不同的因素。但由于奖励函数是自定义代码，你往往无法判断该函数是否在衡量你以为它在衡量的东西，除非你已经用它训练了一个模型并评估了输出。这是 RL 被广泛用于可验证任务（数学、代码、逻辑）但未用于智能体工作流（RAG、客服、工具调用、摘要）的主要原因。 RLVR 为推理模型提供了通用的、自动化的奖励信号，它们可以检查答案并返回 0 或 1。而对于大多数智能体工作流，不存在这样的等价物。区别不在于模型。同一个 Qwen 2.5 14B 可以胜任两种角色。区别在于任务。我们能否验证一个智能体产生的输出可以被自动检查？ # AI 实验室如何应对这个问题？这不是只有开源实践者才注意到的空白。各大 AI 实验室一直在从不同方向汇聚到同一个问题上。 Anthropic 证明了你完全不需要人类参与 RL 循环。他们的宪法 AI（Constitutional AI）工作表明，如果你写下一组原则（一部“宪法”），AI 就可以依据这些原则评估输出，并为 RL 训练生成偏好数据。 ![Image](/doc/1220/image/img_d658ff03b0dd.jpg) AI 依据书面原则评判自己的输出，并将这些评判作为 RL 信号。这是一个重大的概念转变——一份规则文档取代了一支人类评估员大军。 OpenAI 一直在内部研究类似的东西。他们正在开发"通用验证器"（Universal Verifiers），这是一种将 RL 从数学和代码扩展到生物学、医学和通用知识等领域的 technique，在这些领域中，答案无法通过简单的字符串匹配来验证。细节尚未公开，但方向很明确：我们需要能跨任何领域工作的通用奖励信号，而不仅仅是那些具有确定性验证器的领域。 Karpathy 也一直指向这个方向。 ![Image](/doc/1220/image/img_09e356e51379.jpg) 他在 2025 年提出，我们缺少一个针对 LLM 的主要学习范式，他暂时称之为"系统提示学习"（system prompt learning）。其核心思想是：系统提示所承载的信号比标量奖励更丰富，RL 训练应该寻找利用该信号的方法，而不是仅仅依赖手工编写的奖励函数。 # RULER 如果你想看这在实践中的样子，RULER 内置于 OpenPipe 的 ART 框架中（开源，9k+ star），它是一个通用奖励函数，只需一次函数调用就能取代所有那些自定义评分代码。 ![Image](/doc/1220/image/img_55de4c3962c0.jpg) 它使用 LLM-as-judge 来对多条轨迹进行排序，其工作原理是利用 GRPO 强大的同一特性，即只有相对排序才重要。以下是它的工作步骤： ![Image](/doc/1220/image/img_c491a071b0f1.jpg) - 对于每个训练步骤，为同一场景生成 N 条轨迹（通常为 4 到 8 条）。 - RULER 将这 N 条轨迹全部发送给一个评判 LLM（如 o3、o4-mini，甚至是本地部署的 Qwen3 32B）。 - 评判 LLM 阅读智能体的系统提示以理解智能体应该做什么，然后将每条轨迹相对于其他轨迹从 0 到 1 进行评分。有两个特性使这一方法奏效： 1) 相对评分比绝对评分更容易。 LLM 在绝对评分上表现不佳，因为没有共同的校准标准。但询问"这 4 个回答中哪个最符合系统提示的指令"是一个比较任务，LLM 在这方面始终表现出色。 RULER 正是利用了这一点，将所有轨迹放在一起呈现，并要求评判 LLM 对它们进行相互比较排序。 2) GRPO 无论如何都会在每个组内进行归一化。最佳轨迹的绝对得分是 0.9 还是 0.3 并不重要。 GRPO 会取组内的分数，计算均值和标准差，并进行归一化。训练信号来自相对排序，即理解哪些轨迹高于平均水平、哪些低于平均水平。RULER 的相对排序直接对应 GRPO 所期望的内容。 ## 粗略走查在深入代码之前，我们先从概念上梳理一下会发生什么。假设你在训练一个 RAG 智能体。在每个训练步骤中，GRPO 会为同一个查询生成多个响应： ``` Scenario: "What is the refund policy?" Retrieved context: "Refunds within 30 days. Digital products non-refundable..." (Faithful) Response A: "Refunds within 30 days. Email support@example.com." (hallucinated) Response B: "Refunds within 30 days. Also store credit for 90 days." (ignored context) Response C: "Not sure, check the website." (verbose but accurate) Response D: "The policy states that refunds are available within..." ``` 在传统设置中，你会编写一个奖励函数来为每个响应评分： ``` def reward_function(response, context): score = 0.0 if uses_context(response, context): score += 0.4 if not has_hallucination(response, context): score += 0.3 if is_complete(response, context): score += 0.2 if is_concise(response): score += 0.1 return score ``` 这些辅助函数中的每一个（uses_context、has_hallucination、is_complete、is_concise）都是各自的工程项目。你需要精确定义"使用了上下文"是什么意思、决定阈值、处理边界情况，并对所有内容进行测试。使用 RULER，你可以用以下代码取代所有这些： ``` scored_group = await ruler_score_group(group, "openai/o3") ``` 评判 LLM 阅读系统提示（"仅使用检索到的上下文回答。不要添加上下文中没有的信息。"），阅读所有四个响应并进行评分。系统提示已经隐式地定义了忠实性、幻觉和完整性。评判 LLM 直接应用这些标准，而无需用 Python 实现它们。 ## 轨迹与组 ART 将每个智能体响应表示为一个 Trajectory（轨迹），它是一个消息序列（system、user、assistant），并打包了 GRPO 训练所需的元数据。同一场景的多条轨迹组成一个 TrajectoryGroup（轨迹组）。这就是 RULER 评分和 GRPO 训练的单元。 ``` # A single trajectory: one complete agent interaction traj = art.Trajectory( messages_and_choices=[ {"role": "system", "content": "You are a RAG support agent..."}, {"role": "user", "content": "What is the refund policy?\n\n[Context]: ..."}, Choice(finish_reason="stop", index=0, message=ChatCompletionMessage(role="assistant", content="...")), ], reward=0.0, # RULER fills this in ) # A group: multiple trajectories for the same scenario group = art.TrajectoryGroup([traj_a, traj_b, traj_c, traj_d]) # Score the entire group relatively judged_group = await ruler_score_group(group, "openai/o3") ``` reward=0.0 的初始化只是一个占位符。 - 在 ruler_score_group 返回后，每条轨迹的 reward 字段都会更新为评判 LLM 的评分。 - Choice 和 ChatCompletionMessage 对象是 OpenAI 的标准类型，所以如果你已经在使用 OpenAI SDK 进行推理，这些就是你已经在使用的对象。 # 两个具体示例 RULER 有两个层级的 API。 1) 较底层的 `ruler` 函数可以直接使用普通的消息字典，因此不需要任何 ART 特有的对象。这是了解 RULER 实际做什么的最快方式。让我们来看一个最小的示例：针对同一个 RAG 查询的三条轨迹。下面，我们定义了三个消息列表：一个是忠实的，一个产生幻觉，一个完全忽略上下文。 ``` message_lists = [ # 好：准确使用检索到的上下文 [ {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."}, {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."}, {"role": "assistant", "content": "You can get a refund within 30 days of purchase. However, digital products are non-refundable once downloaded. To start the process, email support@example.com."}, ], # 一般：部分使用上下文，添加了幻觉细节 [ {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."}, {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."}, {"role": "assistant", "content": "Our refund policy allows returns within 30 days. You can also get store credit for up to 90 days. Just call our hotline at 1-800-REFUND."}, ], # 差：完全忽略上下文 [ {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."}, {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."}, {"role": "assistant", "content": "I'm not sure about the refund policy. You should check the website or contact someone from the billing team."}, ], ] ``` 接下来，我们运行评分： ``` from art.rewards import ruler scores = await ruler(message_lists, "openai/o3") for label, score in zip(["Faithful", "Hallucinated", "Ignored context"], scores): print(label) print("→", score.score) print("→", score.explanation) ``` 这会产生如下输出： ``` Faithful: → 0.97 → Accurately reflects the retrieved policy details, complete and concise. Hallucinated: → 0.45 → Gives correct 30-day refund info but adds unsupported details (90-day credit, hotline), reducing accuracy. Ignored context: → 0.05 → Provides no useful information and ignores available context. ``` 注意，我们从未编写忠实度检查器，也没有编写幻觉检测器的代码。系统提示词中提到了"Use the retrieved context to answer user queries accurately"（使用检索到的上下文准确回答用户查询），评判模型将其作为评估标准。产生幻觉的回答得分为 0.45（不是零），因为它部分使用了上下文。30 天退款的部分是正确的。评判模型对答对的部分给予部分分数，对编造的部分进行扣分。这是一个细微的区分，如果要用基于规则的奖励函数来编码，需要大量的工程工作。此外，分数分布在 0-1 的范围内：0.97、0.45、0.05，而不是二元的通过/不通过。 RULER 产生的梯度反映了相对质量。GRPO 可以利用这个梯度进行比例化的更新：强烈强化忠实的行为，轻度抑制幻觉模式（因为它部分正确），并强烈抑制忽略上下文的行为。 2) 上面的 `ruler` 函数适用于理解和实验，但 ART 的训练循环是基于 Trajectory 和 TrajectoryGroup 对象运行的。这些对象承载了 GRPO 读取的 reward 字段、用于检查的调试日志，以及 model.train() 所期望的结构。之后，更高层的 ruler_score_group 函数会处理这种转换。下面，我们来看看同一个 RAG 场景如何以真实训练流水线中的方式来构建，现在有 4 条轨迹而不是 3 条。 ``` # The system prompt defines the agent's goal # RULER uses this as the implicit reward function system_msg = { "role": "system", "content": ( "You are a RAG-based support agent. " "Answer user queries using ONLY the retrieved context. " "Do not add information that is not in the context." ), } user_msg = { "role": "user", "content": ( "What is the refund policy?\n\n" "[Retrieved context]: Refunds are available within 30 days " "of purchase. Digital products are non-refundable after " "download. Contact support@example.com to initiate." ), } responses = [ "You can get a refund within 30 days of purchase. Digital products " "are non-refundable once downloaded. Email support@example.com to start.", "Refunds are available within 30 days. You can also get store credit " "for up to 90 days, and our hotline is 1-800-REFUND.", "I'm not sure about the refund policy. Please check the website or " "contact the billing team for more details.", "Based on the information I have, the refund policy states that " "refunds are available within 30 days of purchase. It is important " "to note that digital products cannot be refunded after they have " "been downloaded. If you wish to initiate a refund, you should " "reach out to support@example.com.", ] ``` 现在我们有 4 条轨迹而不是 3 条。第四条是一个冗长但准确的回答，它只使用了检索到的上下文，却用不必要的填充词/句子进行了包装。继续，我们按照之前讨论的那样定义 Trajectories 和 Groups： ``` import art from openai.types.chat.chat_completion import Choice from openai.types.chat import ChatCompletionMessage trajectories = [] for resp in responses: traj = art.Trajectory( messages_and_choices=[ system_msg, user_msg, Choice( finish_reason="stop", index=0, message=ChatCompletionMessage(role="assistant", content=resp), ), ], reward=0.0, ) trajectories.append(traj) group = art.TrajectoryGroup(trajectories) ``` 最后，我们运行评分： ``` from art.rewards import ruler_score_group judged_group = await ruler_score_group(group, "openai/o3", debug=True) ``` 设置 debug=True 后，RULER 会打印裁判的原始推理过程以及实际分数。以下是原始推理内容： ``` { "scores": [ { "trajectory_id": "1", "explanation": "Accurately answers the question using only the retrieved context, concisely and completely.", "score": 0.98 }, { "trajectory_id": "2", "explanation": "Includes unsupported details about store credit and a hotline that are not in the retrieved context, so it violates the instruction to use only the context.", "score": 0.2 }, { "trajectory_id": "3", "explanation": "Does not answer the question despite the needed information being present in the retrieved context.", "score": 0.05 }, { "trajectory_id": "4", "explanation": "Accurately and completely answers the question using only the retrieved context, though slightly more verbose than necessary.", "score": 0.96 } ] } ``` 以下是分数（按排名排序）： ``` Rank 1 | Score: 0.980 — Concise, faithful response Rank 2 | Score: 0.960 — Verbose but accurate response Rank 3 | Score: 0.200 — Hallucinated store credit and hotline Rank 4 | Score: 0.050 — Ignored the retrieved context entirely ``` 如果你仔细观察的话…… - 简洁忠实的回答（0.98）比冗长但准确的回答（0.96）得分略高。两者都只使用了检索到的上下文，两者都是正确的，但系统提示词说“仅使用检索到的上下文来回答”，而简洁版本更直接地做到了这一点。裁判将冗长视为一个轻微的质量问题，而不是正确性问题。这是一种微妙的区分，很难用评分函数来编码——你怎么能写一条规则说“技术上正确但不必要的啰嗦，扣 0.02 分”呢？ - 幻觉回答从第一个实验的 0.45 降到了这里的 0.20。区别在于系统提示词。第一个实验说“使用检索到的上下文进行准确回答。”而这个说“不要添加上下文中不存在的信息。”更严格的指令产生了更严格的评分。裁判自动进行了调整。如果你收紧系统提示词，RULER 会相应收紧它的评估，而无需你更改任何评分代码。 - 忽略上下文的回答在两个实验中都得分为 0.05。当答案就在检索到的上下文中，而智能体却说“我不确定”时，无论系统提示词如何措辞，都没有任何模糊空间。这些已评分的轨迹正是 `model.train()` 所期望的输入，所以接下来让我们看看这一步。 # 完整的训练循环要用这些分数进行实际训练，你需要用真实的模型推理替换硬编码的响应。 ART 的 `gather_trajectory_groups` 负责处理编排工作。本质上，对于每个场景，它都会使用模型当前的权重生成一组轨迹，用 RULER 对其评分，并收集结果用于 GRPO： ``` for step in range(num_steps): groups = await art.gather_trajectory_groups( ( art.TrajectoryGroup( rollout(model, scenario) for _ in range(4) ) for scenario in scenarios ), after_each=lambda g: ruler_score_group( g, "openai/o3"), ) await model.train(groups) # GRPO updates LoRA weights ``` 在每一步中，模型使用其当前权重为每个场景生成 4 个响应，RULER 对它们进行相对排序，GRPO 则强化高分行为并抑制低分行为。智能体在每一次迭代中都会变得更擅长遵循系统提示词的指令。经过多个步骤，模型学会了得分高的模式（忠实性、简洁性、扎根于上下文），并抛弃了得分低的模式（幻觉、忽略上下文、冗长）。请注意，这段代码中没有任何地方定义了奖励函数。 # 自定义评分标准对于大多数任务，系统提示词足以为 RULER 提供有效评分所需的信号。但当你需要更具体的评估标准时，RULER 支持自定义评分标准： ``` custom_rubric = """ - Prioritize responses that are concise and clear - Penalize responses that include emojis or informal language - Reward responses that cite sources """ await ruler_score_group(group, "openai/o3", rubric=custom_rubric) ``` 评分标准是自然语言而非 Python，因此对其进行迭代非常快速。你只需修改一句话，重新运行，然后检查分数。相比之下，编辑奖励函数时，一个放错位置的权重或一个有 bug 的条件可能会悄悄地教会智能体不良行为，而你要到训练结束后才会发现。 # 应用于不可验证任务 RULER 是通用的。它适用于任何任务，不仅仅是那些自定义奖励令人头疼的自由格式任务。实际问题在于 RULER 相比更简单的替代方案何时能带来价值。对于纯确定性任务（SQL 查询是否返回了正确的行？），二元验证器更便宜，并且能提供更清晰的信号。对于纯主观任务（摘要写得好吗？），RULER 是唯一的自动选项。对于介于两者之间的任务（智能体是否找到了正确答案并且解释得很好？），你可以将两者结合： ``` judged_group = await ruler_score_group(group, "openai/o3") for traj in judged_group.trajectories: independent_reward = verify_correctness(traj) # binary 0/1 traj.reward += independent_reward ``` RULER 会将你在 rollout 期间分配的任何奖励保留在单独的指标下，因此你可以在确定性验证之上叠加 LLM 评判打分，而不会丢失任何一种信号。 # 实践细节以下是我基于使用 RULER 总结的一些实用经验： → 你不需要用最昂贵的模型来当评判。像 Qwen3 32B 这样更便宜的模型通常效果就很好。你也可以使用 Claude、通过 Ollama 运行的本地模型，或 LiteLLM 支持的任何模型。这是一个成本与质量的权衡，而不是硬性要求。 → 每组 4 到 8 条轨迹是推荐的范围。少于 4 条会让评判对象太少的比较内容。超过 8 条可能会让评判混乱，并增加成本却没有相应的收益。 → 当一个组中的所有轨迹共享相同的系统提示词和用户消息时（通常如此），RULER 会自动对公共前缀去重。评判只需看到一次共享上下文，后面跟着不同的回复。对于较长的系统提示词或多轮对话，这能显著减少 token 的使用量。 → RULER 会将评判的响应缓存到磁盘。如果你重新运行相同的轨迹，它不会再次调用 API。这在调试期间很有用，尤其是当你反复迭代系统提示词或评分标准时。将 RL 应用于智能体的瓶颈从来不是优化算法。 GRPO 已经能很好地处理这一点。瓶颈一直是奖励信号。 RLVR 通过让环境直接给输出打分，为可验证任务解决了这个问题。 RULER 通过让 LLM 评判对输出进行相对打分，为所有任务（无论可验证与否）解决了这个问题。完整实现位于 ART 仓库中，还有 Colab 笔记本可以带你完整走一遍训练循环。仓库：https://github.com/OpenPipe/ART （别忘了给它点个星 ⭐️） ![Image](/doc/1220/image/img_ee791ed5435c.jpg) 本次分享到此结束！如果你喜欢这个教程：找到我 → @_avichawla 我每天分享关于数据科学、机器学习、LLM 和 RAG 的教程与见解。 ## 相关链接 - [Avi Chawla](https://x.com/_avichawla) - [@_avichawla](https://x.com/_avichawla) - [6.5K](https://x.com/_avichawla/status/2049037299334472015/analytics) - [OpenPipe 的 ART 框架](https://github.com/OpenPipe/ART) - [https://github.com/OpenPipe/ART](https://github.com/OpenPipe/ART) - [@_avichawla](https://x.com/@_avichawla) - [升级到 Premium](https://x.com/i/premium_sign_up) - [2026年4月28日下午4:05](https://x.com/_avichawla/status/2049037299334472015) - [6,529 次浏览](https://x.com/_avichawla/status/2049037299334472015/analytics) - [查看引用](https://x.com/_avichawla/status/2049037299334472015/quotes) --- *导出时间: 2026/4/28 20:47:47*