# How to Build an RL Environment
**作者**: Akshay
**日期**: 2026-07-06T18:35:27.000Z
**来源**: [https://x.com/akshay_pachaar/status/2074200571834515574](https://x.com/akshay_pachaar/status/2074200571834515574)
---
Andrej Karpathy summarized the entire history of LLM training in three nouns:
- text
- conversations
- and environments
Pretraining ran on internet text, supervised fine-tuning ran on curated conversations, and the current era of reinforcement learning runs on environments.
OpenAI's o1 proved that framing by training on math and coding problems with verifiable answers, and DeepSeek-R1 published the recipe openly.
The industry now treats environments as the scarce resource. Anthropic reportedly discussed spending over $1 billion on them in a single year.
Meanwhile, someone is giving them away for free through a Hugging Face-style hub hosting 2,500+ open-source environments.
And today, we'll use their framework to build one of our own from the ground up.
Before we build anything, let's understand what an environment actually is. Every RL setup is a loop with four moving parts, and they map onto anything the model interacts with:
- State: what the model sees right now
- Action: the choice it makes from that state
- Reward: a number that says how good that choice was
- Environment: the thing that holds the state, accepts the action, and returns the reward
RL loop diagram, state → action → reward → environment wired in a circle
The hard part of this loop was always the reward, which traditionally required training a separate model on human preferences.
DeepSeek-R1 replaced that entire model with a plain Python function through GRPO (Group Relative Policy Optimization). Score several answers to the same prompt and push the model toward the ones that beat the group average.
Here is an illustration of how GRPO works:
Build Reasoning LLMs using GRPO
That leaves the environment as the barrier still standing. A reward function can score a final state, but something has to present the task, accept the model's actions, enforce the rules, and produce that final state in the first place.
Frontier labs guard this part as a proprietary asset, which is why few practitioners have ever seen a working environment from the inside.
That team giving environments away is Prime Intellect, and their library, Verifiers, is the framework we'll build with. (100% open-source)
The design is model-agnostic, the rewards are fully verifiable, and the same skeleton works for any turn-based task you want to train on.
# How to read this article
The goal here is simple. We're building a complete RL environment, and we're covering it in a way that stays easy to digest even if this is your first one.
Every idea comes with the code that implements it, kept short enough to read in the flow of the article. The complete, runnable version is shared at the end.
And the game we'll build around is just the working example. By the time you finish, you'll be able to take this exact structure and adapt it to your own use case.
# The Othello RL env: Why a board game
Othello is a two-player game on an 8x8 grid. You place a disc so it traps opponent discs in a straight line, every trapped disc flips to your color, and whoever owns more discs when the board fills up wins.
That flipping rule is why a single move can swing the score hard, and why corners matter so much (a corner disc can never be flipped back).
An Othello board
Every RL part has a home here. The board is the state, the disc placement is the action, the final result feeds the reward, and the game engine that validates moves, flips discs, and plays the other side is the environment.
The LLM plays Black and a built-in engine plays White. After each Black move, White responds, the updated board goes back to the model, and this repeats until the game ends.
RL loop diagram showing LLM, Environment, Opponent engine, and Reward components
# Tech stack
Three tools make this work, each handling one layer:
- Verifiers: the RL framework. It defines the environment, runs the turn loop, and handles evals.
- Lightning AI: an OpenAI-compatible inference API, so the same code calls hosted models like Claude or DeepSeek without provider-specific rewrites.
- vLLM: serves open-weight models locally behind that same OpenAI-compatible endpoint.
The shared interface is what makes the environment model-agnostic. A local Ministral-3B and a hosted GPT-4.1 swap in one line and you change the model name, nothing else.
# The game loop
Here's what the model sees on each turn:
State the model see at each turn
The board, the score, and the list of valid moves are the entire state. Everything the model knows about the game comes from this text.
It must respond with a <think> section followed by a <move> tag, which forces it to reason through the position before committing to a move.
5-step turn flow from starting position through LLM response, validation, White engine move, and updated board
The environment validates the move against the valid list, applies it to the board, lets White respond, and sends the updated board back. An invalid move gets an error message and a retry, with a penalty applied to the reward.
All of that lives in one method. OthelloEnv inherits from MultiTurnEnv in Verifiers, which handles the loop, turn tracking, and termination, and the base class calls your env_response every time the model sends a move:
```
class OthelloEnv(MultiTurnEnv):
def env_response(self, model_output, state):
move = parse_move(model_output)
if not is_valid(move, state.board):
state.penalty += INVALID_MOVE_PENALTY
return error_message(move), state # same turn, retry
state.board = apply_move(state.board, move, player="black")
if not game_over(state.board):
white_move = opponent_engine(state.board, state.difficulty)
state.board = apply_move(state.board, white_move, player="white")
return render_board(state.board), state
```
This is the shape of the logic; the real version also handles board parsing and edge cases the snippet skips. I'll share the full working code later once we understand the setup end-to-end.
# The built-in opponent engine
The builtin engine plays white and has two modes:
Random vs MiniMax opponent modes
- Random picks any legal move. Useful for early training, since the model can win just by making reasonable decisions.
- Minimax simulates future moves, scores each resulting position by corner control, board position, and available moves, and picks the move with the best worst-case outcome. Depth-3 means it looks three moves ahead, enough to set traps and avoid obvious blunders.
A randomness parameter controls how often White ignores its strategy and picks randomly instead. Lower randomness means more consistent, punishing play.
White's moves are generated from the current board state and a fixed game seed, so the same position always produces the same response. That determinism is what lets you compare different models under identical conditions.
Here's how the code for the same looks like:
```
def opponent_engine(board, randomness, depth):
if random.random() < randomness: # random mode
return random.choice(legal_moves(board))
best_move = None # minimax mode: try every move,
best_score = float("-inf") # keep the one with the best score
for move in legal_moves(board):
score = minimax(apply_move(board, move), depth - 1, my_turn=False)
if score > best_score:
best_move, best_score = move, score
return best_move
```
# The reward functions
At game end, four signals combine into a single reward:
Reward components table with what each measures and its weight
Each one is a function that reads something off the final state and returns a number, and the win signal is the simplest:
```
def win_reward_func(state):
result = state.get("result")
if result == "black": # the model's color
return 1.0
if result == "draw":
return 0.5
return 0.0 # loss, or game never finished
```
The other three follow the same shape, and all four combine into one score:
```
def total_reward(state):
return (
win_loss_score(state)
+ piece_advantage(state)
+ format_compliance(state)
- invalid_move_penalty(state)
)
```
The reason for four signals instead of a single win/loss bit is resolution. Early in training most games are losses, and to a pure win/loss reward they all look identical, so the model has nothing to climb.
- Piece advantage separates a close loss from a blowout, giving the model gradient before it starts winning.
- Format compliance carries a low weight so clean formatting never outweighs good play.
- The invalid move penalty is capped so one broken game can't drown out everything the model did right.
Every score comes directly from the game state and rules, with no judge model or LLM evaluator involved, so the reward is fully deterministic and reproducible.
# Wiring it together
Full Othello system diagram with game loop, reward functions, and prime eval
The environment generates games across different starting positions and opponent difficulties, the model plays each one through the loop above, and at game end the four rewards are computed and combined. During evaluation, rewards, token usage, and turn counts are aggregated across all games into a results table.
One function wires everything up, and it's what the prime eval command calls behind the scenes:
```
def load_environment(min_random_move_prob, max_random_move_prob, parse_think):
dataset = generate_games(min_random_move_prob, max_random_move_prob)
parser = XMLParser(fields=["think", "move"])
rubric = Rubric(funcs=[...], weights=[1.0, 1.0, 0.2, 1.0])
return OthelloEnv(dataset, parser, rubric)
```
# What the numbers show
Running an evaluation is one command, with the opponent settings passed as arguments:
```
prime eval run othello -m openai/gpt-4.1 -n 100 \
-a '{"min_random_move_prob": 0.0, "max_random_move_prob": 0.0, "minimax_depth": 3}'
```
Swap the model name to test something else, whether it's hosted through Lightning AI or running on a local vLLM server. Here's how two models fared against two opponents:
Evaluation results tables for gpt-4.1 and Ministral-3B against random and depth-3 minimax opponents
# From evaluation to training
Evaluation tells you where the model struggles, and training fixes it in three stages. The same environment supports all of them:
Three-stage pipeline of data generation, SFT, and RL training sharing the same environment and rewards
Data generation: have your strongest model play a few hundred games and save the results. The same eval command writes straight to a dataset:
```
prime eval run othello -m openai/gpt-4.1 -n 200 \
--save-to-hf-hub --hf-hub-dataset-name your-username/othello-data
```
Filter it down to wins and draws before training on it, so you're not teaching the model your strongest player's mistakes alongside its format.
Supervised fine-tuning: teach format and valid moves first. The Ministral-3B turn counts make the point directly, since unreliable formatting and illegal moves are noise RL can't train through.
RL training: this is where strategy improves. Multiple games are played from the same starting position, each is scored with the same reward functions from evaluation, and the model is updated toward the higher-scoring rollouts.
Stripped to its core, that's the GRPO loop from the top of the article:
```
for prompt in batch:
rollouts = [play_game(model, prompt) for _ in range(group_size)]
rewards = [total_reward(r.final_state) for r in rollouts]
advantage = rewards - mean(rewards) # relative to the group
update_model(model, rollouts, advantage)
```
Each rollout is scored against the average of its own group, so a move that wins 6 of 10 games from a given position is rewarded more than the weaker attempts beside it.
# Adapting this to your own task
Once you remove the Othello specifics and the same MultiTurnEnv gives you a skeleton that fits any turn-based task:
```
class TaskEnv(MultiTurnEnv):
def env_response(self, model_output, state):
action = parse_action(model_output) # your task's syntax
if not is_valid(action, state):
return error_message(action), penalize(state)
state = apply_action(state, action) # your task's rules
if not task_complete(state):
state = environment_step(state) # tool, API call, or opponent
return render_state(state), state # your task's display
```
This isn't specific to games. A coding agent swaps apply_action for running a test suite against generated code, a support agent swaps it for checking whether a tool call retrieved the right record, and a research task swaps it for verifying a claim against a cited source.
Adapting it comes down to four swaps:
- Task logic: your domain's rules for what counts as a valid action and how the state changes
- Response engine: whatever the model reacts to, from a rule-based simulator to a live API to another model
- Reward functions: keep the pattern (outcome signal, partial credit, format, penalty) and replace the domain logic
- State rendering: whatever the model needs to see, whether that's a file diff, a conversation transcript, or a tool's response
The structure underneath stays the same regardless of domain: parse, validate, apply, respond, score. The rubric is the design, and if you get the components right, the training signal takes care of itself.
# Try it yourself
All the code, setup instructions, and ready-to-use GPUs to reproduce these results are in the Lightning AI Studio template:
Build a Custom RL Environment →
Check out Lightning AI Inference →
Lightning AI Inference dashboard listing models with cost, latency, throughput, and context length
Thanks for reading!
Cheers! :)
## 相关链接
- [Akshay](https://x.com/akshay_pachaar)
- [@akshay_pachaar](https://x.com/akshay_pachaar)
- [8.2K](https://x.com/akshay_pachaar/status/2074200571834515574/analytics)
- [Verifiers](https://github.com/PrimeIntellect-ai/verifiers)
- [Build a Custom RL Environment →](https://lightning.ai/lightning-ai/templates/build-a-custom-rl-environment?utm_campaign=akshay&utm_medium=twitter)
- [Check out Lightning AI Inference →](https://lightning.ai/lightning-ai/models?utm_campaign=akshay&utm_medium=twitter)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [2:35 AM · Jul 7, 2026](https://x.com/akshay_pachaar/status/2074200571834515574)
- [8,263 Views](https://x.com/akshay_pachaar/status/2074200571834515574/analytics)
---
*导出时间: 2026/7/7 08:20:17*
---
## 中文翻译
# 如何构建强化学习环境
**作者**: Akshay
**日期**: 2026-07-06T18:35:27.000Z
**来源**: [https://x.com/akshay_pachaar/status/2074200571834515574](https://x.com/akshay_pachaar/status/2074200571834515574)
---
Andrej Karpathy 用三个名词总结了整个 LLM 训练的历史：
- 文本
- 对话
- 环境
预训练基于互联网文本运行，监督微调基于精选的对话运行，而当前的强化学习时代则基于环境运行。
OpenAI 的 o1 通过在具有可验证答案的数学和编程问题上进行训练，证明了这一框架，而 DeepSeek-R1 则公开了其配方。
业界现在将环境视为稀缺资源。据报道，Anthropic 讨论了在一年内花费超过 10 亿美元用于构建环境。
与此同时，有人通过一个托管 2,500 多个开源环境的 Hugging Face 风格中心，免费提供这些环境。
今天，我们将使用他们的框架从零开始构建我们自己的环境。
在构建任何东西之前，让我们先理解环境到底是什么。每个 RL 设置都是一个包含四个组件的循环，它们对应于模型交互的任何事物：
- 状态：模型现在看到的内容
- 动作：它根据该状态做出的选择
- 奖励：一个数字，表示该选择的好坏程度
- 环境：持有状态、接受动作并返回奖励的事物
RL 循环示意图，状态 → 动作 → 奖励 → 环境 连成一个回路
这个循环中最困难的部分一直是奖励，传统上这需要在人类偏好上训练一个单独的模型。
DeepSeek-R1 通过 GRPO（群体相对策略优化）用一个简单的 Python 函数取代了整个模型。对同一个提示词生成的多个答案进行评分，并推动模型朝向击败群体平均水平的答案发展。
以下是 GRPO 的工作原理图：
使用 GRPO 构建推理 LLM
这样一来，环境仍然是阻碍。奖励函数可以对最终状态进行评分，但必须有什么东西来呈现任务、接受模型的动作、执行规则，并首先产生那个最终状态。
前沿实验室将这部分视为专有资产严加看护，这也是为什么很少有从业者能从内部看到真正运行中的环境。
那个免费提供环境的团队是 Prime Intellect，他们的库 Verifiers 是我们将要用来构建的框架。（100% 开源）
该设计与模型无关，奖励完全可验证，并且同一个骨架适用于你想训练的任何回合制任务。
# 如何阅读本文
这里的目标很简单。我们要构建一个完整的 RL 环境，而且我们的讲述方式即使是对你是第一次接触也能轻松理解。
每个想法都配有实现它的代码，篇幅短小，适合在文章流程中阅读。完整的可运行版本将在文末分享。
我们围绕构建的游戏只是一个工作示例。当你读完时，你将能够拿这个确切的结构并将其适应到你自己的用例中。
# Othello RL 环境：为什么选择棋盘游戏
Othello（黑白棋）是一个在 8x8 网格上进行的双人游戏。你放置一枚棋子，使其在直线上夹住对手的棋子，所有被夹住的棋子翻转为你的颜色，当棋盘填满时，拥有更多棋子的一方获胜。
这个翻转规则就是为什么一步棋可以剧烈改变比分，以及为什么角落如此重要（角落的棋子永远无法被翻转回来）。
一个 Othello 棋盘
每个 RL 部分在这里都有归宿。棋盘是状态，落子是动作，最终结果反馈为奖励，而验证移动、翻转棋子并扮演对手的游戏引擎就是环境。
LLM 执黑（先手），内置引擎执白。在每次黑棋移动后，白棋回应，更新后的棋盘返回给模型，如此重复直到游戏结束。
RL 循环示意图，显示 LLM、环境、对手引擎和奖励组件
# 技术栈
三个工具促成了这一切，每个处理一层：
- Verifiers：RL 框架。它定义环境，运行回合循环，并处理评估。
- Lightning AI：一个兼容 OpenAI 的推理 API，因此相同的代码可以调用托管模型（如 Claude 或 DeepSeek）而无需针对特定提供商重写代码。
- vLLM：在本地提供支持开源权重的模型，并隐藏在同一个兼容 OpenAI 的端点后。
共享接口使环境与模型无关。本地的 Ministral-3B 和托管的 GPT-4.1 只需一行代码即可互换——你只需更改模型名称，其他什么都不用改。
# 游戏循环
以下是模型在每个回合看到的内容：
模型在每个回合看到的状态
棋盘、分数和有效移动列表构成了整个状态。模型关于游戏所知的所有信息都来自这段文本。
它必须以一个 ``` section followed by a <move> tag, which forces it to reason through the position before committing to a move.
从起始位置经过 LLM 响应、验证、白棋引擎移动到更新棋盘的 5 步回合流程
环境根据有效列表验证移动，将其应用到棋盘上，让白棋回应，然后将更新后的棋盘发回。无效的移动会收到错误消息并重试，同时奖励会被扣分。
所有这些都存在于一个方法中。OthelloEnv 继承自 Verifiers 中的 MultiTurnEnv，后者处理循环、回合跟踪和终止，并且基类每次模型发送移动时都会调用你的 env_response：
```
class OthelloEnv(MultiTurnEnv):
def env_response(self, model_output, state):
move = parse_move(model_output)
if not is_valid(move, state.board):
state.penalty += INVALID_MOVE_PENALTY
return error_message(move), state # same turn, retry
state.board = apply_move(state.board, move, player="black")
if not game_over(state.board):
white_move = opponent_engine(state.board, state.difficulty)
state.board = apply_move(state.board, white_move, player="white")
return render_board(state.board), state
```
这就是逻辑的形状；真实版本还处理了片段跳过的棋盘解析和边缘情况。一旦我们端到端地理解了设置，我稍后会分享完整的工作代码。
# 内置对手引擎
内置引擎执白（后手），有两种模式：
随机 vs MiniMax 对手模式
- 随机：选择任何合法移动。这对早期训练很有用，因为模型只要做出合理的决定就能赢。
- Minimax：模拟未来的移动，通过角落控制、棋盘位置和可用移动来对每个结果位置进行评分，并选择具有最佳最坏结果的那一步。深度-3 意味着它向前看三步，足以设置陷阱并避免明显的失误。
一个随机性参数控制白棋忽略其策略并随机选择的频率。较低的随机性意味着更一致、更具惩罚性的玩法。
白棋的移动是从当前棋盘状态和固定的游戏种子生成的，因此相同的位置总是产生相同的回应。这种确定性使你能够在相同的条件下比较不同的模型。
以下是相应的代码样子：
```
def opponent_engine(board, randomness, depth):
if random.random() < randomness: # random mode
return random.choice(legal_moves(board))
best_move = None # minimax mode: try every move,
best_score = float("-inf") # keep the one with the best score
for move in legal_moves(board):
score = minimax(apply_move(board, move), depth - 1, my_turn=False)
if score > best_score:
best_move, best_score = move, score
return best_move
```
# 奖励函数
在游戏结束时，四个信号组合成一个奖励：
奖励组件表，列出了每个信号的测量内容及其权重
每一个都是一个读取最终状态并返回数字的函数，其中获胜信号是最简单的：
```
def win_reward_func(state):
result = state.get("result")
if result == "black": # the model's color
return 1.0
if result == "draw":
return 0.5
return 0.0 # loss, or game never finished
```
其他三个遵循相同的形状，所有四个组合成一个分数：
```
def total_reward(state):
return (
win_loss_score(state)
+ piece_advantage(state)
+ format_compliance(state)
- invalid_move_penalty(state)
)
```
之所以使用四个信号而不是单一的胜负位，是因为分辨率。在训练早期，大多数游戏都是失败，对于纯粹的胜负奖励来说，它们看起来都一样，因此模型没有攀登的阶梯。
- 棋子优势：将惜败与惨败区分开来，在模型开始获胜之前就为其提供梯度。
- 格式合规性：权重很低，因此整洁的格式永远不会胜过良好的玩法。
- 无效移动惩罚：设有上限，因此一场糟糕的游戏不会淹没模型所做的其他正确的事情。
每个分数都直接来自游戏状态和规则，不涉及评判模型或 LLM 评估器，因此奖励是完全确定性和可复现的。
# 连接在一起
完整的 Othello 系统图，包含游戏循环、奖励函数和 prime 评估
环境生成跨越不同起始位置和对手难度的游戏，模型通过上面的循环进行每一局，在游戏结束时计算并组合四个奖励。在评估期间，奖励、Token 使用量和回合数会在所有游戏中汇总成结果表。
一个函数将所有东西连接起来，这也是 prime eval 命令在幕后调用的：
```
def load_environment(min_random_move_prob, max_random_move_prob, parse_think):
dataset = generate_games(min_random_move_prob, max_random_move_prob)
parser = XMLParser(fields=["think", "move"])
rubric = Rubric(funcs=[...], weights=[1.0, 1.0, 0.2, 1.0])
return OthelloEnv(dataset, parser, rubric)
```
# 数据显示了什么
运行评估只需要一条命令，对手设置作为参数传递：
```
prime eval run othello -m openai/gpt-4.1 -n 100 \
-a '{"min_random_move_prob": 0.0, "max_random_move_prob": 0.0, "minimax_depth": 3}'
```
交换模型名称以测试其他模型，无论是通过 Lightning AI 托管还是在本地 vLLM 服务器上运行。以下是两个模型在两个对手面前的表现：
gpt-4.1 和 Ministral-3B 在随机和深度-3 minimax 对手下的评估结果表
# 从评估到训练
评估告诉你模型在哪里挣扎，而训练通过三个阶段解决它。同一个环境支持所有阶段：
三阶段流程：数据生成、SFT 和 RL 训练，共享相同的环境和奖励
数据生成：让你最强的模型玩几百局游戏并保存结果。相同的 eval 命令可以直接写入数据集：
```
prime eval run othello -m openai/gpt-4.1 -n 200 \
--save-to-hf-hub --hf-hub-dataset-name your-username/othello-data
```
在训练之前将其筛选为仅包含获胜和平局，这样你就不会在教模型格式的同时，教它你最强选手的错误。
监督微调（SFT）：首先教会格式和有效移动。Ministral-3B 的回合计数直接说明了这一点，因为不可靠的格式和非法移动是 RL 无法训练通过的噪音。
RL 训练：这是策略改进的地方。从相同的起始位置进行多局游戏，每一局都使用与评估相同的奖励函数进行评分，然后更新模型朝向分数较高的轨迹。
剥离到核心，这就是文章开头提到的 GRPO 循环：
```
for prompt in batch:
rollouts = [play_game(model, prompt) for _ in range(group_size)]
rewards = [total_reward(r.final_state) for r in rollouts]
advantage = rewards - mean(rewards) # relative to the group
update_model(model, rollouts, advantage)
```
每个轨迹都针对其所在组的平均值进行评分，因此从给定位置赢得 10 局中 6 局的移动，比旁边较弱的尝试会获得更多奖励。
# 将此适应到你自己的任务
一旦去除了 Othello 的细节，同一个 MultiTurnEnv 就为你提供了一个适合任何回合制任务的骨架：
```
class TaskEnv(MultiTurnEnv):
def env_response(self, model_output, state):
action = parse_action(model_output) # your task's syntax
if not is_valid(action, state):
return error_message(action), penalize(state)
state = apply_action(state, action) # your task's rules
if not task_complete(state):
state = environment_step(state) # tool, API call, or opponent
return render_state(state), state # your task's display
```
这不仅仅适用于游戏。编码代理将 apply_action 交换为针对生成代码运行测试套件，支持代理将其交换为检查工具调用是否检索到了正确的记录，研究任务将其交换为根据引用来源验证声明。
适应它归结为四个交换：
- 任务逻辑：你的领域关于什么算作有效动作以及状态如何变化的规则
- 响应引擎：模型对其反应的任何事物，从基于规则的模拟器到实时 API 再到另一个模型
- 奖励函数：保持模式（结果信号、部分功劳、格式、惩罚）并替换领域逻辑
- 状态渲染：模型需要看到的任何东西，无论是文件差异、对话记录还是工具的响应
无论领域如何，底层的结构保持不变：解析、验证、应用、响应、评分。标准是设计，如果你把组件搞对了，训练信号自然会水到渠成。
# 亲自尝试
所有代码、设置说明以及用于复现这些结果的即用型 GPU 都在 Lightning AI Studio 模板中：
构建自定义 RL 环境 →
查看 Lightning AI 推理 →
Lightning AI 推理仪表板，列出模型及其成本、延迟、吞吐量和上下文长度
感谢阅读！
干杯！ :)