# On SFT, RL, and on-policy distillation
**作者**: will brown
**日期**: 2026-05-01T02:23:07.000Z
**来源**: [https://x.com/willccbb/status/2050038277454143918](https://x.com/willccbb/status/2050038277454143918)
---
Why the SFT-RL pipeline works, where on-policy distillation fits, and how self-distillation goes wrong.
Authors: Will Brown & Claude Opus 4.7
April 30, 2026
[Editor's Note: Arguments are mine, writing is Claude's. This is partially an experiment in trying to get Claude to help speed-write and structure technical research blogs, drafted initially as artifacts and refined via "debate". I have too many blog ideas that I never get around to writing up, but the models finally feel good enough to help out with this (hopefully -- let me know what you think).]
## §1 — The standard pipeline and the compounding argument
Most post-training pipelines are some version of "SFT first, then RL" — pre-train, supervised-finetune to get a baseline, then run RL once SFT data dries up or stops moving the needle. People usually treat this ordering as a convention, but there's a real argument for it that's worth being explicit about. The argument is about which sampling distribution your method gets to compound with, and where the resulting performance ceiling sits.
When I say SFT in this post I mean teacher SFT: training on completions produced by some teacher model. (Plain instruction-tuning on human-curated data is the same shape, with humans as the teacher.) The defining property is that the sampling distribution is fixed at dataset-construction time. As the model improves during training, the data does not. Once the student gets close to the teacher's distribution, more SFT mostly memorizes — the marginal example is no longer informative. SFT's ceiling is roughly the teacher's.
RL is the opposite. The student samples its own rollouts, the gradient updates the policy, and the next batch is sampled from the improved policy. Improvements compound back into the sampling distribution. The ceiling isn't determined by the data — it's determined by whatever the verifier can grade.
This produces a tipping point. When current performance is far below the teacher and teacher data is cheap, SFT bits are extremely cheap-per-improvement — you're learning capabilities you don't have, from a source that does. As current performance approaches the teacher, marginal SFT examples get less informative, and the student's own rollouts start producing genuinely new strategies via lucky exploration that an RL gradient can extract. Past the tipping point, rollout compute is better spent on RL than on more SFT.
Rejection-sampled SFT (SFT-RS, sometimes RFT — sample from the teacher or the student, filter for correctness, train on the survivors) is strictly better than vanilla teacher SFT in expectation, but it doesn't fix the compounding problem. The sampling distribution is still pinned to whatever you're filtering. Once your filter saturates — everything correct is included, everything wrong is discarded — you've hit the same ceiling. SFT-RS shifts the curve up, but it doesn't change its shape.
None of this is new, but it's worth saying clearly because everything else in this post is about how to push past that ceiling, and the available moves depend on what you have access to.
## §2 — Same-family vs. different-family teachers
"Doing SFT" involves choosing a teacher, and the teacher's relationship to your student is a major axis of efficiency that doesn't get enough attention.
Same-family teacher means tokenizer-matched and recipe-matched: same vocabulary, same general training pipeline, ideally the same base model at different scales. Qwen3-32B teaching Qwen3-8B-Base is the canonical example, originating in the Qwen3 technical report and revisited prominently in Thinking Machines' OPD post. The teacher's outputs sit in a distribution that's structurally close to what the student naturally produces. The SFT signal — for each token, "the teacher would have said this" — is mostly about the capability gap, not stylistic differences. Per-token logprobs from the teacher are directly comparable to the student's, because the vocabulary is shared and similar training recipes produce similar distributional shapes.
Different-family teacher means tokenizer mismatch, recipe mismatch, or both. Distilling a frontier closed model into an open base, or moving across model families. Two specific costs:
- Tokenizer mismatch means every teacher completion has to be re-tokenized in the student's vocabulary. You lose information at boundaries, and per-token logprobs no longer correspond to "the same prediction at the same position" between models. Soft-target distillation is essentially impossible without expensive workarounds.
- Recipe mismatch means the teacher's outputs carry stylistic and structural artifacts — how it formats lists, when it uses chain-of-thought, what register it speaks in — that are pipeline byproducts, not capabilities. The student has to absorb those alongside the actual content. A nontrivial fraction of the SFT bits go toward learning surface form rather than competence, and the resulting student often looks like a slightly worse version of the teacher's style without having internalized the teacher's reasoning.
Most cross-family distillation in practice loses something in the gap between "what the teacher knows" and "what the student can absorb without learning the teacher's pipeline." Same-family SFT is much more efficient per unit of capability transferred, even if you can't always afford to limit yourself to it.
This matters because the next move on the algorithm tree — on-policy distillation — is essentially only available in the same-family setting, and that constraint shapes the rest of this post.
## §3 — On-policy distillation as the same-family upgrade
On-policy distillation (OPD, as in Lu et al. 2025 and the Qwen3 technical report before it) is the obvious move once you've laid out the previous two sections. The student samples its own rollouts — so you get RL's compounding in the sampling distribution — but each token in the rollout is graded by the teacher via per-token reverse KL:
The advantage at each token is "how much the teacher prefers this token relative to the student." Dense, on-policy, reverse-KL. The reported numbers are striking — roughly 9–30× less compute than RL on AIME-style benchmarks, with the gap widening when teacher logprob calls can be parallelized.
This raises a question that figure 1 implicitly answers: why does OPD's practical ceiling sit above SFT-RS's, when both target the same teacher? Cheaper sampling is part of it — the teacher only forward-passes over student tokens, which is essentially prefill, and prefill is much cheaper per FLOP than generation. But that mostly buys faster convergence, not a higher ceiling. The deeper reason is on-policy state coverage. SFT and SFT-RS train under the teacher's state distribution but evaluate the student under its own; the exposure-bias gap that opens up over long rollouts caps off-policy practical performance somewhere short of "teacher quality" on the actual evaluation distribution. OPD trains on student rollouts, so the gap doesn't open. Same nominal asymptotic target, different practical ceiling.
The catch is that OPD requires a same-family teacher, and the dependence is harder than people often acknowledge. You need tokenizer match just to compute the loss — the per-token KL is between teacher and student distributions over the same token positions. You also need at least approximate recipe match, because the per-token signal needs to actually be informative on the things you care about. If the teacher and student were trained with very different recipes, the reverse-KL gradient is dominated by "the teacher would have phrased this differently" rather than "the teacher would have reasoned differently here."
So OPD is the "best of both worlds" corner only when you have a same-family teacher available. When you do, it tends to dominate at moderate compute budgets — do mid-train SFT to get into the teacher's rough neighborhood, then OPD instead of RL to approach the teacher's level efficiently (you get to explore with a smaller model, and prefill is cheap). The ceiling is a different question. OPD targets the teacher's distribution by construction, so in the limit it's bounded by the teacher; RL's ceiling is bounded by the verifier, which can be much higher. So the framing isn't "OPD beats RL," it's "OPD gets you to the teacher's level much faster than RL would, and most of the time that's where you wanted to be anyway." When you don't have a same-family teacher, you fall back to either cross-family SFT (eat the recipe-mismatch tax) or RL (eat the sparsity tax) — or, more recently and speculatively, to self-distillation.
## §4 — When you don't have a same-family teacher
The natural response to "OPD is great if you have a same-family teacher" is "what if I don't?" Self-distillation tries to answer this by using the student itself as the teacher, with privileged information in the teacher's context that the student doesn't see at sampling time. Two instantiations have shipped recently; they share the same algorithmic shape and differ only in what privileged info goes to the teacher.
SDFT (Shenfeld et al. 2026) conditions the teacher on an expert demonstration — a worked example, possibly for a different task. Student samples a trajectory without the demonstration in context; teacher computes per-token logprobs over the trajectory while seeing the demonstration; student updates toward teacher via reverse KL. Fully on-policy. The demonstration provides distributional pull without leaking the answer, and the paper makes the assumption underpinning this explicit (more on that in §8).
OPSD (Zhao et al. 2026) conditions the teacher on the ground-truth answer instead. Same setup otherwise. The teacher knows where the trajectory is supposed to end up; the student doesn't. This produces a much sharper distributional shift in the teacher than a demonstration does.
In both cases tokenizer match is automatic (same model). Recipe match is automatic (same model). What you trade for that is that the privileged-info conditioning shifts the teacher's distribution — modestly for SDFT, more aggressively for OPSD — away from the student's natural distribution.
Both sit at exactly the same dial settings as OPD. The only difference is the choice of teacher. That difference turns out to be where the failure mode lives.
## §5 — The geometry of gradients
To see why, it's worth working through what each method's gradient actually looks like in parameter space. The shapes are different in instructive ways.
RL: sparse, but saved by destructive interference
The standard complaint about RL is one bit per episode — a binary "you got it right" spread across thousands of tokens. The framing I find more useful: the apparent sparsity is what keeps RL's gradient updates honest.
In a GRPO-style step, every token in every rollout gets an advantage. Within a group, the advantages have approximately zero mean by construction (subtract the group baseline) and some variance. Each per-token advantage gets multiplied by the gradient of the log-probability of that token, producing a vector in parameter space. The total update is the average over the batch.
Most of those vectors are noise. Reward is sparse and broadcast-assigned; most tokens didn't actually contribute to whether the answer was right. Their advantages are nonzero only because they happened to share a trajectory with a reward, not because they were causally responsible for it. So the batch contains a swarm of small, mostly-random parameter-space vectors, with a small consistent bias along whichever dimensions actually correlate with reward ("think longer," "double-check arithmetic").
The reason large-batch, low-learning-rate RL is robust is not that each per-token gradient is informative — most aren't — but that the uninformative ones cancel. What survives the average is the small consistent component along the directions that actually correlate with reward. There's some empirical support for this: RL updates have been observed to be sparse in parameter space, modifying small subnetworks. That seems like the destructive-interference picture cashing out.
It also explains why RL feels "safe but slow." Each step moves the model a tiny amount in a direction you can trust. The sparsity isn't purely a bug; it's the price for an unbiased estimator.
SFT: dense, biased, but spread out
SFT does the opposite. Every token gets a one-hot label. The gradient density is enormous — one informative update per token, no broadcasting required.
But the SFT gradient distribution does not have zero mean. By construction, it's biased toward the data distribution. There's no analog of advantages cancelling. Constructive interference, not destructive.
So why doesn't SFT blow up? Two reasons. First, the data distribution is itself diverse: the bias points in many slightly different directions across many examples, so you get a soft principal-components decomposition where the model drifts toward the data manifold as a whole rather than any one example. Second, SFT is diffuse. Most of what an SFT step is doing is reinforcing things the model already half-knew. There aren't many tokens where the data is asking the model to put large mass on something it currently treats as near-zero probability.
This is why SFT is forgiving. The data can be a bit messy, the learning rate can be a bit off, and you mostly drift. The bias is real, but it's unconcentrated.
OPSD: dense, biased, and concentrated
OPSD's gradient has a different shape from either of the above, and the difference is instructive — it points at why pure self-distillation isn't quite the final answer.
Consider a long math rollout where the student gets the answer wrong because, somewhere in the chain of thought, it failed to make a key observation — pick the right substitution, notice a trick. Call that token (or short span) the pivot token. For the student, the probability of producing the pivot token might be very low — say 0.01. For the teacher, conditioned on the answer, the probability of that same pivot token is much higher — say 0.6, because once you know where the solution is going, the right substitution becomes obvious.
The per-token reverse KL between "student says 0.01" and "teacher says 0.6" is roughly log(0.6/0.01) ~= 4.1. For a typical token where both put around 0.3, the contribution is essentially zero. So one pivot token contributes on the order of a hundred times more to the loss than a typical one. The gradient is dominated by it.
What does that gradient do? It pushes the student's log-probability of the pivot token up sharply, given the prefix. The corresponding direction in parameter space is "make this rare token less rare in this context." Critically, this update is not being averaged against many other vectors pointing different directions. RL's saving grace was that noise vectors cancelled. SFT's saving grace was that bias was diffuse across many tokens that the student already supports. OPSD has neither. One concentrated tug, in one step, toward a region the model didn't previously believe in.
OPSD ships with per-token point-wise KL clipping — cap the per-vocabulary-entry divergence contribution at each position so that, in their words, "a small subset of stylistic tokens [doesn't] dominate the training signal." Without it, the paper reports performance collapse within ~100 steps. That's figure 4 cashing out. The fix works, and what it tells us is direct: the KL signal in self-with-hint distillation is concentrated enough that you have to budget it. The next move, then, is to look for a teacher whose KL is naturally diffuse, rather than relying on clipping to make a concentrated one tolerable.
## §6 — The sparse/dense × biased/unbiased family
The gradient analysis above places each method in a small taxonomy. The two main axes are sparse vs. dense (does each token get a signal, or just the trajectory) and biased vs. unbiased (does the expected gradient point in a fixed direction relative to the student, or only along the directions reward correlates with).
SFT, OPD, and OPSD all live in the "dense, biased" cell. They're distinguished by where the bias points (toward the data, toward a same-family teacher, toward self-with-hint) and, more importantly for the failure-mode analysis, by how that bias is distributed across tokens. That's the third axis — concentration:
This is the punchline of the geometric analysis. RL is sparse but unbiased. SFT is dense and biased but diffuse. Same-family OPD inherits SFT's diffuseness because the teacher distribution is calibrated to the student's family. OPSD is the unique case where you get density, bias, and concentration at once — which is why it ships with explicit defenses (KL clipping, fixing the teacher to the initial policy) that the other methods don't need.
## §7 — An illustrative meta-algorithm
All four methods are special cases of a single token-level policy gradient:
with two scalar knobs and a teacher-policy choice:
- α ∈ [0,1] — how on-policy the sampling distribution is.
- λ ∈ [0,1] — how much of the per-token advantage comes from a teacher KL versus a sequence-level outcome reward.
- π_T — the teacher policy: which model, conditioned on what context c_T.
Once you fix α and λ, almost everything that matters — bias, concentration, stability — is a function of how π_T differs from π_θ at the per-token level on the student's rollouts. The interior of this space — α, λ ∈ (0,1), or mixed teacher choices — is mostly unexplored, and seems like where the next round of progress lives.
A few things drop out:
- SFT is "distillation from a degenerate teacher": letting π_T(y|x) = δ_{y_data}(y) recovers cross-entropy up to constants. The reason SFT is safe is not that this teacher is bad — it's that it's averaged across many varied examples, which makes the bias diffuse.
- RL is "no teacher": set λ=0 and the per-token signal collapses to broadcast outcome reward. The destructive-interference story follows from this.
- OPSD has identical dial settings to OPD: the only difference is π_T. That clarifies why it can match OPD when it works (same algorithm) and why it's where the failure mode is sharpest (most distributionally aggressive choice of teacher).
A caveat about what this factorization is for: it’s illustrative of the kind of algorithm conceptually aligned with the Pareto / SFT-RL phase-change picture — not a recommendation to mix sources at intermediate (α, λ). I’m not sure that’s even a useful algorithm. The clean corners (SFT, RL, OPD) are where the statistics work out without importance-sampling corrections, and they correspond to qualitatively different regimes that differ mostly in the KL budget β. The interesting axis of variation across them is β; interpolating α and λ is a tangent.
The practically-interesting structure looks more like “pick β, then find the best teacher for that β.” The meta-algorithm above makes the corners visible, but the actually hard problem is the inner one — and teacher optimization isn’t clean or cheap. It’s typically discrete (which model? which prompt? which hint?) and doesn’t decompose neatly into the gradient framing. The Pareto picture in §8 is where that question actually lives.
## §8 — Toward an optimal teacher
The geometry analysis turns the optimal-teacher question into an optimization problem. We want a teacher policy π_T that produces a large reward improvement on the student per step, subject to a hard KL constraint that keeps updates stable. In Lagrangian form:
This traces a Pareto curve as β varies. Different methods are different points on it.
The SDFT paper's framing is worth engaging with directly here, because it offers a candidate route to the curve's interior that doesn't quite work for the reasons we want it to. The paper formalizes the ICL assumption — that a model conditioned on an expert demonstration approximates the optimal policy for the task — and conditions its analysis on the availability of such demonstrations. That assumption is doing a lot of work. Practically, this gives a similar ceiling (conceptually) to SFT and SFT-RS: you're only as good as the demonstrations, and you can't bootstrap beyond them.
We'd rather be in a world with curves more like RL, where the ceiling is only a function of our ability to verify, even if we can't always solve (without extremely expensive sampling). We can make weaker assumptions about the ability to verify (which we need anyway if we want to measure performance) plus mild regularity conditions on the distribution: that performance in accuracy degrades gracefully across the task distribution in a smooth latent space coupling task characteristics to the model's solve likelihood for a given checkpoint, and that updates should be net-positive-EV in distribution solve rate when sufficiently well-regularized — the high-batch-size, low-learning-rate, unbiased limit of, e.g., infinite-batch RL. This is what's needed for compounding gains where the only ceiling is the grader's performance, with the question being how to get there faster while retaining stability.
Policy gradient — at least within outcome-reward GAE — is roughly a minimum-variance unbiased estimator for the gradient of expected reward. The aim is to be able to turn a knob that trades bias for stable-yet-aggressive updates. OPD and SFT are two points on this curve, but both need a real teacher. Using a bigger smarter model improves the teacher but isn't directly workable for OPD without tokenizer match. There are cross-tokenizer methods, but ultimately we want something that lets us sweep teacher performance versus bias cleanly and cost-effectively.
Several plausible approaches target this objective at different levels of the stack:
- Per-task prompt optimization over the Lagrangian. Use something like GEPA over the Lagrangian objective on a per-task basis, with teacher sampling to estimate rewards. The optimizer searches the space of prompts (hints, conditioning) for the one that maximizes E[Δreward] − β·KL on the current student. This is a cheap inner loop that can produce a usable teacher without retraining anything.
- Distribution-level prompt optimization. Train a model to write good minimal hints when given a "bad" hint — whitebox RF access, the answer itself, a known-good demonstration. The training signal is the same Lagrangian, aggregated across the task distribution. The output is a hint-rewriter that turns large privileged-info hints into small ones that move the teacher distribution as little as possible while still improving reward.
- Self-prompt-optimization online RL. Treat hints as rollouts in a parallel environment that improves alongside the main policy via an RL objective, with loss components balanced adaptively (e.g., a smoothed minmax over reward-delta and KL terms). The hint-writer and student co-evolve.
- Train a hint-writing model directly with RL. Use correctness-delta × (1 − KL-delta) as the objective, scoped to a specific model / distribution / task / "in general" (the same way scopes work for judge-model training). The hint-writer is a separate artifact you can ship.
- Borrow from "expert RL + OPD." Some recent models (e.g. DeepSeek V4) layer a teacher signal on top of locally-optimal RL — an analog of the meta-algorithm with both a teacher KL and an outcome reward live simultaneously. The lessons from that line are directly relevant here.
What unifies these is that they all target the Lagrangian objective without requiring a fixed external teacher policy: the "teacher" is constructed (per task, per distribution, or online) to be locally optimal for the current student — high reward, low KL, surgical rather than broadcast.
In the limit, RL might still be optimal in the infinite-compute regime for the hardest heavy-tail problem distributions — the cases where any teacher distribution adds bias the student would've eventually corrected anyway. But it seems quite plausible that there's a nice meta-algorithm for interpolating more cleanly between distillation and RL: without needing a real teacher, and with compute-optimal learning at each point along the curve. Working that out is the open question I find most interesting in this space.
References
· Lu, K. & Thinking Machines Lab. On-Policy Distillation (2025).
· Zhao, S. et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD, 2026).
· Shenfeld, I. et al. Self-Distillation Enables Continual Learning (2026).
· Agarwal, R. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023).
· Mukherjee, A. et al. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (2025).
· Qwen Team. Qwen3 Technical Report (2025).
· Ross, S. et al. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAGGER, 2010).
## 相关链接
- [Pollux9437 reposted](https://x.com/Orange41324306)
- [will brown](https://x.com/willccbb)
- [@willccbb](https://x.com/willccbb)
- [46K](https://x.com/willccbb/status/2050038277454143918/analytics)
- [10:23 AM · May 1, 2026](https://x.com/willccbb/status/2050038277454143918)
- [46.1K Views](https://x.com/willccbb/status/2050038277454143918/analytics)
- [View quotes](https://x.com/willccbb/status/2050038277454143918/quotes)
---
*导出时间: 2026/5/1 14:40:54*
---
## 中文翻译
# 关于SFT、RL与同策略蒸馏
**作者**：will brown
**日期**：2026-05-01T02:23:07.000Z
**来源**：[https://x.com/willccbb/status/2050038277454143918](https://x.com/willccbb/status/2050038277454143918)
---
为什么SFT-RL流水线有效、同策略蒸馏适用于何处，以及自蒸馏如何走向错误。
作者：Will Brown & Claude Opus 4.7
2026年4月30日
[编者按：论点属于我，文字由Claude完成。这在一定程度上是一项实验，尝试让Claude帮助快速撰写并构建技术研究博客的结构，最初以artifact形式起草，并通过“辩论”来完善。我有太多博客想法一直没能写出来，但模型终于感觉足够好了，可以在这方面帮忙（希望如此——欢迎告诉我你的看法）。]
## §1 — 标准流水线与复利论证
大多数后训练流水线都是“先SFT，后RL”的某种版本——预训练，监督微调以获得基线，然后在SFT数据枯竭或不再带来提升时运行RL。人们通常把这种顺序当作惯例，但背后有一个真正的论证值得明确说明。这个论证关乎你的方法能复利叠加的采样分布是什么，以及由此产生的性能上限在哪里。
在本文中我说的SFT指的是教师SFT：在某个教师模型生成的补全上训练。（对人工策划数据进行的普通指令微调是同样的形式，只是以人类作为教师。）其定义性特征是采样分布在数据集构建时就已固定。随着模型在训练中不断改进，数据却不会。一旦学生接近了教师的分布，更多的SFT主要就是在记忆——边缘样本不再有信息量。SFT的上限大致就是教师的水平。
RL则相反。学生对自己产生的rollout进行采样，梯度更新策略，下一批数据则从改进后的策略中采样。改进会复利地反馈到采样分布中。上限不由数据决定——它由验证器能评分的东西决定。
这就产生了一个临界点。当当前性能远低于教师、且教师数据廉价时，SFT比特的每次改进成本极低——你在从一个拥有你所缺乏能力的源头学习这些能力。当当前性能接近教师时，边缘SFT样本的信息量下降，而学生自己的rollout开始通过幸运探索产生真正新颖的策略，RL梯度可以将其提取出来。越过临界点之后，rollout算力花在RL上比花在更多SFT上更划算。
拒绝采样SFT（SFT-RS，有时称RFT——从教师或学生采样，过滤正确性，在幸存样本上训练）在期望上严格优于普通的教师SFT，但它无法解决复利问题。采样分布仍然被钉死在你所过滤的对象上。一旦你的过滤器饱和——所有正确的都被纳入，所有错误的都被丢弃——你就撞上了同样的天花板。SFT-RS把曲线向上平移，但不改变其形状。
这些都不是新观点，但值得清晰地说明，因为本文其余部分讲的都是如何突破那个上限，而可行的路径取决于你手上有什么。
## §2 — 同族教师 vs. 异族教师
“做SFT”涉及选择教师，而教师与学生之间的关系是一个未被足够重视的重要效率轴。
同族教师指的是分词器匹配、配方匹配：相同的词表，相同的大致训练流程，理想情况下是同一基础模型的不同规模版本。Qwen3-32B教Qwen3-8B-Base是经典例子，源自Qwen3技术报告，并在Thinking Machines的OPD文章中得到显著重提。教师的输出所处的分布与学生自然产生的分布在结构上很接近。SFT信号——对每个token来说，“教师会说这个”——主要体现的是能力差距，而非风格差异。教师逐token的logprob与学生的直接可比，因为词表共享，且相似的训练配方产生相似的分布形状。
异族教师指的是分词器不匹配、配方不匹配，或两者兼有。例如把前沿闭源模型蒸馏到开源基座中，或跨模型家族迁移。这带来两个具体代价：
- 分词器不匹配意味着每条教师补全都要在学生词表中重新分词。边界处会丢失信息，且逐token的logprob不再对应“同一位置的同一预测”。没有昂贵的变通手段，软目标蒸馏基本不可行。
- 配方不匹配意味着教师的输出带有风格和结构上的痕迹——它如何排版列表、何时使用思维链、用什么语调说话——这些是流水线的副产物，而非能力。学生不得不在吸收实际内容的同时吸收这些。相当一部分SFT比特被用在学习表层形式而非能力上， resulting的学生往往看起来像教师风格的略差版本，却没有内化教师的推理。
实践中，大多数跨族蒸馏都会在“教师知道什么”与“学生能在不学教师流水线的情况下吸收什么”之间的鸿沟中损失一些东西。同族SFT每单位能力迁移的效率要高得多，即使你并不总能负担得起把自己局限于它。
这一点很重要，因为算法树上的下一步——同策略蒸馏——基本只在同族设定下可用，而这个约束塑造了本文的其余部分。
## §3 — 同策略蒸馏作为同族升级
在线蒸馏（OPD，如 Lu 等人 2025 年及更早的 Qwen3 技术报告所述）在看完前两节的内容后，显然是水到渠成的选择。学生模型自己采样其推演序列——因此你获得了 RL 在采样分布上的复合优势——但推演序列中的每个 token 都由教师模型通过逐 token 反向 KL 进行评分：
每个 token 上的优势是"教师相对于学生更偏好这个 token 的程度"。密集的、在线的、反向 KL。报告的数字相当惊人——在 AIME 风格的基准上，计算量大约比 RL 少 9–30 倍，而且当教师 logprob 调用可以并行化时，差距还会进一步扩大。
这引出一个图 1 已经隐含回答的问题：既然两者都以同一个教师为目标，为什么 OPD 的实际上限高于 SFT-RS？更便宜的采样是其中一部分原因——教师只需对学生 token 做前向传播，这本质上是预填充（prefill），而预填充每 FLOP 的成本远低于生成。但这主要买到的是更快的收敛，而不是更高的上限。更深层的原因是在线状态覆盖。SFT 和 SFT-RS 在教师的状态分布下训练，但在学生自己的分布下评估学生；这种暴露偏差在长推演中造成的差距，把离线方法的实际上限限制在"教师质量"以下的某个位置，而这是在实际评估分布上。OPD 在学生的推演上训练，所以这个差距不会出现。名义上相同的渐近目标，实际不同的上限。
问题在于 OPD 需要同家族的教师，而且这种依赖性比人们通常承认的要苛刻。你需要 tokenizer 匹配才能计算损失——逐 token 的 KL 是在相同 token 位置上教师分布与学生分布之间计算的。你还需要至少大致匹配的训练配方，因为逐 token 的信号必须在你真正关心的事情上确实有信息量。如果教师和学生是用非常不同的配方训练的，反向 KL 梯度将由"教师会换个说法来表达这个"主导，而不是"教师会在这里有不同的推理"。
所以，只有当你有同家族教师可用时，OPD 才是"两全其美"的角落。有的话，它在中等计算预算下往往占主导——先做训练中期 SFT 进入教师的大致邻域，然后用 OPD 而非 RL 来高效接近教师的水平（你可以用较小的模型来探索，而且预填充很便宜）。上限是另一个问题。OPD 从构造上就以教师的分布为目标，所以在极限情况下它受限于教师；RL 的上限受限于验证器，而验证器可以高得多。所以正确的表述不是"OPD 打败 RL"，而是"OPD 让你比 RL 快得多地达到教师的水平，而且大多数时候那正是你想要到达的地方"。当你没有同家族教师时，你就只能退而求其次，选择跨家族 SFT（承担配方不匹配的代价）或 RL（承担稀疏性的代价）——或者，最近更具推测性地，选择自蒸馏。
## §4 — 当你没有同家族教师时
对"如果你有同家族教师，OPD 很棒"的自然回应是："那如果没有呢？"自蒸馏试图通过让学生自身充当教师来回答这个问题，并在教师的上下文中注入特权信息——这是学生在采样时看不到的。最近有两个实例已经落地；它们共享相同的算法形态，区别仅在于提供给教师的特权信息是什么。
SDFT（Shenfeld 等人 2026）以专家示范为教师提供条件——一个 worked example，可能是针对不同任务的。学生采样一条轨迹，但上下文中没有示范；教师在看到示范的情况下计算该轨迹上的逐 token logprob；学生通过反向 KL 向教师更新。完全在线。示范提供了分布上的牵引力而不泄露答案，论文把支撑这一做法的假设写得非常明确（§8 会详细讨论）。
OPSD（Zhao 等人 2026）则改为以标准答案为教师提供条件。其他设置相同。教师知道轨迹应该到哪里结束，而学生不知道。相比示范，这会在教师身上产生更尖锐的分布偏移。
在这两种情况下，tokenizer 匹配都是自动的（同一个模型）。配方匹配也是自动的（同一个模型）。你为此付出的代价是：特权信息条件化会使教师的分布偏离学生的自然分布——SDFT 中是温和的，OPSD 中则更激进。
两者都处于与 OPD 完全相同的拨盘设置上。唯一的区别是教师的选择。而这个区别，恰恰就是失败模式所在之处。
## §5 — 梯度的几何结构
要理解为什么，值得逐个剖析每种方法的梯度在参数空间中到底是什么样子的。它们的形态以富有启发性的方式各不相同。
RL：稀疏，但被破坏性干扰所拯救
对 RL 的标准抱怨是每个 episode 只有一个比特——一个二元的"你答对了"，摊在数千个 token 上。我觉得更有用的表述是：正是这种表面上的稀疏性，让 RL 的梯度更新保持诚实。
在 GRPO 风格的一步中，每个 rollout 中的每个 token 都有一个优势。在同一个组内，优势的均值在构造上近似为零（减去组基线），并具有一定的方差。每个逐 token 优势都乘以该 token 对数概率的梯度，从而在参数空间中产生一个向量。总更新是对整个 batch 取平均。
这些向量中的大多数都是噪声。奖励是稀疏且被广播式分配的；大多数 token 实际上并没有对答案是否正确做出贡献。它们的优势值非零，只是因为它们恰好与奖励共享同一条轨迹，而不是因为它们在因果上导致了奖励的产生。因此，这个批次包含一大群小的、近乎随机的参数空间向量，仅在那些真正与奖励相关的维度上存在一个微小的一致性偏置（“思考更久”、“复查算术”）。
大批量、低学习率 RL 之所以稳健，并不是因为每个 per-token 梯度都含有信息——大多数并不含有——而是因为那些不含信息的梯度会相互抵消。在求平均之后留存下来的，是沿着那些真正与奖励相关的方向上的微小一致性分量。这方面有一些实证支持：有研究观察到 RL 更新在参数空间中是稀疏的，只修改小的子网络。这看起来正是“破坏性干涉”图景的现实印证。
这也解释了为什么 RL 让人感觉“安全但缓慢”。每一步都只让模型朝着一个你可以信任的方向移动一小段。稀疏性并不纯粹是个缺陷；它是无偏估计器的代价。
SFT：稠密、有偏，但弥散
SFT 则相反。每个 token 都有一个 one-hot 标签。梯度密度极大——每个 token 一次有信息量的更新，无需广播。
但 SFT 的梯度分布均值不为零。从构造上讲，它偏向数据分布。不存在优势值相互抵消的类似机制。这是建设性干涉，而非破坏性干涉。
那么为什么 SFT 不会失控？两个原因。第一，数据分布本身是多样的：偏置在许多样本上指向许多略微不同的方向，所以你会得到一种软化的主成分分解效果，模型整体上向数据流形漂移，而不是向任何单个样本漂移。第二，SFT 是弥散的。SFT 一步所做的大部分事情，是在强化模型已经半知半解的东西。数据要求模型把大量概率质量放在它目前视为接近零概率的事物上的 token 并不多。
这就是 SFT 宽容的原因。数据可以有点乱，学习率可以有点偏差，而你大多只是在漂移。偏置是真实存在的，但它并不集中。
OPSD：稠密、有偏，且集中
OPSD 的梯度形状与上述两者都不同，而这个差异很有启发性——它指出了为什么纯自蒸馏并不完全是最终答案。
考虑一个长数学 rollout，学生答错了，因为在思维链的某处，它未能做出一个关键观察——选择正确的代换、注意到某个技巧。把那个 token（或一小段）称为枢纽 token。对学生来说，生成这个枢纽 token 的概率可能非常低——比如 0.01。对老师来说，在已知答案的条件下，同一个枢纽 token 的概率要高得多——比如 0.6，因为一旦你知道解答的方向，正确的代换就变得显而易见了。
“学生说 0.01”与“老师说 0.6”之间的 per-token 反向 KL 大约为 log(0.6/0.01) ≈ 4.1。对于一个两者都给出约 0.3 的典型 token，其贡献基本为零。所以一个枢纽 token 对损失的贡献大约是一个典型 token 的一百倍。梯度被它主导了。
这个梯度做了什么？它在给定前缀的条件下，急剧推高学生的枢纽 token 对数概率。参数空间中对应的方向是“让这个在这个语境下罕见的 token 变得不那么罕见”。关键在于，这一更新并没有与许多指向其他方向的向量做平均。RL 的救命之处在于噪声向量会相互抵消。SFT 的救命之处在于偏置弥散在许多学生本来就支持的 token 上。OPSD 两者都没有。一步之内，一次集中的拉扯，朝向一个模型此前并不相信的区域。
OPSD 自带了 per-token 逐点 KL 裁剪——在每个位置对每个词表条目的散度贡献设上限，用他们的话说，这样“一小撮风格性 token [不至于] 主导训练信号”。没有它，论文报告在约 100 步内出现性能崩溃。那就是图 4 的现实印证。这个修复是有效的，它直接告诉我们：带提示自蒸馏中的 KL 信号足够集中，你必须对它做预算控制。那么下一步，就是去寻找一种 KL 天然弥散的老师，而不是依靠裁剪让一个集中的信号变得可容忍。
## §6 — 稀疏/稠密 × 有偏/无偏家族
上面的梯度分析把每种方法归入了一个小型分类体系。两个主要轴是稀疏 vs. 稠密（是每个 token 都获得信号，还是只有整条轨迹获得信号）以及有偏 vs. 无偏（期望梯度是指向相对于学生的某个固定方向，还是只沿着与奖励相关的方向）。
SFT、OPD 和 OPSD 都处在“稠密、有偏”这一格。它们的区别在于偏置指向哪里（指向数据、指向同族老师、指向带提示的自己），而且对于失败模式分析而言更重要的是，偏置在 token 间如何分布。这是第三个轴——集中度：
这就是几何分析的核心结论。RL 稀疏但无偏。SFT 稠密且有偏但弥散。同族 OPD 继承了 SFT 的弥散性，因为老师分布是与学生的家族校准一致的。OPSD 是唯一一个同时具备稠密性、偏置和集中性的特例——这正是它需要附带其他方法都不需要的显式防御机制（KL 裁剪、把老师固定为初始策略）的原因。
## §7 — 一个示例性的元算法
所有四种方法都是同一个 token 级策略梯度的特例：
带有两个标量旋钮和一个教师策略选择：
- α ∈ [0,1] — 采样分布的 on-policy 程度。
- λ ∈ [0,1] — 每 token 优势中来自教师 KL 与序列级结果奖励的比重。
- π_T — 教师策略：哪个模型，以什么上下文 c_T 为条件。
一旦确定了 α 和 λ，几乎所有重要的东西——偏差、集中度、稳定性——都取决于 π_T 在学生 rollout 的 token 级别上与 π_θ 的差异程度。这个空间的内部区域——α, λ ∈ (0,1)，或混合教师选择——基本上未被探索，而下一轮进展似乎正位于此。
有几点可以直接推出：
- SFT 是“来自退化教师的蒸馏”：令 π_T(y|x) = δ_{y_data}(y)，在相差常数的情况下可恢复交叉熵。SFT 安全的原因并不是这个教师不好，而是它是在许多多样的样本上取平均的，这使得偏差变得弥散。
- RL 是“无教师”：设 λ=0，token 级信号退化为广播式结果奖励。破坏性干扰的故事正源于此。
- OPSD 的旋钮设置与 OPD 完全相同：唯一的区别是 π_T。这说明了为什么它在有效时能与 OPD 匹敌（相同的算法），以及为什么它是失效模式最尖锐的地方（教师选择在分布上最为激进）。
关于这一因子分解的用途需要说明一点：它只是概念上与 Pareto / SFT-RL 相变图景相一致的那类算法的示意——并不是建议在中间的 (α, λ) 处混合数据源。我不确定那是否是一个有用的算法。干净的角点（SFT、RL、OPD）才是无需重要性采样修正统计量就能成立的地方，它们对应于性质不同的机制，而这些机制的主要区别在于 KL 预算 β。它们之间有趣的变异轴是 β；对 α 和 λ 进行插值是一个枝节问题。
实际有趣的结构更像是“先选定 β，然后为该 β 寻找最佳教师”。上述元算法让角点变得可见，但真正困难的问题是内部那个问题——而教师优化既不干净也不便宜。它通常是离散的（哪个模型？哪个提示？哪个提示信息？），并且不能整齐地分解进梯度框架。§8 中的 Pareto 图景才是这个问题真正所在之处。
## §8 — 迈向最优教师
几何分析将最优教师问题转化为一个优化问题。我们想要一个教师策略 π_T，使学生每步获得较大的奖励提升，同时受制于保持更新稳定的硬 KL 约束。用拉格朗日形式表示：
随着 β 的变化，这描绘出一条 Pareto 曲线。不同方法就是曲线上的不同点。
SDFT 论文的框架值得在此直接探讨，因为它提供了一条通往曲线内部的候选路径，但由于我们关心的原因它并不完全奏效。该论文将 ICL 假设形式化——即以专家示范为条件的模型近似任务的最优策略——并将其分析建立在可获得此类示范的前提上。这个假设承担了大量工作。实际上，这在概念上给出了与 SFT 和 SFT-RS 类似的天花板：你的上限取决于示范的质量，且无法超越示范进行自我提升。
我们更希望处于一个曲线更像 RL 的世界里，其中天花板只是我们验证能力的函数，即使我们并不总能解决问题（如果不进行极其昂贵的采样）。我们可以对验证能力做更弱的假设（如果我们想衡量性能，这本来就是必需的），再加上对分布的温和正则性条件：性能在准确率上的退化应在任务分布上表现优雅，即在一个将任务特征与模型在给定检查点下的解题可能性相耦合的光滑潜在空间中；并且当充分正则化时，更新在分布解题率上应是净正期望值的——例如无限批量 RL 的大批量、低学习率、无偏极限。这是实现复合收益所需要的，其唯一的天花板就是评分器的性能，而问题在于如何在保持稳定性的同时更快地达到那里。
策略梯度——至少在结果奖励的 GAE 框架内——大致是期望奖励梯度的最小方差无偏估计量。目标是能够转动一个在偏差与“稳定而激进”的更新之间进行权衡的旋钮。OPD 和 SFT 是这条曲线上的两个点，但两者都需要一个真实的教师。使用更大更聪明的模型可以改善教师，但对于 OPD 而言，若没有分词器匹配就无法直接可行。存在跨分词器的方法，但归根结底，我们想要某种能让我们干净且经济高效地在教师性能与偏差之间进行扫描的东西。
有几种看似可行的方法在技术栈的不同层次上针对这一目标：
- 针对拉格朗日目标进行按任务提示优化。在拉格朗日目标上，按任务使用类似 GEPA 的方法，通过教师采样来估计奖励。优化器在提示空间（提示、条件设定）中搜索能最大化当前学生模型 E[Δreward] − β·KL 的那个提示。这是一个廉价的内循环，无需任何重训练就能产生一个可用的教师。
- 分布层面的提示优化。训练一个模型，在给定一个“糟糕”提示时写出良好的最小提示——白盒 RF 访问、答案本身、一个已知良好的示例。训练信号是同样的拉格朗日目标，在任务分布上聚合。输出是一个提示重写器，将大的特权信息提示转化为小的提示，在仍然提升奖励的同时，尽可能少地移动教师分布。
- 自我提示优化的在线强化学习。将提示视为一个并行环境中的 rollout，该环境通过 RL 目标与主策略一同改进，损失各分量自适应平衡（例如，对奖励增量和 KL 项采用平滑的 minmax）。提示编写器与学生模型共同演化。
- 直接用 RL 训练提示编写模型。使用 correctness-delta × (1 − KL-delta) 作为目标，并限定于特定模型 / 分布 / 任务 / “一般情况”（与 judge 模型训练中的 scope 工作方式相同）。提示编写器是一个可以独立交付的产物。
- 借鉴“专家 RL + OPD”。一些近期模型（例如 DeepSeek V4）在局部最优 RL 之上叠加教师信号——这是那个元算法的类似物，教师 KL 和结果奖励同时存在。那条路线的经验教训在这里直接适用。
这些方法的共同点在于，它们都以拉格朗日目标为导向，而不需要固定的外部教师策略：“教师”是（按任务、按分布或在线）构建的，针对当前学生模型是局部最优的——高奖励、低 KL，是外科手术式的精准而非广播式的泛化。
在极限情况下，对于最难的重尾问题分布——即任何教师分布都会引入偏差、而学生模型最终反正会自行纠正的情况——RL 在无限算力机制下可能仍然是最优的。但似乎很可能存在一个不错的元算法，能在蒸馏和 RL 之间更干净地进行插值：不需要真实的教师，并且在曲线上的每一点都实现计算最优的学习。把它推导出来，是我认为这个领域中最有趣的开放问题。
参考文献
· Lu, K. & Thinking Machines Lab. On-Policy Distillation (2025).
· Zhao, S. et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD, 2026).
· Shenfeld, I. et al. Self-Distillation Enables Continual Learning (2026).
· Agarwal, R. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023).
· Mukherjee, A. et al. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (2025).
· Qwen Team. Qwen3 Technical Report (2025).
· Ross, S. et al. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAGGER, 2010).
## 相关链接
- [Pollux9437 reposted](https://x.com/Orange41324306)
- [will brown](https://x.com/willccbb)
- [@willccbb](https://x.com/willccbb)
- [46K](https://x.com/willccbb/status/2050038277454143918/analytics)
- [2026年5月1日 上午10:23](https://x.com/willccbb/status/2050038277454143918)
- [46.1K 次浏览](https://x.com/willccbb/status/2050038277454143918/analytics)
- [查看引用](https://x.com/willccbb/status/2050038277454143918/quotes)
---
*导出时间: 2026/5/1 14:40:54*
