# Claude Mythos、Looped LLM， and the Depth Scaling Axis
**作者**: Rui-Jie Zhu
**日期**: 2026-04-21T23:44:09.000Z
**来源**: [https://x.com/RidgerZhu/status/2046736781035618602](https://x.com/RidgerZhu/status/2046736781035618602)
---
In October 2024, I was fortunate to join ByteDance Seed as an intern under @GeZhang86038849 , doing research on foundational LLM architectures. At the time, reasoning models had just emerged, and I felt that latent reasoning could be a path toward raising the ceiling on reasoning ability. After a month of experimentation, I found that the simplest strategy was to stack the model on itself. The Looped LLMs was born from this insight. By January 2025, we realized the Looped LLMs could be more than a latent reasoning framework; it could serve as a new foundational architecture for LLMs. We went heads-down for a year on scale-up. During that time I learned firsthand what it means for people to follow the GPUs: my sleep quality was directly correlated with how often metrics collapsed. Thankfully, the paper finally shipped in November 2025.
## Why Are Looped LLMs Suddenly Linked to Mythos?
In April 2026, Anthropic released Claude Mythos Preview, which achieved SOTA on nearly all benchmarks at massive scale. What stands out, as first noted by @ChrisHayduk, is that Mythos showed an unusually large performance leap on tasks like GraphWalk BFS, while the improvements on other benchmarks were relatively modest. GraphWalk BFS deserves special attention because it is designed to strip away world knowledge entirely: nodes and edges are randomly generated, node names carry no stable semantics, and the task requires the model to trace paths step by step, maintain a frontier, and execute a full BFS iteration. In other words, these benchmarks primarily test whether a model can maintain an internal state and continuously update it, probing capabilities closer to state tracking and iterative computation.
This phenomenon echoes the mechanistic analysis in our Scaling Looped LLM paper. In that work, we analyzed the distribution of benchmark improvements brought by looping and found that gains are concentrated in tasks requiring multi-hop reasoning and fact composition, rather than pure knowledge recall. We first validated the advantage of looping on knowledge manipulation (multi-hop) reasoning using synthetic tasks:
and then extended these findings to large-scale natural language / reasoning benchmarks.
That said, there is a gap between a promising inductive bias and an architecture that may truly scale. The advantages Loop shows in controlled experiments do not directly answer a more practical question:
> When we need to continue following the scaling law trajectory and commit real compute budgets to training and inference at the 10T/100T LLM scale, how should we incorporate Loop into next-generation model infrastructure?
History is full of precedents where "beautiful results at small scale collapse at large scale." Before MoE was truly adopted, it went through a long phase where parameter efficiency was theoretically better, yet training was unstable, routing collapsed, and inference scheduling was complex. Loop faces a very similar situation: weight tying in controlled settings can indeed trade fewer parameters for deeper effective computation, but once you enter the scaling-up phase, all these problems follow close behind.
## What Kind of Loop Is Worth Scaling Up?
Scaling up Loop requires clearing three gates simultaneously. Depth stability determines whether the model can be trained at all; inference efficiency determines whether it can be served at acceptable cost; per-FLOPs effectiveness determines whether it deserves the compute budget in the first place. These three problems are tightly coupled: without stable deep loops, there is no way to push recurrence far enough to measure FLOPs effectiveness; without efficient inference, a well-trained looped LLM remains impractical to deploy; and without demonstrated FLOPs returns, solving the first two problems yields an architecture with no economic justification. We examine each in turn.
Depth Stability
Deeper loops eventually lead to excessively deep network structures, creating a situation where "the loop itself works fine, but the network is simply too deep." When the model passes through the same set of blocks repeatedly, the hidden state forms a dynamical system. If this system is unstable across recurrent depth, deeper loops can cause residual explosion, loss spikes, and extreme sensitivity to hyperparameters. One viable approach is from the @hayden_prairie Parcae work to directly apply the lens of RNNs to loops, solving Loop problems with the methods that once solved RNN problems - the stability challenges of Looped Transformers closely resemble those of traditional RNNs and SSMs.
In an RNN, if the spectral radius of the recurrent matrix is too large, the hidden state explodes over time; in an SSM, if the eigenvalues of the state transition matrix are uncontrolled, the system shifts from bounded dynamics to divergent dynamics. The Looped LLMs simply moves this problem from the sequence dimension to the depth dimension. Deeper loops mean longer unrolls; if the recurrent transition is unconstrained, the model easily accumulates error along the depth axis, amplifies residual norms, and triggers loss spikes. Of course, there are also newer approaches to solving this: instead of viewing the loop as an RNN, we can treat it as depth side Attention, which aligns well with Kimi's Attention Residuals and Mixture of Depth Attention.
Inference Efficiency
On the inference side, Looped LLM's bottleneck is very direct: in vanilla looping, each token must serially run through multiple loop iterations, and both latency and KV cache can grow with loop count. If this problem is handled poorly, Loop's parameter efficiency is easily offset by serving cost.
A natural idea here is to adopt a diffusion-forcing-like architecture. In one sentence: the model simultaneously predicts the next token and computes the next loop. Examples include the Parallel Looped Transformer and the parallel sampler for Looped LLM.
Solving the looped LLM inference bottleneck with Diffusion Forcing. Figure (a) shows vanilla sequential looping, while Figure (b) reveals the trick to parallelism: a diagonal compute pattern (orange boxes). In a single cycle, the model deepens older tokens (like step 5 on the left) while simultaneously processing the shallow layers of new tokens.
this approach gives advantages in decoding, but prefilling requires combining with a YOCO scheme (later loops share the KV cache of earlier loops, killing two birds with one stone by reducing KV cache and improving prefilling efficiency). Universal YOCO is a representative case along these lines.
Training FLOPs Effectiveness
The most important scale-up metric in training, and arguably the most critical evaluation criterion for Looped LLMs, is per-FLOPs effectiveness. In an iso-parameter setting, Looped LLMs tends to show advantages more easily, because it reuses the same set of parameters, increasing effective depth. But this is insufficient. In real training and deployment, what we care about is: given one unit of FLOPs, should we buy more tokens, a larger model, longer context, more CoT tokens, or more latent loop depth? In practice, most mainstream Looped LLMs have failed to address this properly, because nearly all comparisons use iso-parameter settings, and when you are scaling up, every FLOP is real money spent. Recently, Parcae and Sparse Growing Transformer are both showing the improvement under iso-flops setting, though still in a relatively small scale, it shows with proper design, we can achieve better performance on iso-FLOPs settings.
iso-FLOPs settings on Parcae paper.
Of course, for pretraining loops from scratch, iso-FLOPs effectiveness would be the holy grail. But there is a more practical route: start with a strong base model, then gradually inject recurrent depth through continue pre-training, annealing, upcycling, depth curriculum, and similar techniques. A good example is Retrofitted Recurrence, where they apply continue pre-training on a non-looped LLM and transfer it to a looped LLM. Ouro's training pipeline itself includes stages such as stable training, CT annealing, LongCT, mid-training, upcycling, and Reasoning SFT, illustrating that the introduction of loop depth is more like a training process question. By adding loops in the later stages, you can improve model performance through a small amount of cheap training FLOPs: given a fixed training FLOPs budget, how much compute should be spent on loop depth? This may be the approach that most directly solves the problem, though it is less elegant. We can still hope for a direct solution to the first problem.
## If we do not have 100K B200 for loop, what should we do?
Closing all three gaps in one shot would be the dream experiment, but compute budgets have a way of waking people up from dreams. The more productive framing is: which pieces of the puzzle can we solve today with modest resources, so that when the big run eventually happens, we already know what to build?
FLOPs-effective Loop Scaling Law
If Looped LLM is to become a scaling axis, it needs to enter the efficient frontier under fixed compute. This requires building a more complete Looped LLM scaling law. Traditional scaling laws study the optimal ratio of parameters to data; Loop scaling laws need to incorporate recurrent depth. Perhaps we also need to compare CoT and Loop: CoT spends compute in text space, while Loop spends compute in latent space. The marginal returns of each may vary by task. GraphWalk, code execution, and formal reasoning may be better suited to latent recurrence; tasks that require interpretable intermediate steps or external tool calls may still be better served by token-level reasoning. A more realistic system would likely use both: first use loops to strengthen internal state, then use a small number of explicit tokens to expose key reasoning.
Attention-level State Update: Implicit Loop
Loop does not have to appear only in full-stack weight tying. Another path is to introduce stronger state updates at the attention or memory mechanism level.
Work like DeltaFormer offers a relevant perspective: starting from kernel functions and the delta rule, it interprets the delta rule as an update mechanism for associative memory and attempts to give Transformers an implicit state space. Such mechanisms could enhance state-tracking capabilities at the attention level. This means future Loop-like models might involve more than a set of transformer blocks being called repeatedly; they might also form some kind of recurrent memory update inside attention itself. Explicit loops unfold along the depth dimension, while Delta-style attention leans more toward updating state along the sequence/memory dimension. From this angle, Loop and attention improvements are unlikely to be in competition. The more probable direction is to introduce stronger attention-level state updates within a more stable recurrent-depth architecture, letting the model simultaneously benefit from depth recurrence and memory recurrence.
CPT / Upcycling / Depth Curriculum
From a training practice perspective, pretraining a very deep looped LLM from scratch may not be the most economical route. A more realistic path is to first train a strong base model, then gradually introduce recurrent depth during continual pretraining or mid-training stages.
This can take several forms. The first is loop-CPT: using loop depth primarily for data in math, code, graph-like reasoning, and long-context state tracking. The second is depth curriculum: start with shallow recurrence to stabilize training, then gradually increase recurrent step, letting the model learn to continue refinement at deeper steps. The third is upcycling: converting some blocks of an existing fixed-depth transformer into recurrent blocks, especially in a middle-looped architecture where the prelude handles embedding, the recurrent middle handles iterative updates, and the coda handles output. Parcae's observations on training depth and test-time scaling ceilings also make depth curriculum more compelling.
Another important direction is distillation. High-recurrence models may produce higher quality output, but they are more expensive to serve. A viable route is to train a deep loop, then distill it into a shallow loop or parallelized structure, preserving as much of the representational capacity from deep recurrence as possible while reducing inference-time steps. This corresponds precisely to the future problem raised by Parcae: how to reduce test-time steps while maintaining quality.
Inference Infrastructure
Even with the various inference optimizations mentioned above (such as diffusion-forcing-style parallel decoding and YOCO KV cache sharing), Looped LLMs still face unique system-level challenges during inference. The core difficulty lies at two levels. First, continuous batching must handle both real tokens and latent loop iterations simultaneously: different requests may be at different loop depths, and the scheduler must coordinate these two heterogeneous types of compute units within a batch. Second, KV cache lifecycle management becomes significantly more complex: whether each loop layer needs an independent KV cache, when intermediate loop caches can be released, and how to remain compatible with existing frameworks like PagedAttention are all engineering problems that require dedicated design. We believe these challenges, though serious, are solvable. Personally, I see them as analogous to the EP/TP scheduling problems MoE faced in its early days: the kind of systems engineering challenge that can be progressively conquered once it receives serious attention
## Closing the Loop
So... returning to the original question: does Mythos actually use Loop? This may not be what matters most. What truly matters is that Loop, as an inductive bias, theoretically offers elegant properties such as parameter-compute decoupling, state-tracking enhancement, and test-time adaptive depth, while also demonstrating viable engineering paths in practice. From controlling depth stability, to optimizing inference parallelism, to the gradual introduction via CPT/upcycling, the engineering ecosystem around Loop is taking shape. We hope that reusable depth has the potential to become the next scaling axis after parameter count, data scale, and context length.
Thanks for @vhjf36495872 bring this idea to me to write the blog, and also thanks to @Zixin_Wen and @zzZixuanWang for reading drafts of this post.
## 相关链接
- [Rui-Jie Zhu](https://x.com/RidgerZhu)
- [@RidgerZhu](https://x.com/RidgerZhu)
- [6.2K](https://x.com/RidgerZhu/status/2046736781035618602/analytics)
- [@GeZhang86038849](https://x.com/@GeZhang86038849)
- [@ChrisHayduk](https://x.com/@ChrisHayduk)
- [@hayden_prairie](https://x.com/@hayden_prairie)
- [Parcae](https://arxiv.org/abs/2604.12946)
- [Attention Residuals](https://arxiv.org/abs/2603.15031)
- [Mixture of Depth Attention](https://arxiv.org/abs/2603.15619)
- [Parallel Looped Transformer](https://arxiv.org/abs/2510.24824)
- [parallel sampler](https://arxiv.org/abs/2510.14961v1)
- [Universal YOCO](https://arxiv.org/abs/2604.01220)
- [Parcae](https://arxiv.org/abs/2604.12946)
- [Sparse Growing Transformer](https://arxiv.org/abs/2603.23998)
- [Retrofitted Recurrence](https://arxiv.org/abs/2511.07384)
- [DeltaFormer](https://openreview.net/forum?id=GSE3oaiDL2)
- [@vhjf36495872](https://x.com/@vhjf36495872)
- [@Zixin_Wen](https://x.com/@Zixin_Wen)
- [@zzZixuanWang](https://x.com/@zzZixuanWang)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [7:44 AM · Apr 22, 2026](https://x.com/RidgerZhu/status/2046736781035618602)
- [6,227 Views](https://x.com/RidgerZhu/status/2046736781035618602/analytics)
- [View quotes](https://x.com/RidgerZhu/status/2046736781035618602/quotes)
---
*导出时间: 2026/4/22 14:04:13*