# WTF is a World Model?
**作者**: Vai Viswanathan
**日期**: 2026-05-01T11:36:22.000Z
**来源**: [https://x.com/vai_viswanathan/status/2050177504392998932](https://x.com/vai_viswanathan/status/2050177504392998932)
---
"World model" is the phrase of the moment. Fei-Fei Li's World Labs raised $230M out of stealth and followed it with a $1B round. Yann LeCun left Meta to start AMI Labs, which raised $1.03B at a $3.5B valuation to build world models from the ground up. Google DeepMind shipped Genie 3. OpenAI pitched Sora as a "world simulator." NVIDIA released Cosmos. Dreamer 4 solved Minecraft's Diamond Challenge from offline data alone. Every robotics lab worth its salt now has a world model paper.
The problem is that when you actually try to pin down what a world model is, you get five different answers depending on who you ask.
This article is an attempt to unpack the term. First, what is a world model? Second, what are the current approaches? And third — the part that matters if you build robots — where do world models actually earn their keep in a modern robotics stack?
## What Are World Models?
The most important thing to understand up front: world modeling is a problem statement, not a model architecture.
The closest analogy here is SLAM — Simultaneous Localization and Mapping. SLAM isn't a specific algorithm. It's a problem: given a stream of sensor data, figure out where you are while building a map of your surroundings. There are dozens of SLAM approaches. Filter-based methods like EKF-SLAM and FastSLAM. Graph-based methods built on optimization backends like GTSAM or Ceres Solver. Different sensor modalities — cameras, LiDAR, IMUs, or fusions of all three. Even within visual SLAM, you have direct methods that work on raw pixel intensities and feature-based methods that extract keypoints like ORB or SIFT.
World models are similar. "Build a world model" is the goal. How you build one is wide open.
LeCun's Definition
The cleanest formal definition comes from Yann LeCun (source):
Given:
- x(t): an observation
- s(t): a previous estimate of the state of the world
- a(t): an action proposal
- z(t): a latent variable proposal
A world model computes:
- h(t) = Enc(x(t)), the representation
- s(t+1) = Pred(h(t), s(t), z(t), a(t)), the prediction
Where:
- Enc() is an encoder (a trainable deterministic function, e.g. a neural net)
- Pred() is a hidden state predictor (also a trainable deterministic function)
- z(t) represents the unknown information that would allow us to predict exactly what happens. It must be sampled from a distribution or varied over a set. It parameterizes the set (or distribution) of plausible predictions.
Let's ground this formal definition with an example.
A Worked Example: A Self-Driving Car
Imagine a self-driving car parked in an empty lot with a single forward-facing camera. It has two controls: a steering angle and an accelerator.
- x(t) — the initial image the camera sees. Asphalt, a horizon line, maybe a fence.
- s(t) — the car's internal state: position, velocity, heading, current steering angle. It starts at (0, 0, 0) with zero velocity.
- a(t) — the action the car decides to take. Let's say we're doing donuts: full steering lock to the left (-1) and full throttle (+1).
- z(t) — the unobservable stuff. How much grip the tires have. A gust of wind. Invisible oil on the pavement. Sensor noise. Anything in the real world that would affect the outcome but that you can't read off the current state. A deterministic model ignores z(t). A probabilistic one samples it from a distribution, giving you a set of plausible futures rather than a single prediction.
Given these, the world model predicts s(t+1) — the next state of the car.
IMPORTANT: this is not a prediction of the next image. It's a prediction of the underlying state. The car's new position, velocity, and heading after a fraction of a second of flooring it at full lock.
The Decoder Extension
If you do want to generate the next image, you add a decoder step (suggested by Kevin Murphy):
x(t+1) = Dec(s(t+1))
This distinction matters more than it looks. LeCun deliberately separates prediction from rendering. The world model's job is to reason about state. Rendering that state back into pixels is a separate, optional step. As we'll see, this separation is exactly where the main schools of world modeling disagree.
## What Are the Current Approaches?
Today there are three dominant paradigms for building modern world models. Each takes a different stance on what to compress, what to predict, and whether to render.
1. Generative World Models
The headline-grabbing category. Models like Sora, Veo, Genie 3, GameNGen, and World Labs' RTFM learn to predict the next frame of video given a past context and an action input.
Source: https://deepmind.google/models/genie/
Architecturally they're usually autoregressive transformers or video diffusion models, often combined. You give the model a starting image (or text prompt), optionally a stream of actions, and it predicts the world forward frame by frame.
The framing is straightforward: if the model can generate the future accurately, it must on some level understand the world. Play a game inside Genie and the environment seems to have consistent physics and object permanence. Drive a virtual car and the road responds.
But these models also come with well-known failure modes:
- Autoregressive drift — small errors compound over long rollouts
- Hallucination — objects appear and disappear when occluded
- Physical implausibility — water flows uphill, rigid objects deform, lighting is inconsistent
- Memory loss — the world changes when you look away
How this fits LeCun's definition: Generative models collapse state and observation into the same thing — they predict directly in observation (pixel) space, skipping the abstract state. LeCun would argue this is an impoverished instantiation: almost all of the model's capacity goes into rendering, leaving less for the parts that matter for reasoning and control.
2. Latent World Models
Instead of predicting pixels, latent world models predict in a compressed, abstract representation space. Two subfamilies dominate, and they're philosophically opposed in a way that's worth taking seriously.
Latent Dynamics Models — Dreamer / DreamerV3 / Dreamer 4, PlaNet, TD-MPC. These grew out of model-based reinforcement learning. A recurrent state-space model (RSSM) learns to predict future latent states conditioned on actions. Crucially, these models are typically trained with reconstruction loss — a decoder forces the latent to retain enough information to regenerate the observation — alongside reward prediction so the latent is grounded in what matters for the task. A policy and value function are then trained on trajectories imagined inside this latent space.
JEPA — Joint Embedding Predictive Architecture — I-JEPA, V-JEPA, V-JEPA 2, and most recently LeWorldModel (LeWM) from AMI Labs and collaborators. A vision encoder produces embeddings of the current state; a predictor predicts the embeddings of future or masked-out regions. No reward. No pixel reconstruction. The model is trained purely via self-supervised prediction in representation space.
Source: "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning"
While both operate in the latent space, they differ in four important ways:
- Training signal. Latent Dynamics Models are trained end-to-end with a reconstruction loss (via a decoder) and usually a reward signal; the latent state is shaped by what's needed to regenerate pixels and predict returns. JEPA is trained with a single self-supervised prediction-in-embedding-space objective — no decoder, no reward.
- Stance on generation. Latent Dynamics Models embrace generation; Dreamer 4, for instance, can imagine full video rollouts from the latent state. JEPA rejects generation as a training objective entirely. LeCun's argument is that the world contains inherently unpredictable details (leaves on a tree, noise in a sensor) and forcing the model to predict them at the pixel level wastes capacity on things that fundamentally cannot be predicted.
- Task specificity. Latent Dynamics Models are typically trained for a specific environment or task, with the reward shaping the latent. JEPA is trained in a task-agnostic, reward-free way on video and then adapted to downstream control via MPC or a small action-conditioned predictor.
- How control happens. Latent Dynamics Models usually train an explicit actor-critic inside the world model. JEPA-based controllers typically do MPC at test time: sample candidate action sequences, simulate forward in embedding space, and pick the one whose predicted embedding lands closest to a goal embedding.
How this fits LeCun's definition: This is the best fit for the formal definition, unsurprisingly. Latent Dynamics Models have a strong Enc(), a strong Pred() via RSSM, stochastic sampling z(t), and a Dec() used during training but often dropped at plan time. JEPA is the purest fit — Dec() is absent by design, which LeCun argues is a feature, not a bug.
3. 3D Neural World Models
The third category commits to 3D geometry as the representation. NeRF pioneered this — an implicit neural function that maps 3D coordinates to color and density, rendered via volumetric ray marching. 3D Gaussian Splatting has largely replaced NeRF in the last two years by swapping the implicit function for an explicit set of millions of small semitransparent gaussians, which render dramatically faster.
World Labs is the flagship company here. Their product Marble generates persistent, explorable 3D worlds from text, images, or video. You export the result as a Gaussian splat file, a mesh with collider geometry, or a video — formats you can drop into Unreal, Unity, Blender, or NVIDIA's Isaac Sim.
How this fits LeCun's definition: Partial fit. 3D Neural models nail the representation half — Enc() is strong, the 3D structure is an excellent state representation. But Pred() is typically weak. A static NeRF or splat doesn't predict anything; it just renders what it was trained to render. Dynamic extensions exist, but they're not the core design. In LeCun terms, these are better described as world representations than world models — and World Labs' push toward RTFM is an explicit attempt to close that gap.
Summary
Two columns do most of the work here. The Pred() column reveals which approaches are actually doing forward prediction in a meaningful sense. Generative models predict — but they conflate s(t+1) and x(t+1) into a single pixel output, skipping the abstract state entirely. NeRF and 3DGS don't really predict at all; they render a fixed scene from new viewpoints. Only Latent Dynamics Models and JEPA have clean, abstract Pred() over actions.
The Dec() column is the real philosophical fault line. Generative and 3D Neural models put most of their capacity into rendering pixels. Latent Dynamics Models use a decoder during training and can optionally drop it at plan time. JEPA refuses a decoder outright. How much a world model invests in decoding back into pixels is arguably the biggest open design question in the field.
## How Are World Models Used in Robotics?
The practical question for anyone building robots: when does a world model actually earn its keep? The honest answer is that world models now show up in almost every phase of the robotics development lifecycle — before training, during training, after training, and at deployment. There are five distinct use cases.
1. Evaluation
Running a policy against a learned world model as a proxy for real-world testing.
This is the use case with the most obvious short-term payoff, because real robot evaluation is brutally slow. Setting up a task, running enough trials for statistical significance, resetting between episodes, and dealing with hardware drift can take days of wall-clock time per policy checkpoint. Teams routinely burn thousands of engineering hours on evaluation alone. A world model lets you run the same suite against hundreds of policy checkpoints in parallel on a cluster — and because the initial frames come from real robot cameras, the evaluation domain is much closer to deployment than a handcrafted simulator.
Examples: WorldGym (r = 0.78 correlation with real-world success rates, 3.3% mean error across VLAs), DreamDojo (r = 0.995 across policy checkpoints), and the Veo World Simulator for Gemini Robotics (validated across 1,600+ real trials).
A closely related use case — really an extension of evaluation — is safety red-teaming. Instead of asking "does this policy succeed?" you ask "what scenarios make this policy fail?" The Veo/Gemini paper is the canonical example: generative image editing drops adversarial objects, distractors, and safety-critical elements into scenes, then rolls the policy out to see whether it does something unsafe. No hardware risk, no staged demonstrations, and you can probe scenarios that would be impossible to set up physically (think infants crawling into a robot's workspace, or liquids on a surgical robot's instruments). This is likely the first place where world models deliver undeniable operational value — the alternative isn't just slower, it's impossible.
2. Direct Planning
Querying the world model at runtime to choose actions.
Instead of learning a policy that maps observations directly to actions, the robot proposes candidate action sequences, simulates them inside the world model, and executes whichever one looks best. The world model is the planner.
The JEPA line is the flagship example. V-JEPA 2-AC uses Model Predictive Control in embedding space — given a goal image, pick actions that minimize predicted-embedding-to-goal distance, then re-plan at the next step. LeWorldModel pushes this further: a compact 15M-parameter JEPA that plans up to 48× faster than foundation-model world models while remaining competitive on manipulation benchmarks. DreamDojo uses a similar propose-simulate-score-execute loop, sampling candidate action chunks from a policy ensemble and picking the best via an external value model, yielding roughly 2× improvement over uniform sampling.
3. Training Gym (RL in Imagination)
Training policies inside a world model. This is the direct descendant of Sutton's 1991 Dyna architecture: learn a model, then use it to generate cheap simulated experience for policy updates.
Source: "World-Gymnast: Training Robots with Reinforcement Learning in a World Model"
3a. Training in the world model
The core idea is straightforward: roll the policy forward in the learned world model, score the resulting rollouts (either with a learned reward function or with a VLM judge), and use the scores to update the policy via RL. Real-world interaction budget drops by orders of magnitude.
This paradigm was first proven on physical robots by DayDreamer (Berkeley, 2022), which trained a quadruped to walk from scratch in one hour with no simulator. It scales up in Dreamer 4 (2025), which solved Minecraft's Diamond Challenge — 20,000+ actions from raw pixels — from offline data alone. On the robotics side, Robotic World Model (RWM) demonstrates zero-shot transfer to ANYmal D legged hardware, World4RL uses a diffusion world model to refine imitation-learned manipulation policies, World-Gymnast trains VLA policies against WorldGym with VLM reward and outperforms supervised finetuning by up to 18× on Bridge robots, and GigaBrain-0.5M* (RAMP) applies the same recipe at VLA-foundation-model scale.
Worth flagging: the modern versions of this are mostly used to refine pretrained policies, not to train from scratch. Bootstrapping from supervised finetuning on a base VLA is now standard, with the world model providing the RL signal on top. Pure from-scratch training in imagination — DayDreamer-style — is still mostly a locomotion-scale phenomenon.
3b. Iterative world model updates
If you stop here, the world model is static — pretrained once and used as a fixed gym. But the same real-world deployments used for evaluation produce trajectories that are exactly what the world model needs to improve. This gives you a closed loop: policy rollouts on real hardware → finetune the world model → better imagined rollouts → better policy updates → repeat.
This is textbook Dyna, 35 years later. World-Gymnast implements it explicitly, collecting real-robot trajectories during AutoEval evaluation and using them to finetune WorldGym for 120k steps between rounds. The authors explicitly cite Sutton 1991 and show that iterative refinement matters: trajectories from a Dyna-updated world model adhere to real-world behavior more closely than those from software simulators.
The implications are bigger than the specific paper. It means a production robotics system can improve its world model as a side effect of doing its normal evaluation work. Every evaluation run is also a data collection run for the model that powers the next round of training. The line between "offline pretraining" and "online deployment" starts to dissolve.
3c. Test-time training (TTT)
The third sub-theme pushes this further: do the policy update at deployment time, from a novel scene that the pretrained policy hasn't seen.
Start the robot in a new environment. Before executing any actions, finetune the policy via RL inside the world model — imagined rollouts only, no real-world interaction. Then deploy the specialized policy on the actual task.
The numbers from World-Gymnast are striking: on a "close the drawer" task with a novel initial frame, test-time training pushes success from 62% to 100%. No real-world rollouts, no additional demonstrations, no new training data — just a quick RL finetune inside the world model before the policy is allowed to touch the hardware. This is a qualitatively new capability. It means a robot can prepare for a task it hasn't seen by practicing in imagination.
Iterative updates and TTT together turn the training gym from a static pretraining environment into a continuous learning substrate. This is the part of the world model story that is least discussed and arguably most consequential for robotics.
4. Synthetic Data Generation
World model as a data factory.
Source: https://www.nvidia.com/en-us/ai/cosmos/
This is distinct from the Training Gym because there's no policy rollout in the loop, no reward signal, no RL. The world model generates trajectories offline — new viewpoints, new backgrounds, new objects, new lighting conditions — and those trajectories are used for standard supervised imitation learning.
GigaBrain-0 is the headline example: a VLA foundation model trained on about 1,000 hours of real robot data augmented with large quantities of world-model-generated data (video generation, real-to-real transfer, human-to-robot transfer, view transfer, sim-to-real transfer). Increasing the synthetic-data ratio monotonically improves generalization across appearance, placement, and viewpoint shifts.
NVIDIA Cosmos is the closest thing to a general-purpose platform for this. Cosmos is explicitly marketed as a world foundation model for physical AI, used to generate synthetic training data at scale for robotics and autonomous driving. It's now being used as a backbone in downstream work like Cosmos-Surg-dVRK (surgical policy evaluation) and as a world model component in broader VLA training pipelines.
World Labs' Marble → Isaac Sim pipeline is another flavor of this: generate a photorealistic kitchen or warehouse environment, export the splat and collider mesh, and feed it into a physics simulator for robot training. Weeks of manual environment curation compress into minutes.
5. World-Action Models (WAMs)
A unified architecture in which the world model is the policy.
Classical VLAs map observation + language → action. WAMs jointly predict the future video and future actions in a single model, training with aligned losses. The claim is that learning to predict how the world evolves under action gives much stronger physical priors than learning actions alone.
Examples: DreamZero (a 14B autoregressive diffusion transformer built on a video diffusion backbone; reports 2× generalization gain over state-of-the-art VLAs and hits real-time closed-loop control at 7 Hz), WorldVLA / RynnVLA-002, GigaWorld-Policy, Motus, and Cosmos-Policy.
Source: https://dreamzero0.github.io
The category also has its own emerging self-critique. Fast-WAM (2026) asks a pointed question: do WAMs actually need to generate video at test time, or is video prediction mainly useful as a training objective? Their answer leans toward the latter — a WAM that video-co-trains but skips future generation at inference runs 4× faster with competitive performance. Early evidence that the core value of the approach may be in representation learning rather than runtime simulation.
Where They Show Up in the Lifecycle
Step back and look at the five use cases together, and a pattern emerges. A world model isn't a single tool you reach for at one moment — it's a substrate that shows up across the entire robotics ML pipeline:
Before you train, the world model generates data. During training, it's the environment. After training, it's the grader and the adversary probing for safety failures. At deployment, it's the planner — or, increasingly, it is the policy. And in the iterative-update regime, the line between training and deployment dissolves entirely: the robot's experience in the world feeds back into the world model, which improves the substrate for the next training cycle.
That's a qualitatively different picture from classical robotics, where maps, simulators, dynamics models, and policies were separate artifacts built by separate teams. The trend is toward one learned substrate that does all of these jobs.
## Conclusion
What should be clear by now is how much variation hides inside the phrase "world model." Four quite different architectural approaches — generative, latent dynamics, JEPA, 3D neural — all claim the label, and they disagree on fundamentals: whether to render pixels, whether to use reward, whether to commit to 3D structure, whether the policy and the world model are separate or fused. On top of that sit five distinct robotics use cases — data generation, training gym, evaluation, planning, unified policy — each touching a different phase of the development lifecycle. "World model" is the problem statement; the space of solutions and applications is still rapidly expanding.
We're at the beginning, not the end, of this story. Evaluation pipelines that run overnight on a cluster instead of a week in a lab will let robotics teams iterate at fundamentally different speeds. Test-time training means a robot can prepare for a new task in imagination before touching hardware. Iterative world model updates mean every deployment improves the substrate for the next training cycle. World-Action Models blur the line between simulator and policy. These aren't incremental improvements — they're the kind of shifts that change what's possible.
As the field develops, we'll see which of these threads actually drive the breakthroughs in robotics — which approaches scale, which use cases become standard infrastructure, which combinations turn out to matter most. Robotics hasn't yet had the inflection moment that LLMs had for language, where a new model paradigm made previously impossible capabilities suddenly routine. World models are the most promising candidate for what that inflection looks like. The next few years will tell us whether the bet pays off.
## 相关链接
- [Vai Viswanathan](https://x.com/vai_viswanathan)
- [@vai_viswanathan](https://x.com/vai_viswanathan)
- [21K](https://x.com/vai_viswanathan/status/2050177504392998932/analytics)
- [$230M](https://x.com/search?q=%24230M&src=cashtag_click)
- [$1B](https://x.com/search?q=%241B&src=cashtag_click)
- [source](https://x.com/ylecun/status/1759933365241921817)
- [suggested by Kevin Murphy](https://x.com/sirbayes/status/1759987256994214341)
- [https://deepmind.google/models/genie/](https://deepmind.google/models/genie/)
- [https://www.nvidia.com/en-us/ai/cosmos/](https://www.nvidia.com/en-us/ai/cosmos/)
- [https://dreamzero0.github.io](https://dreamzero0.github.io/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [7:36 PM · May 1, 2026](https://x.com/vai_viswanathan/status/2050177504392998932)
- [21.5K Views](https://x.com/vai_viswanathan/status/2050177504392998932/analytics)
- [View quotes](https://x.com/vai_viswanathan/status/2050177504392998932/quotes)
---
*导出时间: 2026/5/2 09:24:03*
---
## 中文翻译
# 到底什么是世界模型？
**作者**: Vai Viswanathan
**日期**: 2026-05-01T11:36:22.000Z
**来源**: [https://x.com/vai_viswanathan/status/2050177504392998932](https://x.com/vai_viswanathan/status/2050177504392998932)
---
“世界模型”是当下的热词。李飞飞的 World Labs 在隐身模式下筹集了 2.3 亿美元，随后又完成了 10 亿美元融资。Yann LeCun 离开 Meta 创立了 AMI Labs，该公司以 35 亿美元的估值筹集了 10.3 亿美元，旨在从零开始构建世界模型。Google DeepMind 发布了 Genie 3。OpenAI 将 Sora 宣传为“世界模拟器”。NVIDIA 发布了 Cosmos。Dreamer 4 仅凭离线数据就解决了《我的世界》的钻石挑战。如今，稍有实力的机器人实验室都有关于世界模型的论文。
问题在于，当你试图确切定义“世界模型”是什么时，询问不同的人会得到五种不同的答案。
这篇文章旨在厘清这一术语。首先，什么是世界模型？其次，目前有哪些实现方法？第三——也是构建机器人的人最关心的部分——世界模型在现代机器人技术栈中究竟在哪些方面发挥着实际作用？
## 什么是世界模型？
首先需要理解的最重要的一点是：世界建模是一个问题陈述，而不是一种模型架构。
这里最贴切的类比是 SLAM——即时定位与地图构建。SLAM 不是一种特定的算法，它是一个问题：给定传感器数据流，在构建周围环境地图的同时弄清楚你身处的位置。SLAM 有几十种实现方法。基于滤波的方法，如 EKF-SLAM 和 FastSLAM。基于优化的方法，底层使用 GTSAM 或 Ceres Solver。不同的传感器模态——摄像头、激光雷达、IMU，或三者的融合。即便是在视觉 SLAM 领域，也有直接处理原始像素强度的直接法，以及提取 ORB 或 SIFT 等关键点的基于特征的方法。
世界模型与此类似。“构建一个世界模型”是目标。至于如何构建，则大有可为。
LeCun 的定义
最清晰的形式化定义来自 Yann LeCun（来源）：
给定：
- x(t): 一个观测值
- s(t): 世界状态的先前估计
- a(t): 一个动作提议
- z(t): 一个潜在变量提议
世界模型计算：
- h(t) = Enc(x(t))，即表征
- s(t+1) = Pred(h(t), s(t), z(t), a(t))，即预测
其中：
- Enc() 是一个编码器（一个可训练的确定性函数，例如神经网络）
- Pred() 是一个隐藏状态预测器（也是一个可训练的确定性函数）
- z(t) 代表那些能让我们精确预测接下来会发生什么的未知信息。它必须从一个分布中采样，或者在一个集合上进行变化。它对合理预测的集合（或分布）进行参数化。
让我们用一个例子来落地这个形式化定义。
一个实战案例：自动驾驶汽车
想象一辆停在空旷场地上的自动驾驶汽车，只有一个前置摄像头。它有两个控制装置：方向盘角度和油门。
- x(t) — 摄像头看到的初始图像。沥青路面，地平线，可能还有围栏。
- s(t) — 汽车的内部状态：位置、速度、航向、当前方向盘角度。它从零速度的 (0, 0, 0) 开始。
- a(t) — 汽车决定采取的动作。假设我们在玩漂移：方向盘向左打死 (-1)，油门踩到底 (+1)。
- z(t) — 不可观测的因素。轮胎的抓地力有多大。一阵风。路面上看不见的油污。传感器噪音。现实世界中任何会影响结果但你无法从当前状态中读取的信息。确定性模型会忽略 z(t)。而概率模型会从分布中对其进行采样，从而给你一组合理的未来结果，而不是单一的预测。
鉴于以上因素，世界模型会预测 s(t+1) — 汽车的下一个状态。
**重要提示：这不是对下一张图像的预测，而是对潜在状态的预测。** 即在全方向盘锁定和油门踩到底的状态下行驶零点几秒后，汽车的新位置、速度和航向。
解码器扩展
如果你确实想生成下一张图像，你需要添加一个解码器步骤（由 Kevin Murphy 提出）：
x(t+1) = Dec(s(t+1))
这种区别比看上去更重要。LeCun 刻意将预测与渲染分离开来。世界模型的工作是对状态进行推理。将状态渲染回像素是一个单独的、可选的步骤。正如我们将看到的，这种分离正是世界建模主要流派分歧的根源所在。
## 目前有哪些方法？
目前，构建现代世界模型有三种主导范式。每种范式在压缩内容、预测内容以及是否渲染方面都持不同立场。
1. 生成式世界模型
这是最能占据头条新闻的类别。像 Sora、Veo、Genie 3、GameNGen 以及 World Labs 的 RTFM 这样的模型，致力于在给定过去上下文和动作输入的情况下，预测视频的下一帧。
来源: https://deepmind.google/models/genie/
在架构上，它们通常是自回归 Transformer 或视频扩散模型，或者是两者的结合。你给模型一个起始图像（或文本提示），可选地输入一系列动作，它就会逐帧向前预测世界。
这种框架很直观：如果模型能准确地生成未来，那它在某种程度上肯定理解了世界。在 Genie 里玩游戏，环境似乎具有一致的物理规律和物体恒存性。驾驶虚拟汽车，路面会有相应的反应。
但这些模型也存在众所周知的失效模式：
- 自回归漂移 —— 在长序列推演中，小误差会累积叠加
- 幻觉 —— 物体在被遮挡时出现或消失
- 物理不合理 —— 水往高处流，刚性物体变形，光照不一致
- 记忆丢失 —— 当你移开视线时，世界发生了变化
这与 LeCun 定义的关系：生成式模型将状态和观测值混为一谈——它们直接在观测（像素）空间进行预测，跳过了抽象状态。LeCun 会认为这是一种贫瘠的实现：模型几乎所有的容量都用于渲染，留给推理和控制这些关键部分的能力自然就少了。
2. 潜在世界模型
潜在世界模型不是预测像素，而是在压缩的、抽象的表征空间中进行预测。这里有两个主要的子家族，它们在哲学上相互对立，值得我们认真对待。
潜在动力学模型 —— Dreamer / DreamerV3 / Dreamer 4、PlaNet、TD-MPC。这些模型源于基于模型的强化学习。循环状态空间模型（RSSM）学习在给定动作的条件下预测未来的潜在状态。关键在于，这些模型通常使用重建损失进行训练——通过解码器强制潜在状态保留足够的信息以重建观测值——并结合奖励预测，使潜在状态与任务相关的事物相联系。然后，策略和价值函数在这个潜在空间中想象的轨迹上进行训练。
JEPA —— 联合嵌入预测架构 —— I-JEPA、V-JEPA、V-JEPA 2，以及最近 AMI Labs 及其合作者推出的 LeWorldModel (LeWM)。视觉编码器生成当前状态的嵌入；预测器预测未来或被遮挡区域的嵌入。没有奖励。没有像素重建。该模型纯粹通过表征空间中的自监督预测进行训练。
来源: "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning"
虽然两者都在潜在空间运作，但它们在四个重要方面存在差异：
- 训练信号。潜在动力学模型通过重建损失（通过解码器）以及通常的奖励信号进行端到端训练；潜在状态是根据重建像素和预测回报所需的内容来塑造的。JEPA 使用单一的自监督“在嵌入空间预测”的目标进行训练——没有解码器，没有奖励。
- 对生成的立场。潜在动力学模型拥抱生成；例如 Dreamer 4 可以从潜在状态想象完整的视频推演。JEPA 完全拒绝将生成作为训练目标。LeCun 的论点是，世界包含本质上不可预测的细节（树上的叶子、传感器中的噪音），强迫模型在像素级别预测它们，会浪费模型容量去处理那些根本无法预测的事情。
- 任务特定性。潜在动力学模型通常针对特定环境或任务进行训练，奖励会塑造潜在状态。JEPA 以与任务无关、无奖励的方式在视频上进行训练，然后通过 MPC 或小型的动作条件预测器适应下游控制。
- 控制发生的方式。潜在动力学模型通常在世界模型内部训练一个显式的 Actor-Critic。基于 JEPA 的控制器通常在测试时进行 MPC：采样候选动作序列，在嵌入空间中向前模拟，并选择预测嵌入最接近目标嵌入的那个动作序列。
这与 LeCun 定义的关系：这是最符合形式化定义的一类，这并不意外。潜在动力学模型具有强大的 Enc()，通过 RSSM 实现的强大 Pred()，随机采样 z(t)，以及在训练期间使用但在规划时通常丢弃的 Dec()。JEPA 是最纯粹的匹配 —— Dec() 在设计上是不存在的，LeCun 认为这是一个特性，而不是缺陷。
3. 3D 神经世界模型
第三类致力于使用 3D 几何作为表征。NeRF 是这一领域的先驱——它是一个隐式神经函数，将 3D 坐标映射到颜色和密度，通过体光线行进进行渲染。在过去两年中，3D 高斯泼溅（3D Gaussian Splatting）在很大程度上取代了 NeRF，它用数百万个微小的半透明高斯的显式集合替换了隐式函数，渲染速度大幅提升。
World Labs 是这一领域的旗舰公司。他们的产品 Marble 可以根据文本、图像或视频生成持久的、可探索的 3D 世界。你可以将结果导出为高斯泼溅文件、带有碰撞几何体的网格或视频——这些格式可以直接拖入 Unreal、Unity、Blender 或 NVIDIA 的 Isaac Sim。
这与 LeCun 定义的关系：部分匹配。3D 神经模型在表征部分表现出色——Enc() 很强，3D 结构是一种极佳的状态表征。但 Pred() 通常很弱。静态 NeRF 或泼溅模型不预测任何东西；它们只是渲染它们被训练用来渲染的内容。虽然存在动态扩展，但它们不是核心设计。用 LeCun 的话来说，这些更准确地描述为世界表征，而不是世界模型 —— 而 World Labs 推进 RTFM 正是为了缩小这一差距的明确尝试。
总结
这里主要由两列发挥作用。Pred() 列揭示了哪些方法实际上在有意义的意义上进行了前向预测。生成式模型确实在预测——但它们将 s(t+1) 和 x(t+1) 混淆为单一的像素输出，完全跳过了抽象状态。NeRF 和 3DGS 并不真正进行预测；它们只是从新视角渲染固定场景。只有潜在动力学模型和 JEPA 具有针对动作的、清晰的、抽象的 Pred()。
Dec() 列是真正的哲学分界线。生成式和 3D 神经模型将大部分容量用于渲染像素。潜在动力学模型在训练期间使用解码器，并可以在规划时选择性地丢弃它。JEPA 则完全拒绝解码器。一个世界模型在解码回像素上投入多少，可以说是该领域目前最大的开放式设计问题。
## 世界模型如何用于机器人技术？
对于任何构建机器人的人来说，一个实际的问题是：世界模型在什么时候真正发挥其价值？诚实的回答是，世界模型现在几乎出现在机器人开发生命周期的每个阶段——训练前、训练中、训练后以及部署时。有五种不同的用例。
1. 评估
将针对学习到的世界模型运行策略作为真实世界测试的代理。
这是短期回报最明显的用例，因为真实机器人的评估极其缓慢。设置任务、运行足够多的试验以获得统计显著性、在回合之间重置以及处理硬件漂移，每个策略检查点可能需要数天的实际时间。团队通常仅在评估上就要花费数千小时的工程时间。世界模型使你能够在集群上并行运行数百个策略检查点的相同测试套件——而且因为初始帧来自真实机器人摄像头，评估领域比手工制作的模拟器更接近部署环境。
示例：WorldGym（与真实世界成功率的相关性 r = 0.78，跨 VLAs 的平均误差为 3.3%），DreamDojo（跨策略检查点的相关性 r = 0.995），以及 Gemini 机器人技术的 Veo World Simulator（在 1,600 多次真实试验中进行了验证）。
一个密切相关的用例——实际上是评估的延伸——是安全红队测试。不是问“这个策略会成功吗？”，而是问“什么场景会导致这个策略失败？”。Veo/Gemini 论文是这方面的典型例子：生成式图像编辑将对抗性物体、干扰物和安全关键元素放入场景中，然后推演策略，看它是否做出不安全的操作。没有硬件风险，无需 staged 示范，而且你可以探测那些在物理上无法设置的场景（想象一下婴儿爬进机器人的工作区，或者手术机器人器械上有液体）。这很可能是世界模型带来不可否认的运营价值的第一个地方——替代方案不仅更慢，而且根本不可能实现。
2. 直接规划
在运行时查询世界模型以选择动作。
机器人不再学习将观测值直接映射到动作的策略，而是提出候选动作序列，在世界模型中模拟它们，并执行看起来最好的那个。世界模型就是规划器。
JEPA 系列是这方面的旗舰例子。V-JEPA 2-AC 在嵌入空间中使用模型预测控制（MPC）——给定一个目标图像，选择能使“预测嵌入到目标的距离”最小化的动作，然后在下一步重新规划。LeWorldModel 将这一点更进一步：一个紧凑的 1500 万参数的 JEPA，其规划速度比基础模型世界模型快 48 倍，同时在操作基准测试中保持竞争力。DreamDojo 使用类似的“提议-模拟-评分-执行”循环，从策略集成中采样候选动作块，并通过外部价值模型选择最好的一个，其性能比均匀采样提高了约 2 倍。
3. 训练场
在世界模型内部训练策略。这是 Sutton 1991 年 Dyna 架构的直接后代：学习一个模型，然后利用它生成廉价的模拟体验用于策略更新。
来源: "World-Gymnast: Training Robots with Reinforcement Learning in a World Model"
3a. 在世界模型中训练
核心思想很直观：在学习到的世界模型中向前推演策略，对产生的轨迹进行评分（使用学习到的奖励函数或 VLM 判别器），并使用分数通过强化学习更新策略。真实世界交互的预算将减少几个数量级。
这种范式首次在物理机器人上得到验证是 DayDreamer（伯克利，2022 年），它在一个小时内从头开始训练了一条四足狗行走，无需模拟器。它在 Dreamer 4（2025 年）中扩展规模，仅凭离线数据就解决了《我的世界》的钻石挑战 —— 20,000 多个动作，全部来自原始像素。在机器人方面，Robotic World Model (RWM) 展示了对 ANYmal D 四足硬件的零样本迁移，World4RL 使用扩散世界模型来精炼模仿学习的操作策略，World-Gymnast 使用 VLM 奖励针对 WorldGym 训练 VLA 策略并优于……