# A Functional Taxonomy of World Models
**作者**: Fei-Fei Li
**日期**: 2026-06-03T18:57:10.000Z
**来源**: [https://x.com/drfeifei/status/2062247238143996275](https://x.com/drfeifei/status/2062247238143996275)
---
> “The world is everything that is the case.”
> — Ludwig Wittgenstein, Tractatus Logico-Philosophicus, 1921
## The world is not made of words.
In an earlier essay, we argued that spatial intelligence is AI’s next frontier and that world models are the path to it. Here, the World Labs team and I want to go one level deeper: of the many things now being built and called ‘world models,’ which functional pieces actually compose that capacity — and what is each one for?
Language models have given machines an extraordinary command of concepts, vocabulary, and reasoning, but the physical world, virtual or real, runs on a different substrate. Where language models learn the statistical structure of text, world models learn the statistical structure of space and time: how light falls on a surface, how a garden looks from an angle no camera has captured, how objects respond to force and follow the laws of physics.
That makes “world model” one of the most important and most overloaded terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI each claim to be building world models, and each means something quite different. A video model that produces gorgeous but physically impossible flames, a language model improvising a playable game, and a physics engine that faithfully simulates combustion all go by the same name.
The ancient Greeks could never agree on what the world was made of, whether fire, water, or indivisible atoms, because “world” was never a single thing. It was always a stand-in for whatever totality a given thinker needed to reason about. AI has inherited the same problem, at exactly the moment when the field needs precision.
## The loop beneath the taxonomy
Cutting through that confusion starts with a diagram older than any of the technology in question. Reinforcement learning textbooks, including the canonical Sutton and Barto, have used a version of the same picture for decades to describe how an agent interacts with a world. The formal name for this picture is the partially observable Markov decision process, or POMDP, and the original definition of the term “world model” belongs to that tradition.
An agent, which can be a person, a robot, or a software system, takes actions. Those actions affect the state of the world. The agent never sees the state directly. What reaches the agent are observations: the photons that fall on a retina, the readings from a sensor, and the pixels in a video frame. New observations inform new actions, and the loop continues.
The word “state” needs unpacking, because the meaning shifts from field to field. This is not the chemist’s state, the difference between solid, liquid, and gas. This is the physicist’s and roboticist’s state: a complete description of what is happening in the world at a given moment, including every object, every position, every velocity, every property. State is the underlying reality of the world; complete in principle, but never directly visible to any agent inside it. Observations are an agent’s partial view of that reality. Actions are what the agent does in response.
This loop — agent to action to state to observation and back — is the structure that gave the modern term “world model” its technical meaning. The phrase itself is older, traced to Kenneth Craik’s 1943 proposal that minds reason by running “small-scale models” of reality, and carried into neural networks by the late 1980s and early 1990s. And the loop also explains what people mean by the term today. The different things now being called world models are in fact different projections of this same loop. Each one outputs a different piece of it.
## Three functions of a world model
The first kind of world model is a renderer. A renderer outputs observations in the form of pixels meant for human eyes, and the quality that matters most is visual fidelity. A video model that turns a text prompt into a cinematic drone shot is a renderer. So is an interactive system like Google’s Genie 3, or World Labs’ own RTFM, where the model generates frames in real time conditioned on user input. The model carries no explicit understanding of three-dimensional structure. It produces what a viewer would see, not what is. The buildings in the drone shot may look flawless from above, but try to drive through the city below and they fall apart.
The second kind is a simulator. A simulator outputs state: a geometrically, physically or dynamically faithful representation of the world that humans and computer programs can both compute on and interact with. Where the renderer’s contract is purely visual, the simulator’s contract is structural, demanding geometry that holds up under inspection, physics that respects Newton’s laws, and dynamics that behave the way the world needs to behave given the laws of physics. A simulator serves two consumers at once. Human professionals such as architects, designers, filmmakers, and game developers need accuracy beyond visual plausibility. Computer programs such as reinforcement learning agents, robot controllers, and autonomous vehicles use simulators as training grounds where they can interact with the world at scale, testing scenarios that would be dangerous, expensive, or impossible to run in reality.
The third kind is a planner. A planner outputs actions. Given an observation and a goal, a planner answers the question of what the agent should do next. This is, in many ways, the inverse of the renderer. Where a renderer takes actions as input and produces observations, a planner takes observations as input and produces actions, closing the perception-action loop. Vision-Language-Action models, model-based systems, and the new wave of World Action Models are all attempts at planners: systems that can decide what a robot should do in an unstructured world.
These three categories describe most of what is actually shipping today, and the distinction between them is useful in practice. The categories are not, however, fundamentally separate. The same underlying knowledge of how the world works—geometry, physics, dynamics—sits beneath all of them. A model that can render a cup from any angle ought, in principle, to be able to simulate what happens when the cup is pushed and plan a hand to pick the cup up. Increasingly, the most interesting research deliberately blurs the boundaries between the three.
## Why simulation is the linchpin
Of the three categories, the simulator gets the least public attention, and is the most consequential of the three. This essay addresses this asymmetry.
The renderer is by far the most commercially mature. A number of image- or text-to-video products are expanding in the consumer or enterprise markets rapidly. Google’s Nano Banana model has put renderer-quality image generation in the hands of potentially hundreds of millions of users. The technology is real, and the markets are real. Yet renderers optimize for visual plausibility rather than physical accuracy, and that ceiling matters. Their outputs are beautiful, but they cannot be trusted to design a building or train a robot.
The planner is the most intriguing and the most nascent, closely connected to the rapidly evolving field of robotic learning. The field has produced robotic demos in the last two years that look impressive in videos, but candor is required about what those demos actually show. Almost all have been confined to heavily constrained laboratory setups, with narrow object sets and short task horizons. None have been validated at the complexity, variability, or duration that real-world deployment demands. The gap between a compelling demo reel and a robot that reliably works in a kitchen, a warehouse, or an operating room remains vast. The commercial bets are nonetheless substantial. A wave of well-funded entrants is racing to ship general-purpose planning systems, while the largest infrastructure players are positioning planning atop broader simulation stacks. A robot that can plan is a robot that can work, and the entire industry is racing to be the one that gets there first.
Simulation is the bridge between the two. If language is an abstraction of the world and pixels are a projection of it, then geometry, physics, and dynamics are the world itself. A simulator must work at that level: the structural backbone from which both visual appearance (for renderers) and action consequences (for planners) can be derived.
A model that masters simulation can project its understanding into pixels for human consumption, and into action predictions for embodied agents. A model that masters only rendering, or only planning, cannot do either. The commercial surface area is enormous. NVIDIA’s Omniverse alone targets what the company estimates as more than a trillion dollars of addressable market in factories, warehouses, supply chains, and digital twins. Robotics training, autonomous vehicle testing, architectural visualization, engineering, and drug discovery all depend on something simulation-shaped.
The hardest open problems in the field live there too. Three-dimensional data with explicit geometry, material properties, and physical annotations is orders of magnitude scarcer than the internet video that renderers train on. The sim-to-real gap, which is the difference between how things behave in simulation and how they behave in reality, persists. Generative simulators introduce a new risk on top of that: AI-generated geometry can look correct while containing self-intersections or wrong scale that produce nonsensical physics. Multi-physics simulation at scale, where rigid bodies, deformable objects, fluids, and cloth all interact, remains orders of magnitude more expensive than single-domain simulation.
At World Labs, Marble is our first move into this territory. It takes multimodal prompts (text, image, video, or spatial sketch) and generates explorable 3D environments, outputting Gaussian splats for visual exploration alongside collision meshes a physics engine can operate on. But Marble is only the first chapter of a much longer arc being written across the field as the lines between rendering, simulation, and planning begin to collapse.
## Where the boundaries are collapsing and what comes next
But more is to come. The most important pattern in the field right now is that the three categories are starting to blend into one another. The shared insight is that the knowledge required to render a world, simulate it, and act in it is largely the same. Continuing the earlier example, a model that truly understands how a cup sits on a table (its geometry, material properties, response to force, etc.) should be able to render that cup from any angle, simulate what happens when the cup is pushed, and plan for a hand to pick the cup up. The three categories are three projections of a single underlying understanding.
For example: a small but growing number of recent work from various robotics labs have demonstrated that—at least conceptually—a pretrained video renderer can be used as the backbone for joint world-and-action prediction, suggesting a bridge between the renderer and the planner by letting one model imagine what will happen and what to do. World Labs’ Marble already outputs Gaussian splats and collision meshes from a single model, dissolving the boundary between the renderer and the simulator. Every level is moving from passive output to interactive system, with renderers becoming action-conditioned, simulators generating worlds that are more controllable and editable, and planners deliberating rather than just reacting.
The logical endpoint is a unified world model: one foundation model that can render photorealistic views, produce physically accurate structure, and plan action sequences, switching between output modalities depending on what the downstream consumer needs. We will still face a number of daunting challenges. The data picture is uneven, with renderers awash in internet video while simulators and planners face acute shortages of 3D assets and robot demonstrations. Optimizing for visual beauty can sacrifice the precision a robot or a high-fidelity simulation needs. Reconciling these tensions inside a single architecture is the defining open problem in world model research today, and this is what World Labs sets out to do as we continue to evolve Marble.
The direction, however, is clear. The same bet the field has been making since the late 1980s — that a sufficiently rich model of the world is all that any agent needs to see worlds, build them, and act in them — is the bet now driving an entire generation of research. What gives that “big bet” weight is the convergence already underway: three threads, each already driving and shaping multi-billion-dollar industries on its own, that began as separate research programs are starting to behave like one. Taken together, as the boundaries between them collapse, they will reshape something larger: the relationship between machine intelligence and the physical world it inhabits - the long arc of spatial intelligence.
Language gave machines a way to talk about that world. World models are how machines will finally come to understand, imagine, reason and interact with it.
## 相关链接
- [Fei-Fei Li](https://x.com/drfeifei)
- [@drfeifei](https://x.com/drfeifei)
- [167K](https://x.com/drfeifei/status/2062247238143996275/analytics)
- [earlier essay](https://drfeifei.substack.com/p/from-words-to-worlds-spatial-intelligence)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [2:57 AM · Jun 4, 2026](https://x.com/drfeifei/status/2062247238143996275)
- [167.9K Views](https://x.com/drfeifei/status/2062247238143996275/analytics)
- [View quotes](https://x.com/drfeifei/status/2062247238143996275/quotes)
---
*导出时间: 2026/6/4 09:41:26*
---
## 中文翻译
# 世界模型的功能分类法
**作者**: 李飞飞
**日期**: 2026-06-03T18:57:10.000Z
**来源**: [https://x.com/drfeifei/status/2062247238143996275](https://x.com/drfeifei/status/2062247238143996275)
---
> “世界是一切发生的事情。”
> — 路德维希·维特根斯坦，《逻辑哲学论》，1921年
## 世界不是由词语构成的。
在之前的一篇文章中，我们提出空间智能是人工智能的下一个前沿，而世界模型是实现这一目标的路径。在这里，World Labs 团队和我希望再深入一层：在当下众多被称为“世界模型”的构建物中，究竟有哪些功能要素真正构成了这种能力——以及每一项要素又是为了什么？
语言模型赋予了机器对概念、词汇和推理的非凡掌控力，但物理世界——无论是虚拟的还是现实的——是运行在不同的基质上的。语言模型学习的是文本的统计结构，而世界模型学习的是空间和时间的统计结构：光线如何落在表面上，从未被任何相机捕捉过的角度看花园是什么样子，物体如何响应外力并遵循物理定律。
这使得“世界模型”成为当今 AI 领域最重要但也最被滥用的术语之一。计算机视觉、机器人技术、强化学习和生成式 AI 都声称自己在构建世界模型，但各自的含义大相径庭。一个能生成华丽但在物理上不可能的火焰的视频模型、一个即兴创作出可玩游戏的 language model，以及一个忠实地模拟燃烧过程的物理引擎，都共用同一个名字。
古希腊人从未就世界由什么构成达成一致——无论是火、水还是不可分割的原子——因为“世界”从来就不是单一的事物。它始终是任何特定思想家为了进行推理所需的“整体”的代名词。在恰恰需要精度的时刻，AI 继承了同样的问题。
## 分类法之下的循环
理清这种困惑，要从一张比上述所有技术都更为久远的图表说起。包括经典的 Sutton and Barto 在内的强化学习教科书，几十年来一直使用同一张图表的变体来描述智能体如何与世界互动。这张图表的正式名称是“部分可观察马尔可夫决策过程”，而“世界模型”一词的原始定义正属于这一传统。
一个智能体——可以是人、机器人或软件系统——采取行动。这些行动会影响世界的状态。智能体永远无法直接看到状态。到达智能体处的是观察：落在视网膜上的光子、传感器的读数以及视频帧中的像素。新的观察促发新的行动，循环往复。
“状态”一词需要拆解，因为它的含义在不同领域间会发生漂移。这并不是化学家所说的状态（固态、液态和气态之间的区别）。这是物理学家和机器人专家眼中的状态：对给定时刻世界上正在发生的事情的完整描述，包括每一个物体、每一个位置、每一个速度和每一种属性。状态是世界的潜在现实；原则上它是完整的，但对于其中的任何智能体来说永远无法直接看到。观察是智能体对该现实的部分视图。行动则是智能体对此做出的反应。
这个循环——从智能体到行动，到状态，再到观察并返回——正是赋予现代术语“世界模型”技术含义的结构。这个词组本身历史更久，可追溯至 Kenneth Craik 1943 年提出的“心智通过运行现实的‘小规模模型’进行推理”的假设，并在 20 世纪 80 年代末 90 年代初被引入神经网络。同时，这个循环也解释了人们如今赋予该术语的含义。如今被称作世界模型的各种不同事物，实际上是这同一循环的不同投影。它们各自输出循环中的不同部分。
## 世界模型的三大功能
第一种世界模型是渲染器。渲染器输出以像素形式呈现的观察，旨在供人眼观看，最重要的质量指标是视觉保真度。一个能将文本提示转换为电影级无人机航拍镜头的视频模型就是渲染器。像 Google 的 Genie 3 或 World Labs 自家的 RTFM 这样的交互系统也是如此，模型根据用户输入实时生成帧。这类模型不包含对三维结构的显式理解。它生成的是观察者会看到的样子，而不是实际存在的样子。无人机镜头中的建筑物从上方看可能完美无瑕，但若试图驾车穿梭于其下，它们就会分崩离析。
第二种是模拟器。模拟器输出状态：一种在几何、物理或动力学上都忠实的世界表征，人类和计算机程序都可以对其进行计算和交互。如果说渲染器的契约纯粹是视觉层面的，那么模拟器的契约则是结构层面的，它要求几何结构经得起推敲，物理定律符合牛顿力学，且动力学行为符合世界在物理法则下的应有表现。模拟器同时服务于两类用户。人类专业人士，如建筑师、设计师、电影制作人和游戏开发者，需要的准确度要超越视觉上的合理性。强化学习智能体、机器人控制器和自动驾驶汽车等计算机程序，则将模拟器作为训练场，在其中大规模地与世界互动，测试那些在现实中危险、昂贵或无法运行的场景。
第三种是规划器。规划器输出行动。给定一个观察和一个目标，规划器回答智能体下一步该做什么的问题。这在许多方面是渲染器的逆过程。渲染器将行动作为输入并生成观察，而规划器将观察作为输入并生成行动，从而闭合感知-行动循环。视觉-语言-行动模型、基于模型的系统以及新一代的世界行动模型都是对规划器的尝试：即能够在无结构环境中决定机器人应做什么的系统。
这三类涵盖了如今大部分实际交付的内容，而且它们之间的区分在实践中很有用。然而，这些类别并非根本分离的。对于世界如何运作——几何学、物理学、动力学——的同一底层知识支撑着它们所有。原则上，一个能从任何角度渲染杯子的模型，也应该能够模拟杯子被推时的情形，并规划出一只手拿起杯子的动作。越来越有趣的研究正有意模糊这三者之间的界限。
## 为什么模拟是关键环节
在这三类中，模拟器受到的公众关注最少，却是最具后果的一类。本文将探讨这种不对称性。
渲染器无疑是商业化程度最高的。众多的图像或文本到视频产品正在消费级或企业级市场迅速扩张。Google 的 Nano Banana 模型已将渲染器质量的图像生成交到了数亿潜在用户手中。技术是真实的，市场也是真实的。然而，渲染器优化的是视觉合理性而非物理准确性，这一天花板至关重要。它们的输出很美，但不能被信任去设计建筑或训练机器人。
规划器是最引人入胜也最新兴的领域，与快速发展的机器人学习领域紧密相连。该领域在过去两年里展示了一些在视频中令人印象深刻的机器人演示，但对于这些演示实际展示了什么，我们需要诚实。几乎所有的演示都局限于高度受限的实验室环境，对象集狭窄，任务周期短。没有任何一个在现实世界部署所需的复杂性、多样性或持续时间上得到验证。从一个引人入胜的演示样片到一个能可靠地在厨房、仓库或手术室工作的机器人，之间的差距依然巨大。尽管如此，商业赌注依然巨大。一波资金雄厚的新入局者正竞相推出通用的规划系统，而最大的基础设施提供商则将规划定位在更广泛的模拟堆栈之上。能规划的机器人就是能工作的机器人，整个行业都在争当第一。
模拟是两者之间的桥梁。如果说语言是对世界的抽象，像素是对世界的投影，那么几何、物理和动力学就是世界本身。模拟器必须在这一层面上运作：它是从中衍生出视觉外观（供渲染器）和行动后果（供规划器）的结构骨干。
一个掌握了模拟的模型可以将其理解投射为像素供人类消费，并投射为行动预测供具身智能体使用。一个只掌握渲染或只掌握规划的模型无法做到其中任何一点。商业表面积是巨大的。仅 NVIDIA 的 Omniverse 就瞄准了公司估计价值超过一万亿美元的可寻址市场，涉及工厂、仓库、供应链和数字孪生。机器人训练、自动驾驶测试、建筑可视化、工程和药物发现都依赖于某种形式的模拟。
该领域最难解决的开放问题也集中于此。具有明确几何结构、材料属性和物理注释的三维数据，比渲染器训练所用的互联网视频要少好几个数量级。“仿真到现实”的差距——即事物在仿真中的行为与在现实中的行为之间的差异——依然存在。生成式模拟器在此基础上引入了新的风险：AI 生成的几何体看起来可能正确，但包含自相交或错误的尺度，从而产生荒谬的物理效果。刚体、变形物体、流体和布料同时相互作用的大规模多物理模拟，其成本仍然比单域模拟高出好几个数量级。
在 World Labs，Marble 是我们进入该领域的第一步。它接受多模态提示（文本、图像、视频或空间草图）并生成可探索的 3D 环境，输出高斯溅点用于视觉探索，同时输出物理引擎可操作的碰撞网格。但 Marble 只是更宏大篇章的第一章，随着渲染、模拟和规划之间的界限开始崩塌，这一篇章正在整个领域展开。
## 边界正在何处崩塌，未来是什么
但更多的事情即将到来。该领域目前最重要的模式是，这三类正开始相互融合。共同的洞见是：渲染世界、模拟世界和在其中行动所需的知识在很大程度上是相同的。继续之前的例子，一个真正理解杯子如何放在桌子上（其几何形状、材料属性、受力响应等）的模型，应该能够从任何角度渲染该杯子，模拟杯子被推时会发生什么，并规划出一只手拿起杯子的动作。这三类是单一底层理解的三个投影。
例如：来自各个机器人实验室的少量但越来越多的近期工作表明——至少在概念上——预训练的视频渲染器可以用作联合世界和行动预测的主干，通过让一个模型想象会发生什么以及该做什么，从而在渲染器和规划器之间架起桥梁。World Labs 的 Marble 已经能从单一模型输出高斯溅点和碰撞网格，消融了渲染器和模拟器之间的界限。每一个层面都在从被动输出转向交互系统，渲染器变得以行动为条件，模拟器生成更可控和可编辑的世界，规划器则在进行 deliberation（深思熟虑）而不仅仅是反应。
逻辑上的终点是一个统一的世界模型：一个基础模型，能够渲染逼真的视图，生成物理上准确的结构，并规划行动序列，根据下游消费者的需求在输出模态之间切换。我们仍将面临许多艰巨的挑战。数据情况参差不齐，渲染器拥有丰富的互联网视频，而模拟器和规划器则面临 3D 资产和机器人演示的严重短缺。优化视觉美感可能会牺牲机器人或高保真模拟所需的精度。在单一架构内调和这些张力是当今世界模型研究中定义性的开放问题，这正是 World Labs 在继续演进 Marble 时着手要做的事情。
然而，方向是明确的。该领域自 20 世纪 80 年代末以来一直下的注——即一个足够丰富的世界模型就是任何智能体观看世界、构建世界并在其中行动所需的全部——正是如今驱动整整一代研究的大赌注。赋予这个“大赌注”分量的是已经发生的融合：三条线程，每一条本身已经在驱动和塑造着数十亿美元的产业，它们始于独立的研究项目，现在却开始表现得像是一个整体。随着它们之间的界限崩塌，合在一起，它们将重塑更宏大的东西：机器智能与其所处的物理世界之间的关系——空间智能的漫长弧线。
语言给了机器一种谈论那个世界的方式。世界模型将是机器最终理解、想象、推理并与那个世界互动的方式。