# In Search for the World Model Harness for Gaming
**作者**: Alberto Hojel
**日期**: 2026-05-20T20:15:27.000Z
**来源**: [https://x.com/AlbyHojel/status/2057193508822536459](https://x.com/AlbyHojel/status/2057193508822536459)
---
What makes a game fun? And when will world models be “fun”? Can a video world model carry a game that’s actually fun to play?
Try Worlds Research Station for a limited time, with one explorer, a team at Mission Control, and a generated world reacting in real time. Then read on for our approach.
Gameplay from our action-conditioned real-time video model
Video world models already show sparks of what may become a new foundation for interactive experiences. In real time, they can generate scenes with complex real-world physics and lighting, across an almost endless diversity of worlds. These are phenomena that are incredibly hard to simulate with a traditional game engine. These models offer a possible path for visualizing and authoring high-fidelity game-like experiences to a broader category of indie game developers.
> The term “world model” is often used to describe the modeling of state dynamics (aka a forward dynamics model), arguably implying a strong understanding of the world. Action-conditioned video models predict pixels that have the appearance and dynamics of the real and fantastical worlds they are trained on. [1]
However, the initial renditions of interactive video models are often not accepted as traditional “games”. While visuals play a central role in gaming, simpler elements of game design such as goals and rewards are also important.
At Roblox, we set out to see how far we could push video world models, focusing specifically on gameplay and interactions. As the first step, we trained an action-conditioned 14B Text-Image-to-Video (TI2V) video world model that runs at 24fps, and gave it to a team of Roblox game developers to see if they could build a game with it. Along the way, we designed a lightweight harness, which we call a Game Cartridge, that couples the Roblox engine with a real-time Vision Language Model (VLM) observer, allowing the engine to manage abstract game state while the video model generates the world. Here’s what our in-house game developers built, what broke, and why we think world models will need programmable harnesses to become games…
## What unique constraints do games impose?
To build a compelling game we needed to ensure a few levers of control built into our model.
Movement: an agile locomotion control rig.
- 1st and 3rd person
- Diverse actions like jumping, sprinting, swimming, emoting
- Modifiers like wings, vehicles, snowboards
Interactions: the space of actions within games is incredibly large beyond the few degrees of freedom needed for movement.
- Manipulating the world with tools or interacting with objects
- Collecting and equipping/unequipping items
- Responsive reaction from characters and the environment
Scene control: honoring creator intent and editability.
- Changing the scene composition, adding or removing objects
- NPCs with non-trivial behavior
- Physics-level control like the strength of perceived gravity in space or underwater
The baseline conditioning in video models is not suitable for interactive, immersive gaming. A single homogenous text prompt makes it difficult to edit some components without affecting others. To allow for finer control at inference, we decomposed the text conditioning into a few core pieces: Actions (movement), World (for the environment and NPCs), Character (main avatar and equipment), Dynamics (perspective and physics) [2].
Here we showcase a few of those control levers in action:
Starting with agile locomotion (sprinting/jumping) and aggressive global scene editing, here we cycle through 7 completely different prompts cleanly morphing between the environments and character rigs. Starting with a frog in a bustling NYC avenue, we can transform into a firefighter in a warehouse, a camel caravan, riding a dragon, to a 1st-person porch in the countryside, to a backpacker in the Himalayas, to a diver underwater, and finally back to almost the same bustling NYC avenue. Notice the taxi traffic behavior responds to the player approaching.
Next, we start with a farmer using a sickle to harvest wheat, emphasizing the manipulation of the environment by use of tools with motion animations. We then show local scene editing by switching the avatar’s equipped item to a firing rifle, then to a flaming torch, which causes a fire to spread.
By orchestrating prompts, we could elicit quite interesting dynamics from our model as shown above. This surface-level interaction paradigm allows the player to serve as a puppeteer or dungeon-master, scheduling and timing the prompts to edit the world under their control and desires. However, a typical game does not expose omnipotent power to the player.
We hypothesized that a system where a developer writes a code harness around the VWM to dynamically select text prompts could enable a few core aspects of gaming that are missing from purely neural world models today: creators, state, and progression. Creators could use it to make immersive worlds that reflect their stylistic vision, with a persistent abstract state that progresses predictably as they play. By dynamically selecting prompts in a controlled fashion as the world is getting generated, we placed control of the scene under the responsibility of the developer’s code allowing us to implement quest progression.
We were inspired by the earliest of games, teletype games, which featured little to no visuals, yet had incredibly deep worlds with rich game logic. Take Zork or Oregon Trail for instance, which are simple text state machines that process text input and produce the next state of the world, allowing you to explore rooms and interact with objects.
## Our Harness
The first component of our system is this programmable code harness running within the Roblox game engine, which we call a Game Cartridge. Here, a developer writes traditional Luau code to define a state machine that tracks variables (like current objects in the world, health, or quest progression) and executes the logic. A sidecar service on the game engine then serializes the data model into text prompts fed to the VWM.
To close the loop, the engine needs to know what is happening in the generated video frames because the action outcomes are ultimately dictated by the VWM. We utilize a VLM as a perception model, observing the generated frames. The developer defines "visual triggers" within their code harness (e.g., querying the VLM, "Has the player collected an energy canister?" alongside the latest frames). The VLM continuously watches the generated pixels for these specific events. When a visual trigger is detected, the VLM fires a callback to the code harness. This synchronizes a primitive sense of positioning and action outcomes [3] to the game engine, which then updates the abstract state (e.g., updating player health and incrementing the score counter) and, in turn, selects the next text prompt (e.g., spawning a portal or taking you to the next scene). This grounds the video model's outputs into playable mechanics. We also tried directly connecting the VLM to write prompts back to the VWM, removing the Roblox game engine entirely, but found greater success allowing the system to store variables and run deterministic code.
Once we had this simple harness, we were able to define programmable game loops and execute them end-to-end with the video world model. Because this system allows a creator to build an experience, bake it into a repeatable state, and share it with others, it started to show signs of the game-making process many of us are familiar with. Befitting of the name Game Cartridges as they provide the relevant structure to encapsulate an experience powered by a VWM.
# Game Cartridges
Our devs structured the narrative into chapters, defining scenes by images, and driving the state transition logic via Luau code, supplemented by the VLM perception model.
> As one developer noted: “Building a game with this process required learning how to create visuals with the AI model itself, combining prompts and different pictures to create worlds. Then learning how to manipulate those worlds to get things to spawn in, and building a bridge between the model and the actual Roblox game engine which controls the majority of the logic.“
Today we are releasing access for a limited time to “Worlds Research Station,” one of the prototype Game Cartridges built by our developers in Roblox Innovation Studio. This prototype features a tangible, cooperative gameplay loop: one player explores dynamic, generated worlds controlled by their teammates back at Mission Control. As the explorer, you'll need to scavenge for energy cells to replenish your health and survive the elements.
Wanna try it yourself? Play the game here.
Through developer testing, two core limitations emerged that point toward meaningful directions for further research:
## Control and Flexibility:
Targeted placement of objects in the scene felt hard to control with text prompts as their spatial descriptions need to be invariant to player WASD movement. Our current video world model lacked native 3D spatial grounding, making layout design challenging [5]. The prompt “a canister appears in front of the character” can become a conflicting signal if you turn around and walk away, causing new canisters to continuously re-appear. However, a prompt like “a canister appears at the base of the tree” performs better. This perhaps highlights the value of 3D or relative layout information, and how video models can be co-developed with developers in mind to elicit desired mechanics [4].
Because our model was trained on 5-second clips paired with global world captions, its counterfactual understanding wasn’t strong enough to handle long horizon state transitions by itself. Developers observed that during the rollout, models often lost track of which step of the chain the video is currently on, degenerating into looping behavior. For example, if you prompt an autoregressive video model with “a rocket launches from the ground” and roll out for a long horizon, you end up seeing looping behavior where the rocket takes off, lands, and takes off again; in this case, updating the prompt to “a rocket in space” is eventually required to avoid repeating the “launch.” Future endeavors should attempt to instill better counterfactual (conditional mechanics) understanding into the VWM.
## Visual Consistency:
While the code harness effectively manages the abstract state and utilizes serialized text prompts to inject detailed environmental descriptions, a sliding window KV cache alone is insufficient to ensure strict visual consistency (e.g., maintaining a character's exact identity when they turn away from the camera and back). Extending these windows with compression or spatially indexed caches [6] is one path to maintaining visual object permanence. Visual drift in appearances can also be a signal of a lack of proper conditioning, where the text prompt is not specific enough to constrain the generative model. Dynamically injecting reference images into the prompt to ground multiple identities and object appearances on the fly could bridge the gap further (a task potentially well suited for a harness).
# What’s next?
In our mission to become the highest-fidelity, interactive, multiplayer gaming platform, our exploration using Game Cartridges illustrated that by augmenting a Video World Model with a programmable coding harness, a developer could begin to add game-like mechanics to what was previously just an explorable world without objectives. However, it also highlighted the limitations of Video World Models for owning the majority of the visual state, and suggested that more of the underlying game logic and 3D state should be implemented in the Roblox game engine. The video model is then used to uplift the view into high-fidelity or any style desired by the game developer [5].
Game Cartridges also serve as an evaluation harness, helping us benchmark the consistency, coherence, and responsiveness of these models in the pursuit of game making. In the future, when video world models become more robust, we may still desire having programmable code harnesses around them. We similarly envision this becoming a useful pre-visualization tool in the game development lifecycle, helping prototype quick visual aesthetic and gameplay ideas starting from images and text.
# Footnotes
[1] I’ve been interested in world models without knowing it since I was young, when I would wake up without the ability to recall any image of my nightly adventures leading me to believe that I hadn’t dreamt.
Growing up we are often told to dismiss our dreams and our nightmares. This often leads many people to believe that they cannot dream at all or that they did not dream last night. This is not usually true! Humans often dream 6-7+ times, but as soon as we wake up we tell ourselves: “Ah! It was just a dream” and we train ourselves to forget. We actively suppress our memories of dreams; and years of this results in one believing they did not dream because they simply cannot remember.
And thus comes the first tool in an oneironaut’s toolbox: the dream journal. Waking up and immediately jotting down every image you can remember from your nightly adventures that you can recall. And after about a week of actively recalling your dreams, you become surprisingly aware of their behaviors. You start to gain the ability to piece together the boundaries of dreams (aka REM sessions), and you start to notice patterns. This is how I became conscious that I was dreaming during a dream for the first time when I was younger; I realized I had entered a very similar dream I had jotted down in my journal a few days ago. And thus I became lucid for the first time. [A Field Guide to Lucid Dreaming: Mastering the Art of Oneironautics by Dylan Tuccillo, Jared Zeizel, & Thomas Peisel.]
At the behest of my will to dream, I became enthralled by the art of oneironautics: becoming aware that you are asleep experiencing a simulation architected by yourself. A world where by merely thinking you can conjure up whatever your heart desires and even summon and talk to the internal representations you have of the people around you and those you love.
After taking interest in why it is that our brain developed this universal behavior and how it appears across many species, I became interested in the concept of internal world models.
Although the origin of this concept goes all the way back to ancient Greek philosophers [https://people.idsia.ch/~juergen/world-model-boom.html], I like to point to Kenneth Craik who in 1943 questioned how it is that organisms may reason and act intelligently in the real world. He describes an internal world model that allows organisms to play out certain scenarios in their heads, to simulate the outcomes based on their prior experience.
“If the organism carries a “small-scale model” of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilise the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to the emergencies which face it.” (The Nature of Explanation, Craik 1943)
The predictive power that you and I experience and many other species across the world do too, is not programmed in the sense that a game engine is. Rather it is learned through experience as we interact with our complicated world. Evolution found it useful for agents to be able to simulate other agents in their head in order to predict their responses and take actions that would result in rewards and preserve the chances of survival. And even deeper than that, one doesn’t get a lot of chances to get chased by a lion out in the real world. The first time you do, you better know what to do. And I wonder if that is why we all have nightmares of being chased by lions while we are young.
[2] This scheme avoids spatially coupling the main character within the environment (e.g.,“riding a bicycle on a path” which is not invariant to WASD actions, causes paths to unnaturally reappear when trying to steer off-road). We mapped raw WASD and mouse inputs directly into a structured text action grammar -- for example: “Movement: Forward; Camera: Left; Action: Jump;” -- temporally aligned to the latent frames where those actions occur. This unified text-space action scheme supports custom keybindings during training and inference, where interaction keys like spacebar, shift, E, can be mapped into semantic text actions like E -> “Open door.”
[3] In this case, we are essentially using the VLM as an external reward model to measure whether a certain condition has been met or not. Instead of having a separate perception model, DreamerV4 [https://danijar.com/project/dreamer4/] explicitly predicts the 0->1 reward and value of the state against a particular defined task like “mine_diamonds” across the rollout. Although they ultimately used it to train RL agents, it nonetheless showcases how a reward model can be used to “measure progress against a quest.” One can imagine training this capability into the VWM itself like Dreamer does, removing the need for the VLM observer.
[4] Among many recent works exploring video world models for gaming, one of note is MultiGen [https://ryanpo.com/multigen/] which allows for persistent editable multiplayer worlds via explicit geometry
[5] While in this exploration, the Roblox game engine ultimately only handles abstract variables and game logic, a proper 3D integration is an active area of research and something we hope to demo in the future. https://about.roblox.com/newsroom/2026/04/roblox-reality-hybrid-architecture-democratizing-photorealistic-multiplayer-gaming
[6] We already see video equivalents of RAG in the form of city scale spatial caches that aim to provide persistent lean memory via a 3D-grounded keyframe database allowing the model to condition on the geometry embedded inside of multiview image references [https://seoul-world-model.github.io/]
## 相关链接
- [Alberto Hojel](https://x.com/AlbyHojel)
- [@AlbyHojel](https://x.com/AlbyHojel)
- [58K](https://x.com/AlbyHojel/status/2057193508822536459/analytics)
- [Try Worlds Research Station](https://www.roblox.com/games/137611642510063/Worlds-Research-Station)
- [Play the game here](https://www.roblox.com/games/137611642510063/Worlds-Research-Station)
- [https://people.idsia.ch/~juergen/world-model-boom.html](https://people.idsia.ch/~juergen/world-model-boom.html)
- [https://danijar.com/project/dreamer4/](https://danijar.com/project/dreamer4/)
- [https://ryanpo.com/multigen/](https://ryanpo.com/multigen/)
- [https://about.roblox.com/newsroom/2026/04/roblox-reality-hybrid-architecture-democratizing-photorealistic-multiplayer-gaming](https://about.roblox.com/newsroom/2026/04/roblox-reality-hybrid-architecture-democratizing-photorealistic-multiplayer-gaming)
- [https://seoul-world-model.github.io/](https://seoul-world-model.github.io/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:15 AM · May 21, 2026](https://x.com/AlbyHojel/status/2057193508822536459)
- [58.9K Views](https://x.com/AlbyHojel/status/2057193508822536459/analytics)
- [View quotes](https://x.com/AlbyHojel/status/2057193508822536459/quotes)
---
*导出时间: 2026/5/21 10:05:17*
---
## 中文翻译
# 寻找游戏的世界模型驾驭框架
**作者**： Alberto Hojel
**日期**： 2026-05-20T20:15:27.000Z
**来源**： [https://x.com/AlbyHojel/status/2057193508822536459](https://x.com/AlbyHojel/status/2057193508822536459)
---
是什么让一款游戏变得有趣？世界模型何时才能变得“有趣”？视频世界模型能否承载一款真正好玩的游戏？
限时体验 Worlds Research Station：一名探索者、一个任务控制中心团队，以及一个实时响应的生成世界。然后请继续阅读我们的方法。
我们的动作条件实时视频模型的游戏画面
视频世界模型已经显现出可能成为互动体验新基础的火花。它们可以实时生成具有复杂真实世界物理效果和光照的场景，跨越几乎无穷无尽的世界多样性。这些现象用传统游戏引擎来模拟是极其困难的。这些模型为更广泛的独立游戏开发者群体提供了一条可视化并创作高保真类游戏体验的可能路径。
> “世界模型”一词常被用来描述状态动态的建模（又称前向动态模型），可以说这暗含着对世界的深刻理解。动作条件视频模型预测的像素具有其训练所用的真实世界和奇幻世界的外观与动态特性。[1]
然而，交互式视频模型的最初版本往往不被视为传统意义上的“游戏”。虽然视觉在游戏中扮演着核心角色，但游戏设计中更简单的元素，如目标和奖励，同样重要。
在 Roblox，我们着手探索视频世界模型能被推进到什么程度，特别专注于游戏玩法和交互。作为第一步，我们训练了一个动作条件的 14B 文本-图像生成视频（TI2V）视频世界模型，能以 24fps 运行，并将其交给一个 Roblox 游戏开发者团队，看看他们能否用它开发出一款游戏。在此过程中，我们设计了一个轻量级驾驭框架，我们称之为 Game Cartridge（游戏卡带），它将 Roblox 引擎与一个实时视觉语言模型（VLM）观察器耦合，使引擎能够管理抽象的游戏状态，而视频模型则生成世界。以下是我们的内部游戏开发者构建的内容、出现的问题，以及为什么我们认为世界模型需要可编程的驾驭框架才能成为游戏……
## 游戏带来了哪些独特的约束？
要构建一款引人入胜的游戏，我们需要确保模型内置几个控制杠杆。
移动：灵活的运动控制装置。
- 第一人称和第三人称
- 多样的动作，如跳跃、冲刺、游泳、表情动作
- 修饰项，如翅膀、载具、滑雪板
交互：游戏中动作的空间极其庞大，远不止移动所需的少数自由度。
- 使用工具操控世界或与物体交互
- 收集和装备/卸下物品
- 角色和环境的响应式反应
场景控制：尊重创作者意图并保持可编辑性。
- 更改场景构成，添加或移除物体
- 具有非平凡行为的 NPC
- 物理层面的控制，如太空中或水下感知到的重力强度
视频模型的基础条件设定并不适合互动、沉浸式的游戏。单一的同质文本提示使得在不影响其他组件的情况下编辑某些组件变得困难。为了在推理时实现更精细的控制，我们将文本条件分解为几个核心部分：动作（移动）、世界（环境和 NPC）、角色（主化身和装备）、动态（视角和物理）[2]。
在这里我们展示了其中一些控制杠杆的实际效果：
从灵活的运动（冲刺/跳跃）和大刀阔斧的全局场景编辑开始，这里我们在 7 个完全不同的提示之间循环，在环境和角色装置之间干净利落地变形。从熙熙攘攘的纽约大道上的一只青蛙开始，我们可以变身为仓库里的消防员、骆驼商队、骑龙飞行、乡村门廊的第一人称视角、喜马拉雅山的背包客、水下的潜水员，最后回到几乎相同的熙熙攘攘的纽约大道。请注意出租车交通行为会对玩家接近做出反应。
接下来，我们从一个用镰刀收割小麦的农民开始，强调通过带有动作动画的工具来操控环境。然后我们展示局部场景编辑：将化身的装备切换为一把正在射击的步枪，再切换为一支燃烧的火炬，从而引发火势蔓延。
通过编排提示，我们可以从模型中激发出相当有趣的动态效果，如上所示。这种表层交互范式允许玩家充当操纵者或地下城主，通过调度和安排提示的时机，按照自己的控制意愿来编辑世界。然而，典型的游戏并不会赋予玩家全能的权力。
我们假设，一个由开发者在 VWM 周围编写代码驾驭框架以动态选择文本提示的系统，可以实现当今纯神经世界模型所缺失的几个游戏核心要素：创作者、状态和进程。创作者可以用它制作反映其风格愿景的沉浸式世界，并拥有一个随游玩可预测推进的持久抽象状态。通过在世界生成的过程中以受控方式动态选择提示，我们将场景的控制权交由开发者的代码负责，从而实现了任务进程系统。
我们的灵感来源于最早的游戏——电传打字机游戏，它们几乎没有视觉画面，却拥有极其深邃的世界和丰富的游戏逻辑。以《Zork》或《俄勒冈之旅》为例，它们是简单的文本状态机，处理文本输入并产生世界的下一个状态，让你可以探索房间并与物品互动。
## 我们的框架
我们系统的第一个组件是运行在 Roblox 游戏引擎内的可编程代码框架，我们称之为游戏卡带。在这里，开发者编写传统的 Luau 代码来定义一个状态机，该状态机追踪变量（如世界中的当前物品、生命值或任务进度）并执行逻辑。然后，游戏引擎上的一个旁路服务会将数据模型序列化为文本提示，输入到视频世界模型（VWM）中。
为了形成闭环，引擎需要知道生成的视频帧中正在发生什么，因为动作结果最终由 VWM 决定。我们利用视觉语言模型（VLM）作为感知模型，观察生成的帧。开发者在他们的代码框架中定义“视觉触发器”（例如，结合最新帧向 VLM 查询“玩家是否收集了能量罐？”）。VLM 持续监视生成的像素，以检测这些特定事件。当检测到视觉触发器时，VLM 会向代码框架触发回调。这将一种原始的位置感和动作结果 [3] 同步到游戏引擎，引擎随后更新抽象状态（例如，更新玩家生命值并增加分数计数器），进而选择下一个文本提示（例如，生成一个传送门或将你带到下一个场景）。这将视频模型的输出落地为可玩的机制。我们也尝试过直接让 VLM 将提示写回 VWM，完全移除 Roblox 游戏引擎，但发现允许系统存储变量并运行确定性代码的做法更为成功。
有了这个简单的框架，我们就能够定义可编程的游戏循环，并使用视频世界模型端到端地执行它们。因为这个系统允许创作者构建一个体验，将其固化为可重复的状态，并分享给他人，它开始显现出我们许多人所熟悉的游戏制作过程的迹象。这与其“游戏卡带”之名十分相称，因为它们提供了将一个由 VWM 驱动的体验封装起来的相关结构。
# 游戏卡带
我们的开发者将叙事结构化为章节，通过图像定义场景，并通过 Luau 代码驱动状态转换逻辑，辅以 VLM 感知模型。
> 正如一位开发者所说：“用这个流程构建游戏需要学习如何用 AI 模型本身来创造视觉画面，组合提示词和不同的图片来创建世界。然后学习如何操纵这些世界，让物体在其中生成，并在模型与控制大部分逻辑的实际 Roblox 游戏引擎之间搭建桥梁。”
今天，我们在限时开放“世界研究站”的访问权限，这是我们的开发者在 Roblox Innovation Studio 中构建的原型游戏卡带之一。这个原型具有切实可行的合作玩法循环：一名玩家探索动态生成的世界，而这些世界由身处任务控制中心的队友控制。作为探索者，你需要寻找能量电池来补充生命值，并在恶劣环境中生存下来。
想亲自试试吗？点击这里玩游戏。
通过开发者测试，我们发现了两个核心局限性，它们为进一步研究指明了有意义的方向：
## 控制与灵活性：
场景中物体的定点放置很难通过文本提示来控制，因为它们的空间描述需要对玩家 WASD 移动保持不变性。我们目前的视频世界模型缺乏原生的 3D 空间落地能力，使得布局设计颇具挑战 [5]。诸如“一个罐子出现在角色面前”这样的提示，如果你转身走开，就可能变成冲突信号，导致新罐子不断重复出现。然而，像“一个罐子出现在树下”这样的提示效果更好。这或许凸显了 3D 或相对布局信息的价值，以及如何在开发视频模型时将开发者的需求纳入考量，以引出期望的机制 [4]。
由于我们的模型是在配有全局世界描述的 5 秒片段上训练的，其反事实理解能力不足以独自处理长时程的状态转换。开发者观察到，在推理生成过程中，模型常常无法追踪视频当前处于链条的哪一步，退化为循环行为。例如，如果你用“一枚火箭从地面发射”提示一个自回归视频模型并生成较长的时序，你会看到火箭起飞、降落、再次起飞的循环行为；在这种情况下，最终需要将提示更新为“太空中的一枚火箭”才能避免重复“发射”场景。未来的努力应尝试向 VWM 中灌输更好的反事实（条件机制）理解能力。
## 视觉一致性：
虽然代码工具架能够有效管理抽象状态，并利用序列化的文本提示注入详细的环境描述，但仅靠滑动窗口 KV 缓存还不足以确保严格的视觉一致性（例如，当角色转过身背对镜头再转回来时，保持其完全相同的身份特征）。通过压缩或空间索引缓存来扩展这些窗口 [6]，是维持视觉对象恒常性的一条路径。外观上的视觉漂移也可能是条件约束不足的信号，即文本提示不够具体，无法充分约束生成模型。在提示中动态注入参考图像，以便在运行时锚定多个身份和物体的外观，可以进一步弥合这一差距（这项任务可能非常适合由工具架来承担）。
# 接下来是什么？
在我们致力于打造最高保真度、可交互的多人游戏平台的使命中，我们通过 Game Cartridges 展开的探索表明：通过为视频世界模型配备可编程的代码工具架，开发者可以开始在一个原本只是可探索、没有目标的世界中加入类似游戏的机制。然而，这也凸显了视频世界模型在承载大部分视觉状态方面的局限性，并表明更多的底层游戏逻辑和 3D 状态应该交由 Roblox 游戏引擎来实现。随后，视频模型则被用来将画面提升为高保真效果，或游戏开发者所期望的任何风格 [5]。
Game Cartridges 同时还充当了一个评估工具架，帮助我们在游戏制作的过程中对这些模型的一致性、连贯性和响应性进行基准测试。未来，当视频世界模型变得更加强大时，我们可能仍然希望在它们周围配备可编程的代码工具架。同样地，我们设想它将成为游戏开发生命周期中一个有用的预可视化工具，帮助开发者从图像和文本出发，快速原型化视觉美学和玩法创意。
# 脚注
[1] 我其实在不知不觉中早已对世界模型产生兴趣。小时候，我醒来时完全无法回忆起夜间梦境的任何画面，以至于我曾以为自己根本没有做梦。
成长过程中，我们常常被告知不要把梦和噩梦当回事。这往往让许多人相信自己根本不会做梦，或者昨晚没有做梦。但事实通常并非如此！人类每晚往往做梦 6-7 次甚至更多，但一醒来我们就对自己说：“啊！不过是个梦罢了”，于是我们训练自己将其遗忘。我们主动压抑对梦的记忆；经年累月，结果就是一个人相信自己没有做梦，只是因为他们根本记不起来。
由此便引出了“逐梦者”（oneironaut）工具箱中的第一件工具：梦境日记。醒来后立即记下你能回忆起的夜间奇遇中的每一个画面。在坚持主动回忆自己的梦境大约一周之后，你会惊人地察觉到梦的规律。你开始能够拼凑出梦境的边界（即 REM 快速眼动周期），并开始注意到其中的模式。正是这样，年少时的我第一次在梦中意识到自己正在做梦——我发现自己进入了一个几天前刚刚记在日记里的、非常相似的梦境。于是，我第一次获得了清醒梦的体验。[《A Field Guide to Lucid Dreaming: Mastering the Art of Oneironautics》（清醒梦实地指南：掌握逐梦的艺术），作者 Dylan Tuccillo、Jared Zeizel 和 Thomas Peisel。]
出于对做梦的强烈意愿，我迷上了“逐梦术”（oneironautics）这门艺术：意识到自己正处于睡梦中，体验着一个由自己构建的模拟世界。在这个世界里，仅仅通过想象，你就能召唤出心中所渴望的一切，甚至可以召唤出你对身边之人以及所爱之人的内在表征，并与他们交谈。
在对“我们的大脑为何进化出这种普遍行为，以及它如何在许多物种中显现”产生兴趣之后，我又对“内在世界模型”这一概念产生了兴趣。
尽管这一概念的起源可以追溯到古希腊哲学家 [https://people.idsia.ch/~juergen/world-model-boom.html]，但我更愿意援引 Kenneth Craik，他在 1943 年提出了这样一个问题：生物体如何在现实世界中理性行事、智能行动。他描述了一种内在世界模型，使生物体能够在头脑中推演特定情境，并根据过往经验模拟其结果。
“如果生物体在其头脑中携带一个关于外部现实及其自身可能行动的‘小尺度模型’，它便能够尝试各种替代方案，推断出其中最优的一种，在未来情境出现之前就做出反应，利用对过去事件的知识来应对现在和未来，从而在各个方面以更充分、更安全、更能干的方式应对所面临的紧急情况。”（《解释的本质》（The Nature of Explanation），Craik，1943 年）
你我所体验到的、世界上许多其他物种也同样拥有的预测能力，并不是像游戏引擎那样被编程出来的。相反，它是我们在与这个复杂世界互动的过程中，通过经验习得的。进化发现，让智能体能够在头脑中模拟其他智能体、预测它们的反应，并采取能带来奖励、保障生存的行动，是非常有用的。更进一步说，一个人在现实中被狮子追逐的机会并不多。第一次遇到时，你最好已经知道该怎么做。我在想，这是否正是为什么我们年少时都会做被狮子追赶的噩梦。
[2] 这种方案避免了将主角与环境进行空间耦合（例如“在路径上骑自行车”对 WASD 操作不具备不变性，当试图驶离道路时会导致路径不自然地重现）。我们将原始的 WASD 和鼠标输入直接映射到结构化的文本动作语法中——例如：“Movement: Forward; Camera: Left; Action: Jump;”——并在时间上与这些动作发生的潜在帧对齐。这种统一的文本空间动作方案支持在训练和推理过程中使用自定义按键绑定，其中空格、Shift、E 等交互按键可以映射为语义化的文本动作，例如 E -> “Open door”（开门）。
[3] 在这种情况下，我们实际上是使用 VLM 作为外部奖励模型来衡量某个条件是否已满足。DreamerV4 [https://danijar.com/project/dreamer4/] 没有使用单独的感知模型，而是在整个 rollout 过程中，针对特定定义的任务（如“mine_diamonds”），显式地预测状态的 0->1 奖励和价值。虽然他们最终用它来训练 RL 智能体，但它仍然展示了奖励模型如何被用来“衡量任务进度”。可以想象，像 Dreamer 那样将这种能力训练到 VWM 本身中，从而无需 VLM 观察者。
[4] 在近期许多探索用于游戏的视频世界模型的工作中，值得关注的是 MultiGen [https://ryanpo.com/multigen/]，它通过显式几何实现了可持久化编辑的多人世界
[5] 虽然在这项探索中，Roblox 游戏引擎最终只处理抽象变量和游戏逻辑，但真正的 3D 整合是一个活跃的研究领域，我们希望在未来进行演示。https://about.roblox.com/newsroom/2026/04/roblox-reality-hybrid-architecture-democratizing-photorealistic-multiplayer-gaming
[6] 我们已经看到了 RAG 的视频等价形式，即城市尺度的空间缓存，它旨在通过一个 3D 锚定的关键帧数据库提供持久的轻量记忆，使模型能够以多视角图像参考中嵌入的几何信息为条件 [https://seoul-world-model.github.io/]
## 相关链接
- [Alberto Hojel](https://x.com/AlbyHojel)
- [@AlbyHojel](https://x.com/AlbyHojel)
- [58K](https://x.com/AlbyHojel/status/2057193508822536459/analytics)
- [Try Worlds Research Station](https://www.roblox.com/games/137611642510063/Worlds-Research-Station)
- [Play the game here](https://www.roblox.com/games/137611642510063/Worlds-Research-Station)
- [https://people.idsia.ch/~juergen/world-model-boom.html](https://people.idsia.ch/~juergen/world-model-boom.html)
- [https://danijar.com/project/dreamer4/](https://danijar.com/project/dreamer4/)
- [https://ryanpo.com/multigen/](https://ryanpo.com/multigen/)
- [https://about.roblox.com/newsroom/2026/04/roblox-reality-hybrid-architecture-democratizing-photorealistic-multiplayer-gaming](https://about.roblox.com/newsroom/2026/04/roblox-reality-hybrid-architecture-democratizing-photorealistic-multiplayer-gaming)
- [https://seoul-world-model.github.io/](https://seoul-world-model.github.io/)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [4:15 AM · May 21, 2026](https://x.com/AlbyHojel/status/2057193508822536459)
- [58.9K Views](https://x.com/AlbyHojel/status/2057193508822536459/analytics)
- [View quotes](https://x.com/AlbyHojel/status/2057193508822536459/quotes)
---
*导出时间： 2026/5/21 10:05:17*
