LeWorldModel: 利用双损失函数从像素中学习稳定的物理世界模型

Junfan Zhu 朱俊帆 2026-04-23 20.8 KB 82.1K 118 𝕏 文章列表

文章介绍了 LeWorldModel (LeWM)，一种新的世界模型架构。它解决了当前世界模型（如 DINO-WM 和 PLDM）的局限，仅通过两个损失项（Next-embedding 预测损失和 SIGReg）实现了从原始像素开始的稳定端到端训练。实验表明，LeWM 能在潜在空间中自发涌现出时间直线性和物理约束，有效避免了表示偏差和模型崩塌问题。

WorldModel JEPA 深度学习物理仿真 SIGReg 端到端训练计算机视觉强化学习 LeWorldModel

# 135. LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses **作者**: Junfan Zhu 朱俊帆 **日期**: 2026-04-22T19:06:44.000Z **来源**: [https://x.com/junfanzhu98/status/2047029352995639341](https://x.com/junfanzhu98/status/2047029352995639341) --- ![Image](/doc/987/image/img_75a3000a06ba.jpg) ![Image](/doc/987/image/img_a3543f38942d.jpg) 1. Current Landscape of World Models ## 1.1 DINO-WM DINO-WM uses a pretrained Vision Transformer (ViT) encoder to convert video frames into latent features, and then trains an implicit predictor on top of these features. However, the encoder is frozen and cannot be trained end-to-end. This leads to a fundamental consequence: > The predictor is forced to learn a truncated (information-limited) world model. A precise analogy (which reflects a technical limitation, not just intuition): - The model is like a student who can only read someone else’s notes, - But is never allowed to solve problems independently. ## Representation Bias Problem (Critical Technical Point) The encoder is pretrained on ImageNet. This introduces a representation prior (“visual genetics”): - It is optimized for coarse-grained semantic classification (e.g., cats vs. dogs), - Not for fine-grained physical resolution. Concrete failure mode: > It can correctly classify objects, but cannot resolve physical changes at millimeter scale, e.g., a 2 mm displacement of a wooden block. Therefore: - The encoder behaves like wearing highly myopic glasses trained on ImageNet. - It fundamentally cannot “see” micro-level physical state changes. Key implication: > No matter how powerful the predictor is, if it is attached to an encoder that cannot perceive fine-grained physical reality, the system is effectively “blind reasoning about physics” (blind men touching an elephant). ## 1.2. Predictive Latent Dynamic Model (PLDM) PLDM is end-to-end trainable. However, this comes with two strict limitations: (1) Training Instability - Training is unstable - Easily collapses (2) Reward Signal Constraint (Hard Constraint) PLDM uses reward signals as prediction targets. Therefore: > It can only operate in environments with explicitly defined reward signals (e.g., games). This is not “works better”—it is a hard applicability constraint: - Without a well-defined reward → the method is not applicable. ## 1.3. Joint Embedding Predictive Architecture (JEPA) JEPA removes the need for pixel reconstruction. Training objective: - Encode current frame → embedding - Predict next-step embedding - Ensure predicted delayed embedding ≈ actual delayed embedding However, JEPA has two unresolved core problems: (1) Collapse Problem The encoder may output a trivial solution: - Map all inputs → the same constant vector (e.g., all zeros) This makes prediction trivial, but destroys all information. (2) Pixel-Level End-to-End Training + Stability (Joint Constraint) The challenge is not just “raw pixels”: > The system must simultaneously achieve: - pixel-level input - end-to-end training - stable optimization (no collapse) Most JEPA variants fail to satisfy all three simultaneously. 2. LeWM: Minimal, Stable JEPA from Raw Pixels LeWorldModel is the first JEPA that satisfies: > Stable end-to-end training directly from raw pixels, using only two loss terms: Next-embedding prediction loss A Gaussian-distribution regularizer (SIGReg) ## 2.1 End-to-End Training from Raw Pixels Unlike DINO-WM: - The encoder is not frozen - Perception (encoder) and cognition (predictor) are trained through a single unified data pipeline from input to output This removes the ImageNet-induced “myopic bias”. Insight: > The encoder’s “visual genetics” determines what physical structure it can perceive. Thus: - A classification-trained encoder → coarse semantics only - A jointly trained encoder → evolves together with physical reasoning Interpretation: > LeWM removes the “myopic glasses” and allows the visual system and physical intuition to co-evolve synchronously. This is the true meaning of end-to-end training, and a necessary condition for physical understanding. ## 2.2 “Only” One Hyperparameter PLDM requires 6 hyperparameters. LeWM requires exactly one: - λ (weight of SIGReg) Because: > SIGReg is intrinsically robust, and does not require delicate tuning. Result: - Plug-and-play training - Stable convergence without complex hyperparameter search 3. Collapse Problem in JEPA JEPA objective: - Encoder: ( x_t \rightarrow z_t ) - Predictor: ( (z_t, a_t) \rightarrow \hat{z}_{t+1} ) - Minimize: ( | \hat{z}{t+1} - z{t+1} | ) Degenerate solution: > Encoder outputs a constant vector (e.g., all zeros) Then: - Predictor sees identical inputs every time - Prediction becomes trivial - Loss is minimized This is collapse via trivial solution. 4. SIGReg: Gaussian Integral Signature Regularization Gaussian Integral Signature Regularization ## 4.1 Core Intuition (Shadow / Signature Interpretation) - “Signature” = projection (shadow) of a high-dimensional object - Gaussian distribution = test of non-degeneracy If all projections are Gaussian: > The original high-dimensional embedding cannot collapse to a single point Because: - A collapsed vector → all projections are constant (degenerate), not Gaussian - Gaussian requires: variance, dispersion, sample diversity ## 4.2 Mathematical Foundation Cramér–Wold theorem > A high-dimensional distribution is fully determined by the distributions of its 1D projections. Thus: - No need to test full distribution - Testing sufficiently many 1D projections is sufficient ## 4.3 SIGReg Procedure Sample 1024 random directions from a standard normal distribution Project each embedding vector onto all 1024 directions For each projection: Perform Epps–Pulley statistical test. Verify that the marginal distribution is close to standard normal. The target of the test is: The test statistic → 0 (perfect Gaussian match). Aggregate all projection test results into a regularization loss term Add this term to the overall training objective Effect: > The encoder is explicitly penalized if any projection (“shadow”) fails the Gaussian test Thus: - The encoder cannot take shortcuts - Every projection must be “valid” ## 4.4 Hyperparameter Only one: - λ (weight of SIGReg) Because SIGReg is highly robust: - No fine-tuning required - Stable across settings ## 5. Physical Probing Experiment Train LeWM in a PushT environment: Task: push a block to a target location ## 5.1 Linear Probe After training: - Freeze the model - Train a linear regressor on embeddings to predict: position, orientation, robot end-effector position If embeddings encode physics: > A linear model should recover accurate physical quantities. Result: - Successful → embeddings encode structured physical information ## 5.2 Out-of-Distribution Intervention Intervention: > Teleport the block to a physically impossible position Observation: > Embedding-space anomaly values spike sharply (large-scale surge, not minor deviation) Interpretation: - The model has learned that: Objects cannot teleport Critical distinction: > This constraint is internalized in the latent representation, NOT directly read from pixel-level surface features. ## 5.3 Insight This is the fundamental difference: - Video prediction: predicts pixels - World models: encode physical constraints and invariants 6. Emergent Temporal Straightness Important fact: > There is NO explicit temporal smoothness term in the loss. Model is not told of: - “predictions should be temporally smooth” Yet observed: > In embedding space, trajectories between two states are approximately straight lines ## 6.1 Interpretation - The model is NOT hallucinating smooth transitions - It is learning physically valid displacement trajectories Key claim: > Temporal straightness emerges without any prior (no inductive bias) ## 6.2 Implication for Control If the world model produces: - smooth and physically consistent latent trajectories Then: - planning based on this model is reliable - avoids unrealistic trajectories caused by model artifacts 7. Performance Planning Time - LeWM: 0.98 seconds - DINO-WM: 47 seconds Success Rate - PLDM: 78% - LeWM: 96% ## 7.1 Why DINO-WM Is Slow Because: Encoder is frozen → Representation loses task-relevant information → Requires additional online propagation steps → To compensate for information loss LeWM: - End-to-end trained encoder - Representation is natively aligned with prediction task - No extra computation required ## 7.2 Real-World Implication Speed determines deployability in robotics: - 0.98 seconds: Can handle dynamic obstacles, Can react to unexpected events in real time - 47 seconds: Not deployable in real control loops 8. Limitations 8.1 Model Size - ~15M parameters - “Small as an ant” (i.e., severely under-scaled) Consequence: > Cannot model high-complexity physical environments Example: - Fails in OGBench-Cube (more complex simulation environment) 8.2. No Real Robot Deployment - Not tested on physical robotic systems ## 9. Conclusion LeWM demonstrates: > JEPA + SIGReg is a viable paradigm Key achievements: - Stable training from raw pixels - Collapse eliminated via principled regularization - Physical laws encoded in latent space - Real-time planning capability > Scaling requires: - larger models - more expressive architectures - real-world validation ## References [1] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. https://arxiv.org/pdf/2603.19312 [2] 🦤 135. LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic ## 相关链接 - [Junfan Zhu 朱俊帆](https://x.com/junfanzhu98) - [@junfanzhu98](https://x.com/junfanzhu98) - [15K](https://x.com/junfanzhu98/status/2047029352995639341/analytics) - [https://arxiv.org/pdf/2603.19312](https://arxiv.org/pdf/2603.19312) - [https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic](https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [3:06 AM · Apr 23, 2026](https://x.com/junfanzhu98/status/2047029352995639341) - [15.2K Views](https://x.com/junfanzhu98/status/2047029352995639341/analytics) - [View quotes](https://x.com/junfanzhu98/status/2047029352995639341/quotes) --- *导出时间: 2026/4/23 12:08:30* --- ## 中文翻译 # 135. LeWorldModel：从像素中学习物理——仅用两个损失项实现稳定的世界模型 **作者**: Junfan Zhu 朱俊帆 **日期**: 2026-04-22T19:06:44.000Z **来源**: [https://x.com/junfanzhu98/status/2047029352995639341](https://x.com/junfanzhu98/status/2047029352995639341) --- ![Image](/doc/987/image/img_75a3000a06ba.jpg) ![Image](/doc/987/image/img_a3543f38942d.jpg) 1. 世界模型领域的现状 ## 1.1 DINO-WM DINO-WM 使用预训练的 Vision Transformer（ViT）编码器将视频帧转换为潜在特征，然后在这些特征之上训练一个隐式预测器。然而，编码器是冻结的，无法进行端到端训练。这导致一个根本性的后果： > 预测器被迫学习一个被截断的（信息受限的）世界模型。一个贴切的类比（这反映的是技术局限，而不仅仅是直觉）： - 这个模型就像一个只能读别人笔记的学生， - 却从不被允许独立解题。 ## 表征偏差问题（关键技术点）该编码器是在 ImageNet 上预训练的。这引入了一种表征先验（“视觉基因”）： - 它是为粗粒度语义分类（例如猫与狗）优化的， - 而不是为细粒度的物理分辨率设计的。具体的失效模式： > 它可以正确分类物体，但无法解析毫米尺度的物理变化，例如木块 2 毫米的位移。因此： - 该编码器就像戴着一副在 ImageNet 上训练出来的高度近视眼镜。 - 它从根本上无法“看到”微观层面的物理状态变化。关键启示： > 无论预测器多么强大，如果它连接的是一个无法感知细粒度物理现实的编码器，整个系统实际上就是在“盲目推理物理”（盲人摸象）。 ## 1.2. 预测性潜在动态模型（PLDM） PLDM 是可端到端训练的。然而，这带来了两个严格的限制： (1) 训练不稳定 - 训练不稳定 - 容易崩溃 (2) 奖励信号约束（硬性约束） PLDM 使用奖励信号作为预测目标。因此： > 它只能在具有明确定义奖励信号的环境（例如游戏）中运行。这不是“效果更好”——而是一个硬性的适用性约束： - 没有明确定义的奖励 → 该方法不适用。 ## 1.3. 联合嵌入预测架构（JEPA） JEPA 消除了对像素重建的需求。训练目标： - 编码当前帧 → 嵌入 - 预测下一步嵌入 - 确保预测的延迟嵌入 ≈ 实际的延迟嵌入然而，JEPA 有两个尚未解决的核心问题： (1) 崩溃问题编码器可能输出一个平凡解： - 将所有输入映射 → 同一个常量向量（例如全零）这使预测变得平凡，但摧毁了所有信息。 (2) 像素级端到端训练 + 稳定性（联合约束）挑战不仅在于“原始像素”： > 系统必须同时实现： - 像素级输入 - 端到端训练 - 稳定优化（不崩溃）大多数 JEPA 变体无法同时满足这三点。 2. LeWM：从原始像素出发的极简、稳定 JEPA LeWorldModel 是首个满足以下条件的 JEPA： > 直接从原始像素进行稳定的端到端训练，仅使用两个损失项：下一嵌入预测损失一个高斯分布正则化项（SIGReg） ## 2.1 从原始像素进行端到端训练与 DINO-WM 不同： - 编码器不被冻结 - 感知（编码器）和认知（预测器）通过从输入到输出的单一统一数据管线进行训练这消除了 ImageNet 带来的“近视偏差”。洞见： > 编码器的“视觉基因”决定了它能感知什么样的物理结构。因此： - 用分类任务训练的编码器 → 只能获得粗粒度语义 - 联合训练的编码器 → 与物理推理共同演化解读： > LeWM 摘掉了“近视眼镜”，让视觉系统与物理直觉同步共同演化。这才是端到端训练的真正含义，也是实现物理理解的必要条件。 ## 2.2 “仅有”一个超参数 PLDM 需要 6 个超参数。 LeWM 只需要一个： - λ（SIGReg 的权重）因为： > SIGReg 具有内在的鲁棒性，不需要精细调参。结果： - 即插即用式训练 - 无需复杂的超参数搜索即可稳定收敛 3. JEPA 中的崩溃问题 JEPA 目标： - 编码器：( x_t \rightarrow z_t ) - 预测器：( (z_t, a_t) \rightarrow \hat{z}_{t+1} ) - 最小化：( | \hat{z}{t+1} - z{t+1} | ) 退化解： > 编码器输出一个常量向量（例如全零）那么： - 预测器每次看到的输入都相同 - 预测变得平凡 - 损失被最小化这就是通过平凡解导致的崩溃。 4. SIGReg：高斯积分签名正则化高斯积分签名正则化 ## 4.1 核心直觉（影子 / 签名解读） - “签名” = 高维对象的投影（影子） - 高斯分布 = 非退化的检验如果所有投影都是高斯的： > 原始的高维嵌入不可能坍缩到一个单点因为： - 坍缩的向量 → 所有投影都是常量（退化），而非高斯 - 高斯要求：方差、离散度、样本多样性 ## 4.2 数学基础 Cramér–Wold 定理 > 一个高维分布完全由其一维投影的分布所决定。因此： - 无需检验完整分布 - 检验足够多的一维投影即可 ## 4.3 SIGReg 流程从标准正态分布中采样 1024 个随机方向将每个嵌入向量投影到全部 1024 个方向上对每个投影：执行 Epps–Pulley 统计检验。验证边缘分布接近标准正态分布。检验的目标是：检验统计量 → 0（完美的高斯匹配）。将所有投影测试结果汇总为一个正则化损失项将该损失项加入整体训练目标效果： > 如果任何投影（“影子”）未通过高斯测试，编码器将被显式惩罚因此： - 编码器无法走捷径 - 每个投影都必须“有效” ## 4.4 超参数只有一个： - λ（SIGReg 的权重）因为 SIGReg 高度鲁棒： - 无需微调 - 在各种设置下都稳定 ## 5. 物理探测实验在 PushT 环境中训练 LeWM：任务：将一个方块推到目标位置 ## 5.1 线性探测训练完成后： - 冻结模型 - 在嵌入上训练一个线性回归器来预测：位置、朝向、机器人末端执行器位置如果嵌入编码了物理信息： > 线性模型应能恢复出准确的物理量。结果： - 成功 → 嵌入编码了结构化的物理信息 ## 5.2 分布外干预干预： > 将方块传送到物理上不可能的位置观察： > 嵌入空间的异常值急剧飙升（大规模激增，而非轻微偏差）解读： - 模型已经学到：物体不能瞬移关键区别： > 这一约束被内化在潜在表示中，而不是直接从像素级表面特征读取。 ## 5.3 洞见这是根本区别： - 视频预测：预测像素 - 世界模型：编码物理约束与不变量 6. 涌现的时间直线性重要事实： > 损失函数中没有任何显式的时间平滑项。模型从未被告知： - “预测应在时间上平滑” 然而观察到： > 在嵌入空间中，两个状态之间的轨迹近似为直线 ## 6.1 解读 - 模型并非在幻觉出平滑过渡 - 它在学习物理上有效的位移轨迹核心论断： > 时间直线性是在没有任何先验（无归纳偏置）的情况下涌现的 ## 6.2 对控制的意义如果世界模型产生： - 平滑且物理一致的潜在轨迹那么： - 基于该模型的规划是可靠的 - 避免了由模型伪影导致的不现实轨迹 7. 性能规划时间 - LeWM：0.98 秒 - DINO-WM：47 秒成功率 - PLDM：78% - LeWM：96% ## 7.1 为什么 DINO-WM 慢原因：编码器被冻结 → 表示丢失了与任务相关的信息 → 需要额外的在线传播步骤 → 以补偿信息损失 LeWM： - 端到端训练的编码器 - 表示天然与预测任务对齐 - 无需额外计算 ## 7.2 现实意义速度决定了机器人的可部署性： - 0.98 秒：可以应对动态障碍物，可以实时对意外事件作出反应 - 47 秒：无法部署在真实控制回路中 8. 局限性 8.1 模型规模 - 约 1500 万参数 - “小如蚂蚁”（即严重欠扩展）后果： > 无法建模高复杂度的物理环境例如： - 在 OGBench-Cube（更复杂的仿真环境）中失败 8.2. 未进行真实机器人部署 - 未在实体机器人系统上测试 ## 9. 结论 LeWM 证明了： > JEPA + SIGReg 是一种可行的范式关键成果： - 从原始像素实现稳定训练 - 通过有原则的正则化消除了坍缩 - 物理定律被编码在潜在空间中 - 实时规划能力 > 扩展需要： - 更大的模型 - 更具表达力的架构 - 真实世界验证 ## 参考文献 [1] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. https://arxiv.org/pdf/2603.19312 [2] 🦤 135. LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic ## 相关链接 - [Junfan Zhu 朱俊帆](https://x.com/junfanzhu98) - [@junfanzhu98](https://x.com/junfanzhu98) - [15K](https://x.com/junfanzhu98/status/2047029352995639341/analytics) - [https://arxiv.org/pdf/2603.19312](https://arxiv.org/pdf/2603.19312) - [https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic](https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic) - [升级至 Premium](https://x.com/i/premium_sign_up) - [2026年4月23日凌晨3:06](https://x.com/junfanzhu98/status/2047029352995639341) - [15.2K 次浏览](https://x.com/junfanzhu98/status/2047029352995639341/analytics) - [查看引用](https://x.com/junfanzhu98/status/2047029352995639341/quotes) --- *导出时间: 2026/4/23 12:08:30*