# 135. LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses
**作者**: Junfan Zhu 朱俊帆
**日期**: 2026-04-22T19:06:44.000Z
**来源**: [https://x.com/junfanzhu98/status/2047029352995639341](https://x.com/junfanzhu98/status/2047029352995639341)
---
1. Current Landscape of World Models
## 1.1 DINO-WM
DINO-WM uses a pretrained Vision Transformer (ViT) encoder to convert video frames into latent features, and then trains an implicit predictor on top of these features.
However, the encoder is frozen and cannot be trained end-to-end.
This leads to a fundamental consequence:
> The predictor is forced to learn a truncated (information-limited) world model.
A precise analogy (which reflects a technical limitation, not just intuition):
- The model is like a student who can only read someone else’s notes,
- But is never allowed to solve problems independently.
## Representation Bias Problem (Critical Technical Point)
The encoder is pretrained on ImageNet.
This introduces a representation prior (“visual genetics”):
- It is optimized for coarse-grained semantic classification (e.g., cats vs. dogs),
- Not for fine-grained physical resolution.
Concrete failure mode:
> It can correctly classify objects, but cannot resolve physical changes at millimeter scale, e.g., a 2 mm displacement of a wooden block.
Therefore:
- The encoder behaves like wearing highly myopic glasses trained on ImageNet.
- It fundamentally cannot “see” micro-level physical state changes.
Key implication:
> No matter how powerful the predictor is, if it is attached to an encoder that cannot perceive fine-grained physical reality, the system is effectively “blind reasoning about physics” (blind men touching an elephant).
## 1.2. Predictive Latent Dynamic Model (PLDM)
PLDM is end-to-end trainable.
However, this comes with two strict limitations:
(1) Training Instability
- Training is unstable
- Easily collapses
(2) Reward Signal Constraint (Hard Constraint)
PLDM uses reward signals as prediction targets.
Therefore:
> It can only operate in environments with explicitly defined reward signals (e.g., games).
This is not “works better”—it is a hard applicability constraint:
- Without a well-defined reward → the method is not applicable.
## 1.3. Joint Embedding Predictive Architecture (JEPA)
JEPA removes the need for pixel reconstruction.
Training objective:
- Encode current frame → embedding
- Predict next-step embedding
- Ensure predicted delayed embedding ≈ actual delayed embedding
However, JEPA has two unresolved core problems:
(1) Collapse Problem
The encoder may output a trivial solution:
- Map all inputs → the same constant vector (e.g., all zeros)
This makes prediction trivial, but destroys all information.
(2) Pixel-Level End-to-End Training + Stability (Joint Constraint)
The challenge is not just “raw pixels”:
> The system must simultaneously achieve:
- pixel-level input
- end-to-end training
- stable optimization (no collapse)
Most JEPA variants fail to satisfy all three simultaneously.
2. LeWM: Minimal, Stable JEPA from Raw Pixels
LeWorldModel is the first JEPA that satisfies:
> Stable end-to-end training directly from raw pixels, using only two loss terms:
Next-embedding prediction loss
A Gaussian-distribution regularizer (SIGReg)
## 2.1 End-to-End Training from Raw Pixels
Unlike DINO-WM:
- The encoder is not frozen
- Perception (encoder) and cognition (predictor) are trained through a single unified data pipeline from input to output
This removes the ImageNet-induced “myopic bias”.
Insight:
> The encoder’s “visual genetics” determines what physical structure it can perceive.
Thus:
- A classification-trained encoder → coarse semantics only
- A jointly trained encoder → evolves together with physical reasoning
Interpretation:
> LeWM removes the “myopic glasses” and allows the visual system and physical intuition to co-evolve synchronously.
This is the true meaning of end-to-end training, and a necessary condition for physical understanding.
## 2.2 “Only” One Hyperparameter
PLDM requires 6 hyperparameters.
LeWM requires exactly one:
- λ (weight of SIGReg)
Because:
> SIGReg is intrinsically robust, and does not require delicate tuning.
Result:
- Plug-and-play training
- Stable convergence without complex hyperparameter search
3. Collapse Problem in JEPA
JEPA objective:
- Encoder: ( x_t \rightarrow z_t )
- Predictor: ( (z_t, a_t) \rightarrow \hat{z}_{t+1} )
- Minimize: ( | \hat{z}{t+1} - z{t+1} | )
Degenerate solution:
> Encoder outputs a constant vector (e.g., all zeros)
Then:
- Predictor sees identical inputs every time
- Prediction becomes trivial
- Loss is minimized
This is collapse via trivial solution.
4. SIGReg: Gaussian Integral Signature Regularization
Gaussian Integral Signature Regularization
## 4.1 Core Intuition (Shadow / Signature Interpretation)
- “Signature” = projection (shadow) of a high-dimensional object
- Gaussian distribution = test of non-degeneracy
If all projections are Gaussian:
> The original high-dimensional embedding cannot collapse to a single point
Because:
- A collapsed vector → all projections are constant (degenerate), not Gaussian
- Gaussian requires: variance, dispersion, sample diversity
## 4.2 Mathematical Foundation
Cramér–Wold theorem
> A high-dimensional distribution is fully determined by the distributions of its 1D projections.
Thus:
- No need to test full distribution
- Testing sufficiently many 1D projections is sufficient
## 4.3 SIGReg Procedure
Sample 1024 random directions from a standard normal distribution
Project each embedding vector onto all 1024 directions
For each projection: Perform Epps–Pulley statistical test. Verify that the marginal distribution is close to standard normal.
The target of the test is: The test statistic → 0 (perfect Gaussian match).
Aggregate all projection test results into a regularization loss term
Add this term to the overall training objective
Effect:
> The encoder is explicitly penalized if any projection (“shadow”) fails the Gaussian test
Thus:
- The encoder cannot take shortcuts
- Every projection must be “valid”
## 4.4 Hyperparameter
Only one:
- λ (weight of SIGReg)
Because SIGReg is highly robust:
- No fine-tuning required
- Stable across settings
## 5. Physical Probing Experiment
Train LeWM in a PushT environment: Task: push a block to a target location
## 5.1 Linear Probe
After training:
- Freeze the model
- Train a linear regressor on embeddings to predict: position, orientation, robot end-effector position
If embeddings encode physics:
> A linear model should recover accurate physical quantities.
Result:
- Successful → embeddings encode structured physical information
## 5.2 Out-of-Distribution Intervention
Intervention:
> Teleport the block to a physically impossible position
Observation:
> Embedding-space anomaly values spike sharply (large-scale surge, not minor deviation)
Interpretation:
- The model has learned that: Objects cannot teleport
Critical distinction:
> This constraint is internalized in the latent representation, NOT directly read from pixel-level surface features.
## 5.3 Insight
This is the fundamental difference:
- Video prediction: predicts pixels
- World models: encode physical constraints and invariants
6. Emergent Temporal Straightness
Important fact:
> There is NO explicit temporal smoothness term in the loss.
Model is not told of:
- “predictions should be temporally smooth”
Yet observed:
> In embedding space, trajectories between two states are approximately straight lines
## 6.1 Interpretation
- The model is NOT hallucinating smooth transitions
- It is learning physically valid displacement trajectories
Key claim:
> Temporal straightness emerges without any prior (no inductive bias)
## 6.2 Implication for Control
If the world model produces:
- smooth and physically consistent latent trajectories
Then:
- planning based on this model is reliable
- avoids unrealistic trajectories caused by model artifacts
7. Performance
Planning Time
- LeWM: 0.98 seconds
- DINO-WM: 47 seconds
Success Rate
- PLDM: 78%
- LeWM: 96%
## 7.1 Why DINO-WM Is Slow
Because:
Encoder is frozen
→ Representation loses task-relevant information
→ Requires additional online propagation steps
→ To compensate for information loss
LeWM:
- End-to-end trained encoder
- Representation is natively aligned with prediction task
- No extra computation required
## 7.2 Real-World Implication
Speed determines deployability in robotics:
- 0.98 seconds: Can handle dynamic obstacles, Can react to unexpected events in real time
- 47 seconds: Not deployable in real control loops
8. Limitations
8.1 Model Size
- ~15M parameters
- “Small as an ant” (i.e., severely under-scaled)
Consequence:
> Cannot model high-complexity physical environments
Example:
- Fails in OGBench-Cube (more complex simulation environment)
8.2. No Real Robot Deployment
- Not tested on physical robotic systems
## 9. Conclusion
LeWM demonstrates:
> JEPA + SIGReg is a viable paradigm
Key achievements:
- Stable training from raw pixels
- Collapse eliminated via principled regularization
- Physical laws encoded in latent space
- Real-time planning capability
> Scaling requires:
- larger models
- more expressive architectures
- real-world validation
## References
[1] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. https://arxiv.org/pdf/2603.19312
[2] 🦤 135. LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic
## 相关链接
- [Junfan Zhu 朱俊帆](https://x.com/junfanzhu98)
- [@junfanzhu98](https://x.com/junfanzhu98)
- [15K](https://x.com/junfanzhu98/status/2047029352995639341/analytics)
- [https://arxiv.org/pdf/2603.19312](https://arxiv.org/pdf/2603.19312)
- [https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic](https://www.linkedin.com/pulse/135-leworldmodel-learning-physics-from-pixels-stable-world-ieaic)
- [Upgrade to Premium](https://x.com/i/premium_sign_up)
- [3:06 AM · Apr 23, 2026](https://x.com/junfanzhu98/status/2047029352995639341)
- [15.2K Views](https://x.com/junfanzhu98/status/2047029352995639341/analytics)
- [View quotes](https://x.com/junfanzhu98/status/2047029352995639341/quotes)
---
*导出时间: 2026/4/23 12:08:30*