因子图与“世界模型”：统一的感知与规划框架

✍ Frank Dellaert🕐 2026-04-24📦 13.0 KB 🟢 已读 𝕏 文章列表

文章探讨了世界模型与因子图之间的深层联系。作者指出，因子图本质上是基于能量的模型，可用于状态估计和SLAM。通过将世界模型锚定在“现在”，文章提出了STAG框架，该框架利用统一的因子图优化机制，同时处理基于过去数据的感知和基于未来目标的规划，实现了感知与行动的无缝融合。

世界模型因子图机器人 SLAM 规划感知能量模型 STAG 优化论文

# Factor Graphs and "World Models" **作者**: Frank Dellaert **日期**: 2026-04-22T16:55:03.000Z **来源**: [https://x.com/fdellaert/status/2046996213669450096](https://x.com/fdellaert/status/2046996213669450096) --- ![Image](/doc/1035/image/img_42208567ae8b.jpg) There is a lot of excitement right now around world models: learned latent-state representations that support prediction, imagination, and planning. The recent work around JEPA is especially interesting, as it aims to learn useful latent dynamics that can support downstream decision making and planning, as in the recent LeWorldModel [3] paper, and more broadly in @ylecun's 2022 position paper A Path Towards Autonomous Machine Intelligence [2]. This connects naturally to Yann's earlier work on energy-based models [1], where the central idea is not to generate outputs one token at a time, but to define an energy landscape and then minimize it. The jury is still out, but I’m a fan 🙂. More importantly, from a factor-graph point of view, this all feels very familiar. ## 1. Factor graphs are energy-based models A factor graph defines a probability distribution up to normalization: p(x | z) ∝ ∏ᵢ ϕᵢ(xᵢ) where each factor ϕᵢ expresses a local probabilistic relationship among some subset of variables xᵢ, conditioned on measurements z. Taking the negative log gives an energy: E(x) = −log p(x | z) = Σᵢ Eᵢ(xᵢ) + constant Hence, maximum a posteriori inference is energy minimization. That observation has been the basis of state estimation, smoothing, and SLAM for a long time. In robotics, we already know how to represent dynamics, measurements, priors, and constraints as factors. We already know how to optimize the resulting energy efficiently. In that sense, factor graphs are a very concrete and highly structured form of energy-based world modeling, exactly the perspective developed in the Dellaert and Kaess’s "Factor Graphs for Robot Perception" booklet [4]. ## 2. From estimation to planning ![Image](/doc/1035/image/img_7c785282cef7.jpg) The Vector mobile manipulator, with an omni-drive base and a 6-DOF Kinova JACO2 arm, is solving the STEAP problem. The task involves picking up an object from the white table on the right and dropping it off on the white table on the left. The semi-transparent robots show the trajectory taken, while the solid robot is the goal configuration In our earlier STEAP work [5] (Simultaneous Trajectory Estimation And Planning), we extended the same factor-graph machinery to planning. There, estimation and planning were treated in a unified optimization framework over an entire trajectory. The same graph could encode what the robot had observed, how it could move, and what trajectory it should follow to get from point a to point b. That was an important step, but it still assumes a fixed interval [a, b]: there is a start, there is a goal, and we optimize over the trajectory connecting the two. The more interesting question is what happens if we anchor the whole construction at the current time t, rather than thinking in terms of one fixed offline interval. ## 3. Anchor the graph at “now” Suppose we center everything around the present time t. Then we can think of the robot’s world as split into two closely related factor graphs: - a graph over the recent past, used for perception and state estimation - a graph over the near future, used for planning and control Both graphs share the same state representation and the same dynamics model. The difference is what information they are tied to: - the past is tied to measurements - the future is tied to objectives This is exactly the kind of structure one would expect from a practical world model: infer the hidden state from recent evidence, then optimize the likely future under the same transition model. ## 4. Perception as fixed-lag smoothing ![Image](/doc/1035/image/img_0d2d40823736.jpg) A fixed-lag smoothing graph for perception. Each dynamics factor connects neighboring states together with a known control input, while unary measurement factors summarize the sensor evidence over the last five seconds. For perception, consider a factor graph over the last five seconds: xₜ₋₅, xₜ₋₄, …, xₜ On this graph we place: - dynamics factors between successive states, each coupled to a known control input uₖ - unary measurement factors that connect each state to the relevant sensor observations Optimizing this graph gives a fixed-lag smoother. Rather than filtering only forward, we re-estimate the recent trajectory using all measurements within the lag window. That gives a better estimate of the current state xₜ, while still keeping the computation bounded and online. In this inference problem, the controls over the recent past are part of the observed history: commanded wheel velocities, torques, or thrusts are already known, and so they enter the graph as known variables attached to the dynamics factors. This is the standard factor-graph view of perception: the robot’s recent past is inferred by minimizing an energy defined by dynamics plus measurements. ## 5. Planning as a future factor graph ![Image](/doc/1035/image/img_b7ae307ecbf5.jpg) A model-predictive control (MPC) graph for planning. The *same* dynamics factors connect the future states, but now the controls are optimization variables, and the unary terms are goal or objective factors rather than measurements. Now look at the next five seconds: xₜ, xₜ₊₁, …, xₜ₊₅ Again, we use the same dynamics factors. That is important: the robot should think about the future using the same transition structure with which it interprets the past. But in the planning graph, the controls are no longer known. They are optimized jointly with the future states over the horizon. This is the familiar collocated optimal control view: states and controls live on the same time grid and are solved for together. The unary factors are also different. Instead of measurement factors, we introduce goal or objective factors. Let us denote these as gₜ₊₁, …, gₜ₊₅ over the horizon. These could represent: - goal-reaching costs - collision-avoidance costs - control-effort penalties - stability or balance objectives - task-specific preferences Optimizing this future graph gives a trajectory in the familiar model-predictive control style: solve over a short horizon, execute the first action, move forward in time, then rebuild and re-optimize. The structure is almost the mirror image of perception. In the past, unary factors tell us what the sensors observed. In the future, unary factors tell us what we want. ## 6. STAG: Sense-Think-Act via factor graphs ![Image](/doc/1035/image/img_736085b51203.jpg) The combined picture: a single rolling factor graph centered on the current state. The left half explains the recent past from measurements and known controls, while the right half plans the near future with goal factors and optimized controls. Put these two pieces together and a simple picture emerges: - Sense: gather the recent evidence from measurements and controls - Think: optimize factor graphs over both the recent past and the near future - Act: execute the first part of the optimized future trajectory This is really Sense-Think-Act with factor Graphs. Both sensing and planning are carried out by optimization over factors; "thinking" is the optimization layer that spans both. Let us call it STAG. The appealing part is that STAG does not require a sharp separation between “perception” and “planning”. Both are instances of the same underlying idea: define a state write down dynamics add factors minimize energy Past and future are handled by nearly the same computational machinery, just with different unary terms. One graph explains evidence. The other graph expresses intent. The picture is clear: a single rolling graph spans both the recent past and near future, anchored at the current state xₜ. On the left are measurement factors. On the right are goal factors gₜ₊₁, …, gₜ₊₅. In the middle sits the current latent state, tying perception and planning together. ## 7. Open Questions Once stated this way, the core questions for any robot become very clear. For a given platform, what do we need to define, or perhaps learn? What is the state? Is it pose and velocity? Joint angles and contact models? A full belief state over robot and environment? Can we learn this representation? What is the dynamics model? Is it analytic, learned, or hybrid? Is it first-order, second-order, or manifold-valued? What is the measurement model? How do cameras, IMUs, force sensors, wheel encoders, or tactile signals connect to the latent state? What are the objectives for the future? What should the robot optimize for over the next few seconds? Is this where reinforcement learning enters the picture? One could also take issue with the specific topology of the factor graphs and whether they should even be probabilistic at all. Conditional random fields (CRFs), for example, do not insist on having unary factors for perception. And in general, joint states in the past and future could be hooked up to neural networks that measure their energy. Finally, optimization could be taken to be more general: sampling based methods like MPPI might just work just as well or better. I don’t have answers to all these questions, but we’ve started to play around with this model, and I think it is an interesting direction. ## Summary The current interest in world models, JEPA, and energy-based models is exciting. But from the perspective of factor graphs, there is also a deep continuity with ideas that robotics and estimation have used for years. Factor graphs are already energy-based models. Their negative log likelihood is already an energy. State estimation and SLAM have long exploited this fact. STEAP showed that planning can live in the same optimization framework. The next step is to anchor everything at the present and view the robot as carrying two tightly coupled graphs: - a past graph for perception - a future graph for planning Using the same state and the same dynamics on both sides gives a simple and powerful recipe for online robot intelligence: both sensing and planning are carried out by optimization over factors, and "thinking" is the optimization layer that spans them. We act on the resulting plan. That is STAG. ## Acknowledgements The STEAP work was done with Mustafa Mukadam, Jing Dong and Byron Boots. Recent thinking in this space benefited a lot from discussions with Andrew Wong, Aaron Bong, Yuxin Yang, Ali Hasnain, and Harish Ravichandrar. ## References "A Tutorial on Energy-Based Learning", Yann LeCun, Sumit Chopra, Raia Hadsell, Marc'Aurelio Ranzato, Fu-Jie Huang, in *Predicting Structured Outputs*, MIT Press, 2006. A Path Towards Autonomous Machine Intelligence, Yann LeCun, position paper, version 0.9.2, June 27, 2022. “LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels”, Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero, arXiv, March 13, 2026. “Factor Graphs for Robot Perception”, Frank Dellaert, Michael Kaess, Foundations and Trends in Robotics, 2017, Vol. 6, No. 1-2, pp. 1-139. “STEAP: Simultaneous Trajectory Estimation and Planning”, Mustafa Mukadam, Jing Dong, Frank Dellaert, Byron Boots, Autonomous Robots, 2019, Vol. 43, No. 2, pp. 415-434. This article was also posted on the GTSAM blog. ## 相关链接 - [Frank Dellaert](https://x.com/fdellaert) - [@fdellaert](https://x.com/fdellaert) - [22K](https://x.com/fdellaert/status/2046996213669450096/analytics) - [LeWorldModel [3] paper](https://doi.org/10.48550/arXiv.2603.19312) - [@ylecun](https://x.com/@ylecun) - [A Path Towards Autonomous Machine Intelligence](https://openreview.net/pdf?id=BZ5a1r-kVsf) - [Yann's earlier work on energy-based models](https://yann.lecun.org/exdb/publis/pdf/lecun-06.pdf) - [Dellaert and Kaess’s "Factor Graphs for Robot Perception" booklet [4].](https://doi.org/10.1561/2300000043) - [STEAP](https://doi.org/10.1007/s10514-018-9770-1) - ["A Tutorial on Energy-Based Learning"](https://yann.lecun.org/exdb/publis/pdf/lecun-06.pdf) - [A Path Towards Autonomous Machine Intelligence](https://openreview.net/pdf?id=BZ5a1r-kVsf) - [“LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels”](https://doi.org/10.48550/arXiv.2603.19312) - [“Factor Graphs for Robot Perception”](https://doi.org/10.1561/2300000043) - [“STEAP: Simultaneous Trajectory Estimation and Planning”](https://doi.org/10.1007/s10514-018-9770-1) - [GTSAM blog](https://gtsam.org/2026/04/21/factor-graphs-and-world-models.html) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [12:55 AM · Apr 23, 2026](https://x.com/fdellaert/status/2046996213669450096) - [22.7K Views](https://x.com/fdellaert/status/2046996213669450096/analytics) - [View quotes](https://x.com/fdellaert/status/2046996213669450096/quotes) --- *导出时间: 2026/4/24 10:56:53*

🔗 相关文档