How Should World Models Reason About What They Have Seen?

JACK: Joint Axis Correction from Kinesthetic Feedback

Blog post by Arthur Jakobsson

Read the JACK extended abstract

Accompanying extended abstract (JACK): Arthur Jakobsson, Eugene Lee, Bardienus Pieter Duisterhof, Huang Huang, Jeff Tan, Romina Mir, Jonathan Francis, Jean Oh, and Jeffrey Ichnowski.

We've all had this experience: you pull on a door handle and it doesn't budge... but it sure looks like it should be a pull door. You push it, and now it opens. This is "bad design" when humans operate it, but it's an impasse for robots. In this work, we explore how we can get robots to learn from interactions and perform interactions to learn information about the scene.

A robot opening an unfamiliar cabinet has to work out what went wrong with its first attempt. Did it pull in the wrong direction, misjudge the hinge, or, even more confusing for it, encounter a locked door? Its next action should help it distinguish these possibilities. That requires a model of how the hinge could and should move given the robot's current understanding of it.

Video: Vox and 99% Invisible. Watch it on YouTube if the embed does not load.

For complex physical interactions, I think robots need world models that adapt as they act. A robot may have seen a cabinet many times without knowing where its hinges are or whether a drawer is stuck. Moving a part, and sometimes failing to move a part, gives the model information that we can leverage for a better understanding of the world.

In JACK (Joint Axis Correction from Kinesthetic Feedback), we describe this model as an explicit, revisable belief about an object’s joints: whether each joint rotates or slides, where its axis lies, and how confident the robot should be in that estimate. A push can test the estimate, and a planner can use it to choose a motion. We organize the approach into belief, act, and observe. Representing the joints is the easier part. The harder problem is choosing a push that will distinguish between the joint estimates the robot still considers possible.

Vision ObservationpriorBeliefchooseActionsact &observeObservationsupdate

Observations update the robot’s beliefs, which guide its next action.

Belief

Suppose a robot sees a cabinet and estimates the axis of a door hinge. That estimate is useful, but it is still a hypothesis. A small error in the axis may make the robot push against the door’s actual constraint. An unfamiliar mechanism may even violate the assumed joint type. A world model for this task should carry more than one coordinate: it should represent plausible joint types, axes, and pivots, the evidence for each, and how much remains unknown.

This ambiguity has motivated other approaches. FlowBotHD uses a history-aware diffusion model to represent more than one plausible articulation mode, including cases where a closed door could be pushed, pulled, or slid [1].

In JACK, we start with a visual estimate for each joint and treat it as a prior to be tested. We propose representing this belief as a mixture of explanations: the initial hypothesis, geometric candidates fitted to observed motion, the belief carried from earlier interactions, and an outlier possibility. An initial guess can be far enough off that it needs replacing rather than a small adjustment. A meaningful confidence estimate would tell the robot both when to trust the current joint estimate and when another observation is worth obtaining.

Even a useful visual model can leave decision-relevant facts unresolved. Kim, Nakamura, and Bajcsy’s study of partially observable safety constraints distinguishes not knowing whether a system is safe now from failing to predict a violation before it happens [2].

Act

Once a robot can say what it does not know, it can choose an action for the evidence it is likely to produce. That is the central idea of interactive perception that I want to build on. Actions can help a robot sense, and we should recouple actions and perception instead of first observing and then acting [3]. In JACK, we design the investigator to select an interaction expected to reduce uncertainty about the least certain joint. Each action is designed as a small experiment that the model proposes to learn more about the system it is interacting with. Similarly, ASID chooses robot exploration to identify physical parameters [4], and Plan2Explore uses disagreement among learned predictors to seek experiences expected to improve a world model [5].

Simulation render of a robot arm moving an articulated mast-like structure, with faint earlier arm positions showing its motion.

JACK simulation render visualizing multiple actions performed on an articulated mast to identify joint locations.

(a) hypothesis(b) interaction(c) corrected beliefhypothesis axis and 90% conecandidate fitted to the motiontracked surface points

Our proposed sequence in JACK: a joint hypothesis, a motion that tests it, and a revised belief. Diagram from the workshop extended abstract; it illustrates the method, not measured performance.

Observe

The important evidence is not simply that the scene changed. It is how a part moved in response to the robot’s action. We design our geometric front end in JACK to track points on the surface, group points that move together, and fit plausible rotational or sliding joints to those motions. When action and perception are coupled, we gain more information than when they are considered independently because the action is designed to induce specific, observable responses.

As objects get more complex, so do the sensing methods required to understand them. Multiple studies have demonstrated the benefit of combining sensing modalities. Buchanan et al., for example, combine vision and proprioceptive sensing for online articulation estimation [6].

In JACK, we then use a belief transformer to weigh these candidates together with the earlier hypothesis and the history of interaction. We want it to determine which physical explanation the available evidence supports and how uncertain that choice remains. While most actions are likely to move joints, we intend them to reduce uncertainty about the scene. For example, if the model wants to determine a joint’s limits, it may intentionally push against a limit and use tactile feedback to identify it.

hypothesis per jointaxis, pivot, type, limits,or unknownscene and wrist RGB-Dwhere the parts aretactile pads, effortcontact, slip, stall,resistancepoint trackshow the surface movedgeometric front endrigid grouping, screw fitsworld & parent-relativebelief transformerone token per joint,per keyframeand per candidate;attention over thewhole prefix, re-runat every keyframemixture over explanationshypothesis + residual, each fit,carried belief, outlier; spreadstype and limitsdoubt headspread drop per candidate actionnext action: the investigator picks the skill the doubt head predicts will shrink the belief fastest

Our proposed model in JACK. Geometric fits and multimodal observations inform a belief transformer; a predicted reduction in uncertainty can guide the next action. Diagram from the workshop extended abstract.

The belief should improve as observations accumulate. If the robot tries one joint and sees only a few millimetres of motion, that evidence may be weak. A later action may make the axis clearer. In JACK, we retain the earlier hypothesis and the evidence from each interaction when updating the estimate. Our investigator can choose a deliberate probe when uncertainty is high, and it can stop probing once the model is adequate for the task. This is a choice the robot can make, not a required sequence that every action must follow.

There are useful precedents for learning during interaction, though they do not maintain the explicit belief over an individual object’s joints that we propose in JACK. DayDreamer learns a world model online from real robot experience [7]. PlayPolicy uses a learned play policy to gather interaction data for a world model, then uses that model to evaluate candidate actions [8]. I want to connect continued learning like this to a more concrete account of what the robot has learned about the particular object in front of it.

There is substantial prior work behind this loop. World modeling, learning through contact, and choosing informative actions are established research directions. In JACK, we bring them together around a specific question: how should a robot revise an explicit geometric belief about an articulated mechanism? The following threads are representative, rather than a complete survey.

Learning models for prediction and control. Ha and Schmidhuber’s World Models learns compact visual and temporal representations for control [9], and DreamerV3 learns behavior through imagined trajectories in a learned model [10]. The representation need not be a latent vector: Li and colleagues learn particle dynamics for rigid bodies, deformable materials, and fluids [11]. Plan2Explore [5], DayDreamer [7], and PlayPolicy [8] connect model learning to exploration or continued physical interaction in different ways. We propose a narrower role for our belief in JACK: make the joint type, axis, and pivot explicit, and revise them from observed part motion.

Interactive perception and uncertain mechanisms. Bohg and colleagues’ survey [3] places action inside the perception process. For articulated objects, Sturm, Stachniss, and Burgard fit probabilistic kinematic models to noisy observations [12]. Hausman and colleagues maintain a belief over articulation models and select actions to reduce uncertainty [13]. These are direct precedents for treating a joint estimate as a hypothesis and an action as a test. ASID [4] extends informative exploration to identifying physical parameters for simulation and control. We build on this work on learning by probing. Our proposed contribution is how we weigh geometric candidates, a visual prior, and accumulated multimodal evidence in a learned belief.

Reconstructing objects through interaction. Ditto builds geometry and articulation from observations before and after an interaction [14]. H-SAUR repeatedly hypothesizes, simulates, acts, and updates to understand articulation [15]. FlowBotHD [1] uses interaction history to handle ambiguous articulation modes, while Buchanan and colleagues [6] fuse visual priors and proprioceptive sensing in an online geometric estimator. ForceTwin uses instrumented human interaction to build physics-informed digital twins [16]. These works make the overlap concrete: interaction can supply information that appearance leaves unresolved. In JACK, we focus on enabling a robot to choose its own probes across an object’s joints and maintain a mixture of geometric explanations as evidence accumulates.

Knowing what the observations leave out. The study of partially observable safety constraints [2] underscores that a useful model must preserve information relevant to the decision. An explicit joint belief does not remove partial observability or establish safety on its own. The broader direction I want to pursue is to keep refining such beliefs from evidence encountered during routine work, including observations that were not the purpose of the action.

Conclusion

As a field, robotics has achieved impressive feats, both through learning at scale and long before it became popular. I think we need to start adjusting how we interpret scenes. I believe that coupling actions, beliefs about the world, and observations is one of the keys to helping our robots perform better in real life. In JACK, we explore an instance of this belief-based world model, and I am excited to explore future work that follows the same notion.

References

  1. Yishu Li, Wen Hui Leng, Yiming Fang, Ben Eisner, and David Held. FlowBotHD: History-Aware Diffuser Handling Ambiguities in Articulated Objects Manipulation. Conference on Robot Learning (CoRL), 2024.
  2. Matthew Kim, Kensuke Nakamura, and Andrea Bajcsy. How Well Do Latent World Models Understand Partially Observable Safety Constraints? arXiv preprint, 2025 (revised 2026).
  3. Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S. Sukhatme. Interactive Perception: Leveraging Action in Perception and Perception in Action. IEEE Transactions on Robotics, 2017.
  4. Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Patrick Yin, Dieter Fox, and Abhishek Gupta. ASID: Active Exploration for System Identification in Robotic Manipulation. International Conference on Learning Representations (ICLR), 2024.
  5. Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to Explore via Self-Supervised World Models. International Conference on Machine Learning (ICML), 2020.
  6. Russell Buchanan, Adrian Röfer, João Moura, Abhinav Valada, and Sethu Vijayakumar. Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing. IEEE International Conference on Robotics and Automation (ICRA), 2024.
  7. Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. DayDreamer: World Models for Physical Robot Learning. Conference on Robot Learning (CoRL 2022), proceedings published 2023.
  8. Botao He, Yiming Dou, Yiannis Aloimonos, Jitendra Malik, Akash Sharma, and Ruoshi Liu. PlayPolicy: Dexterity from Supervised Play. arXiv preprint, 2026.
  9. David Ha and Jürgen Schmidhuber. World Models. arXiv preprint, 2018.
  10. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models. arXiv preprint, 2023 (revised 2024).
  11. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids. International Conference on Learning Representations (ICLR), 2019.
  12. Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A Probabilistic Framework for Learning Kinematic Models of Articulated Objects. Journal of Artificial Intelligence Research, 2011.
  13. Karol Hausman, Scott Niekum, Sarah Osentoski, and Gaurav S. Sukhatme. Active Articulation Model Estimation through Interactive Perception. IEEE International Conference on Robotics and Automation (ICRA), 2015.
  14. Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building Digital Twins of Articulated Objects From Interaction. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  15. Kei Ota, Hsiao-Yu Tung, Kevin A. Smith, Anoop Cherian, Tim K. Marks, Alan Sullivan, Asako Kanezaki, and Joshua B. Tenenbaum. H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions. IEEE International Conference on Robotics and Automation (ICRA), 2023.
  16. Tim Engelbracht, René Zurbrügg, Mayank Mittal, Marco Hutter, Marc Pollefeys, Hermann Blum, and Zuria Bauer. ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction. arXiv preprint, 2026.