MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
Open accessThe DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Abstract
Robotic systems that aspire to operate in uninstrumented real-world environments must perceive the world directly via onboard sensing. Vision-based learning systems aim to eliminate the need for environment instrumentation by building an implicit understanding of the world based on raw pixels, but navigating the contact-rich high-dimensional search space from solely sparse visual reward signals significantly exacerbates the challenge of exploration. The applicability of such systems is thus typically restricted to simulated or heavily engineered environments since agent exploration in the real-world without the guidance of explicit state estimation and dense rewards can lead to unsafe behavior and safety faults that are catastrophic. In this study, we isolate the root causes behind these limitations to develop a system, called MoDem-V2, capable of learning contact-rich manipulation directly in the uninstrumented real world. Building on the latest algorithmic advancements in model-based reinforcement learning (MBRL), demo-bootstrapping, and effective exploration, MoDem-V2 can acquire contact-rich dexterous manipulation skills directly in the real world. We identify key ingredients for leveraging demonstrations in model learning while respecting real-world safety considerations – exploration centering, agency handover, and actor-critic ensembles. We empirically demonstrate the contribution of these ingredients in four complex visuo-motor manipulation problems in both simulation…
Overview
CoChat Summary
TL;DR
MoDem-V2 adapts visual model-based RL with demonstrations for real-world contact-rich manipulation by adding conservative exploration via BC-centered actions, gradual agency transfer to MPC, and actor-critic ensembles for uncertainty-aware planning. It learns four visuo-motor tasks directly on a physical robot while reducing safety violations relative to MoDem.
Key findings
- In simulation across four tasks, MoDem-V2 and MoDem both significantly outperform DAPG (State) in sample-efficiency despite DAPG having privileged state and dense rewards; FERM matches on easier pushing tasks but cannot learn bin picking or in-hand reorientation.
- MoDem-V2 commits significantly fewer safety violations than MoDem; MoDem’s violations sharply increase at the onset of online interaction, and on real inclined pushing MoDem triggered a safety fault within the first two exploration episodes.
- Ablations show Centering is a necessary sub-component of Schedule; Schedule is generally safer than Ensemble, while Ensemble typically has better sample-efficiency; combined MoDem-V2 achieves Ensemble-like sample-efficiency and Schedule-like safety. Torque penalization alone is ineffective at preventing early safety violations.
- With only ten demos, MoDem-V2 enabled a real robot to exceed its initial BC policy with about two hours or less of online training data, learning pushing, picking, and in-hand manipulation.
Methods
- Built on MoDem/TD-MPC visual MBRL: learns a latent world model and policy, then uses short-horizon MPC. Three modifications: sample actions centered around the BC policy, agency transfer from BC actions to MPC via α initialized at 0 and linearly increased to 1.0, and actor-critic ensembles for uncertainty-aware planning with weights \(w_1>0\), \(w_2<0\).
- Hardware/tasks: Franka Panda arm; Robotiq two-fingered gripper for planar/inclined pushing and bin picking; 10-DOF D’Manus hand for in-hand reorientation; three RealSense D435 cameras. Four tasks: planar pushing, inclined pushing, bin picking, in-hand reorientation. Robot controlled at 12.5 Hz (80 ms per step).
- Evaluation: simulation comparisons against MoDem, DAPG, and FERM; ablations of the three design choices; real-world deployment. All experiments train an initial policy with only ten demos, and each evaluation is aggregated over 30 trials. Safety violations are hardware faults requiring human intervention in the real world; in simulation they are torque-limit violations or excessive contact force (100 N).
Limitations
- Stated: the method requires a small number of demonstrations, which may not always be available or easy to collect; it assumes the environment can be reset to a narrow set of starting states; future work is needed to reuse the learned world model across changes in manipulated object and task goal.
- Evident: real-world evaluation of MoDem was not feasible due to unsafe behavior, so the real-world results do not include a direct baseline comparison.
- Evident: real-world reward and success detection still relies on task-specific vision modules (e.g., color thresholding, depth camera, image subtraction) and reset mechanisms, which may limit generality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 4 Not checked · 38 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Third, we use actor-critic ensembles to estimate the epistemic uncertainty of the value estimations of these short-horizon trajectories, allowing the agent to avoid overly optimistic actions.”
What the sources say
- Partly supportedcites [12] [12] Z. Huang, S. Zhou, B. Zhuang, and X. Zhou, 'Learning to run with actor-critic ensemble,' arXiv preprint arXiv:1712.08987 , 2017. - sourceExperimental — accuracy not yet measured
The Method (p. 1) and Inference-Time Actor Critic Ensemble (p. 2) describe critics scoring actors’ proposed actions to avoid actions with fatal consequences. They do not describe estimating epistemic uncertainty, evaluating short-horizon trajectories, or avoiding actions because their value estimates are overly optimistic. (LLM, conf=0.87)
- Not matchedcites [13] [13] S. Fujimoto, H. Hoof, and D. Meger, 'Addressing function approximation error in actor-critic methods,' in International conference on machine learning . PMLR, 2018, pp. 1587-1596. -
Its entry has no title and year that we can search by.
- Partly supportedcites [14] [14] K. Chua, R. Calandra, R. McAllister, and S. Levine, 'Deep reinforcement learning in a handful of trials using probabilistic dynamics models,' Advances in neural information processing systems , vol. 31, 2018. - sourceExperimental — accuracy not yet measured
Section 4 (p. 4) says bootstrapped ensembles provide estimates of epistemic uncertainty, and Appendix A.7 (p. 15) says the planning method can avoid relying on unpredictable outcomes. But the ensembles model dynamics, not actor-critic values of short-horizon trajectories; the paper does not establish the claimed mechanism for avoiding overly optimistic actions. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We indeed found MoDem to be infeasible for direct application in the real world due to excessively aggressive exploratory behavior.”
- Also stated in: § C. Real World Results · p. 5 · hedged
“As suggested by our simulation experiments, the original MoDem is unsafe for learning real-world manipulation tasks.”
- Also stated in: § C. Real World Results · p. 5 · plain
“Due to the unsafe behavior MoDem induces and the significant human intervention that would be required, it is not feasible to evaluate MoDem in the real world.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § C. Real World Results · p. 5 · hedged
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that MoDem-v2's conservative exploration significantly enhances its safety profile compared to other baselines, while still retaining the sampleefficient learning capability of MoDem.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that MoDem exerts excessive forces and torques far beyond those exemplified in the provided demonstrations (see Section VI).”
- Also stated in: § A. Simulated Comparison to Baselines · p. 5 · plain
“MoDem's safety violations sharply increase as the interaction phase begins.”
- Also stated in: § C. Real World Results · p. 5 · plain
“the robot triggered a safety fault within the first two exploration episodes because it exerted excessive force/torque on the incline.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § A. Simulated Comparison to Baselines · p. 5 · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that MoDem-v2's conservative exploration significantly enhances its safety profile compared to other baselines, while still retaining the sampleefficient learning capability of MoDem.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 44 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 5
Citations
Cites
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 1×
“…the following five components: where h θ , d θ , R θ and Q θ are learned end-to-end using a combination of joint-embedding predictive learning [11], reward prediction, and Temporal Difference (TD) learning, and π θ is a deterministic policy that learns to maximize Q θ conditioned on a latent…”
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · cited 1×
“While this provides a solution for efficiently finding the maximum value of Q over actions, it is subject to significant overestimation bias [13].”
BackgroundDomain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1×
“One approach to developing robot manipulation policies that avoid such safety restrictions is simulation to reality transfer [1], [2], [3], [4].”
BackgroundLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · cited 1×
“…learning on real hardware, most of which fall into one of three categories: learning from human demonstrations [36], [37], [38], [39], learning from large uncurated datasets [40], [41], [20], learning from online interaction via RL [42], [43], or any combination of them [17], [44], [45], [46].”
BackgroundMastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214 · cited 2×
“For continuous control, planning can be formalized as Model Predictive Control (MPC) [20], [6], [7], [9], whereas Monte-Carlo Tree Search (MCTS) is used for discrete action spaces [22], [23].”
BackgroundMethodSafe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., & Schoellig, A. P. (2021) · Annu. Rev. Control. Robotics Auton. Syst. · doi:10.48550/arxiv.2108.06266 · Cited by 1,143
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047 · cited 1×
“…learning on real hardware, most of which fall into one of three categories: learning from human demonstrations [36], [37], [38], [39], learning from large uncurated datasets [40], [41], [20], learning from online interaction via RL [42], [43], or any combination of them [17], [44], [45], [46].”
Background(2022) · doi:10.15607/rss.2023.xix.025 · Cited by 916
Research works printed but not linked: 1
- [28] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, 'Domain randomization for transferring deep neural networks from simulation to the real world,' in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2017, pp. 23-30. - · DOI · Paper page
Cited by
TD-MPC2: Scalable, Robust World Models for Continuous Control
Hansen, N., Su, H., & Wang, X. (2023) · arXiv · doi:10.48550/arxiv.2310.16828 · Cited by 8
“…has demonstrated strong empirical performance in prior work (Hansen et al., 2022; 2023; Yuan et al., 2022; Yang et al., 2023; Feng et al., 2023; Chitnis et al., 2023; Zhu et al., 2023; Lancaster et al., 2023), most successes have been limited to single-task learning with little emphasis on scaling.”
BackgroundMethodSelf-citationV-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Mojtaba, Komeili, Muckley, M., Rizvi, A., Roberts, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.09985 · Cited by 8
“…Agrawal et al., 2016; Ha and Schmidhuber, 2018; Hafner et al., 2019b; Nair et al., 2022; Wu et al., 2023b; Tomar et al., 2024; Hu et al., 2024; Lancaster et al., 2024), or utilizing more structured representation spaces such as keypoint representations (Manuelli et al., 2020; Das et al.,…”
DreamerNav: learning-based autonomous navigation in dynamic indoor environments using world models
Shanks, S., Embley-Riches, J., Liu, J. F., Delfaki, A. M., Ciliberto, C., & Kanoulas, D. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1655171 · Cited by 22
Leveraging Single and Multi-task Reinforcement Learning Algorithms for Autonomous Mobile Aloha Robot
Narendra, A., Makarov, D. A., & Panov, A. I. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-77688-5_42 · Cited by 2
Olivares, D., Fournier, P., Vasishta, P., & Marzat, J. (2026) · SN Computer Science · doi:10.1007/s42979-026-04751-w · Cited by 2
“Another path to smooth actions for TD-MPC (for which CAPS is not applicable) is MoDeM-v2 [60] which aims at learning safe policies by biasing the initial data distribution towards a desired behavior, in our case an action smooth controller.”
Metadata
- Authors
- Lancaster, P., Hansen, N., Rajeswaran, A., & Kumar, V.
- Affiliations
- Meta (United States) (US)
- University of California San Diego (US)
- Venue
- 2024 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2024-05-13
- Pages
- 7530-7537
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W4401416094
- Cited by
- 8
- References
- 53
- Citations by year
- 2026: 5 · 2025: 1 · 2024: 2
- FWCI
- 2.31
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
- Keywords
- model-based reinforcement learning, robotic manipulation, dexterous manipulation, contact-rich manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation. https://app.cochat.ai/papers/10.1109/icra57147.2024.10611121
- The paper
- Lancaster, P., Hansen, N., Rajeswaran, A., & Kumar, V. (2024). MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation. 2024 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra57147.2024.10611121
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.