CoChat

MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation

Open access
Lancaster, Patrick, Hansen, Nicklas, Rajeswaran, Aravind, Kumar, Vikash · 2024 · 2024 IEEE International Conference on Robotics and Automation (ICRA)

The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.

Abstract

Robotic systems that aspire to operate in uninstrumented real-world environments must perceive the world directly via onboard sensing. Vision-based learning systems aim to eliminate the need for environment instrumentation by building an implicit understanding of the world based on raw pixels, but navigating the contact-rich high-dimensional search space from solely sparse visual reward signals significantly exacerbates the challenge of exploration. The applicability of such systems is thus typically restricted to simulated or heavily engineered environments since agent exploration in the real-world without the guidance of explicit state estimation and dense rewards can lead to unsafe behavior and safety faults that are catastrophic. In this study, we isolate the root causes behind these limitations to develop a system, called MoDem-V2, capable of learning contact-rich manipulation directly in the uninstrumented real world. Building on the latest algorithmic advancements in model-based reinforcement learning (MBRL), demo-bootstrapping, and effective exploration, MoDem-V2 can acquire contact-rich dexterous manipulation skills directly in the real world. We identify key ingredients for leveraging demonstrations in model learning while respecting real-world safety considerations – exploration centering, agency handover, and actor-critic ensembles. We empirically demonstrate the contribution of these ingredients in four complex visuo-motor manipulation problems in both simulation…

Overview

CoChat Summary

TL;DR

MoDem-V2 adapts visual model-based RL with demonstrations for real-world contact-rich manipulation by adding conservative exploration via BC-centered actions, gradual agency transfer to MPC, and actor-critic ensembles for uncertainty-aware planning. It learns four visuo-motor tasks directly on a physical robot while reducing safety violations relative to MoDem.

Key findings

  • In simulation across four tasks, MoDem-V2 and MoDem both significantly outperform DAPG (State) in sample-efficiency despite DAPG having privileged state and dense rewards; FERM matches on easier pushing tasks but cannot learn bin picking or in-hand reorientation.
  • MoDem-V2 commits significantly fewer safety violations than MoDem; MoDem’s violations sharply increase at the onset of online interaction, and on real inclined pushing MoDem triggered a safety fault within the first two exploration episodes.
  • Ablations show Centering is a necessary sub-component of Schedule; Schedule is generally safer than Ensemble, while Ensemble typically has better sample-efficiency; combined MoDem-V2 achieves Ensemble-like sample-efficiency and Schedule-like safety. Torque penalization alone is ineffective at preventing early safety violations.
  • With only ten demos, MoDem-V2 enabled a real robot to exceed its initial BC policy with about two hours or less of online training data, learning pushing, picking, and in-hand manipulation.

Methods

  • Built on MoDem/TD-MPC visual MBRL: learns a latent world model and policy, then uses short-horizon MPC. Three modifications: sample actions centered around the BC policy, agency transfer from BC actions to MPC via α initialized at 0 and linearly increased to 1.0, and actor-critic ensembles for uncertainty-aware planning with weights \(w_1>0\), \(w_2<0\).
  • Hardware/tasks: Franka Panda arm; Robotiq two-fingered gripper for planar/inclined pushing and bin picking; 10-DOF D’Manus hand for in-hand reorientation; three RealSense D435 cameras. Four tasks: planar pushing, inclined pushing, bin picking, in-hand reorientation. Robot controlled at 12.5 Hz (80 ms per step).
  • Evaluation: simulation comparisons against MoDem, DAPG, and FERM; ablations of the three design choices; real-world deployment. All experiments train an initial policy with only ten demos, and each evaluation is aggregated over 30 trials. Safety violations are hardware faults requiring human intervention in the real world; in simulation they are torque-limit violations or excessive contact force (100 N).

Limitations

  • Stated: the method requires a small number of demonstrations, which may not always be available or easy to collect; it assumes the environment can be reset to a narrow set of starting states; future work is needed to reuse the learned world model across changes in manipulated object and task goal.
  • Evident: real-world evaluation of MoDem was not feasible due to unsafe behavior, so the real-world results do not include a direct baseline comparison.
  • Evident: real-world reward and success detection still relies on task-specific vision modules (e.g., color thresholding, depth camera, image subtraction) and reset mechanisms, which may limit generality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icra57147.2024.10611121

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 0 Not valid · 4 Not checked · 38 Own results and interpretations

19 more claims

Create a free account to see all 44 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 19
  • Probably not a substantial claim. 5

Create a free account to see these claims.

Citations

Cites

Show all 57 references

Research works printed but not linked: 1

  1. [28] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, 'Domain randomization for transferring deep neural networks from simulation to the real world,' in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2017, pp. 23-30. - · DOI · Paper page

Cited by

See all 10

Metadata

Authors
Lancaster, P., Hansen, N., Rajeswaran, A., & Kumar, V.
Affiliations
  • Meta (United States) (US)
  • University of California San Diego (US)
Venue
2024 IEEE International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2024-05-13
Pages
7530-7537
ISSN
1050-4729 (ISSN-L)
OpenAlex
W4401416094
Cited by
8
References
53
Citations by year
2026: 5 · 2025: 1 · 2024: 2
FWCI
2.31
Type
conference paper
Language
en
Open access
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
Keywords
model-based reinforcement learning, robotic manipulation, dexterous manipulation, contact-rich manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation. https://app.cochat.ai/papers/10.1109/icra57147.2024.10611121
The paper
Lancaster, P., Hansen, N., Rajeswaran, A., & Kumar, V. (2024). MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation. 2024 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra57147.2024.10611121

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.