CoChat
Sign Up
PreprintCited by 12Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

Open access
Mandlekar, Ajay, Nasiriany, Soroush, Wen, Bowen, Akinola, Iretiayo, Narang, Yashraj, Fan, Linxi, Zhu, Yuke, Fox, Dieter·2023·arXiv
DOI 10.48550/arxiv.2310.17596arXiv 2310.17596
Open this paper in CoChat — free accountOpen PDF

Abstract

Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We introduce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new contexts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks, such as multi-part assembly and coffee preparation, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably to collecting additional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at https://mimicgen.github.io .

AI snapshot

Written from the paper’s full text.

TL;DR

MimicGen is a data-generation system that reuses a small set of human demonstrations by parsing them into object-centric subtasks, transforming those segments to new object poses, and executing them to collect new robot demonstrations. The authors use it to generate over 50K demonstrations across 18 tasks from about 200 human demonstrations, and show that imitation-learning agents trained on the generated data perform strongly across varied scenes, objects, and robot arms.

Key findings

  • MimicGen generated 50K+ new demonstrations for 18 tasks across 2 simulators and a physical robot arm using only ∼200 source human demos; most simulated tasks used 10 source demos and 1000 generated demos per task variant.
  • Generated data substantially improved agents over source data: Square rose from 11.3% to 90.7%, Threading from 19.3% to 98.0%, and Three Piece Assembly from 1.3% to 82.0% on the default distribution.
  • Agents trained on 200 MimicGen demos performed comparably to agents trained on 200 human demos in most cases, despite MimicGen using only 10 source human demos; performance also improved from 200 to 1000 demos but showed diminishing returns from 1000 to 5000.
  • MimicGen transferred across robot arms and objects: on Square D0, policy success was 80%–91% across Panda, Sawyer, IIWA, and UR5e; on Mug Cleanup, image-based agents achieved 90.7% on an unseen mug and 75.3% on a set of 12 mugs.

Methods

  • MimicGen assumes a delta end-effector pose action space, a known sequence of object-centric subtasks, and observable object poses at the start of each subtask during data generation. It parses source human demos into subtask segments, selects and spatially transforms segments for new object poses, stitches them with interpolation, executes them, and keeps only successful task attempts.
  • Data were collected across basic, contact-rich, long-horizon, mobile manipulation, and Factory tasks; source datasets were mostly 10 human demos per task, with 25 for Mobile Kitchen and 10 robomimic Square PH demos for Square. Generated datasets were 1000 demonstrations per task variant.
  • Policies were trained with Behavioral Cloning using an RNN policy (BC-RNN) on image and low-dimensional observations, reporting maximum success rate across evaluations over 3 seeds.

Limitations

  • Stated: MimicGen assumes a known sequence of object-centric subtasks, requires object pose estimates at the start of each subtask during data generation, and assumes one reference object per subtask. It filters generated data only by task success, so datasets can be biased; it uses linear interpolation that does not guarantee collision-free motion; and it was demonstrated on quasi-static tasks with rigid objects, with novel objects assumed to come from the same category.
  • Stated: Mobile manipulation support assumes the robot does not move base and arm simultaneously and simply copies mobile base actions from the reference segment; multi-arm tasks are not supported.
  • Evident from the text: real-world policy performance was much lower than simulation (Stack 36% vs. 100%; Coffee 14% vs. ∼90%), which the authors partly attribute to larger interpolation segments used for hardware safety and smaller real-world datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.17596

Cites

120 of 120 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

103 of 113 cited sources checked · 64 supported · 2 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 1

    A relatively-large dataset of 200 demonstrations was required to achieve a modest success rate of 73.3% on the robomimic coke can task.

    “Consider a case study from robomimic [7], in which the agent is tasked with moving a coke can from one bin into another. This is a simple task involving a single scene, single object, and single robot; however, a relatively-large dataset of 200 demonstrations was required to achieve a modest success rate of 73.3%.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın, 'What matters in learning from offline human demonstrations for robot manipulation,' in Conference on Robot Learning (CoRL) , 2021. source

      Section 3.1 identifies the Can task, and Appendix B.1 describes a 200-demonstration Can-Paired dataset. But Appendix B.1 says BC-RNN can achieve near-perfect performance on just 100 good-quality Can demonstrations, contrary to the claim that 200 were required for a 73.3% success rate. Section 4.2 associates 73.3% with a Square dataset, not the Can task; page numbers are unavailable in our copy. (LLM, conf=0.87)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Interpretationown data
    § 6.3 MimicGen Analysis · p. 7

    These results showcase the value of using replay-based mechanisms for data collection instead of directly using them to deploy as policy.

    “These results showcase the value of using replay-based mechanisms for data collection instead of directly using them to deploy as policy as in prior works [8, 11].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [8] B. Wen, W. Lian, K. Bekris, and S. Schaal, 'You only demonstrate once: Category-level manipulation from single visual demonstration,' in Robotics: Science and Systems (RSS) , 2022. source
    • cites [11] N. Di Palo and E. Johns, 'Learning multi-stage tasks with one demonstration via self-replay,' in Conference on Robot Learning . PMLR, 2022, pp. 1180-1189. source
  • Interpretationcitation
    § C FAQ · p. 17

    The tasks MimicGen was demonstrated on have pose variation well beyond typical prior works using BC from human demos.

    “The tasks also have pose variation well beyond typical prior works using BC from human demos [1,3-7,30,33,55,56].”
    Quote in the paperNot a substantial claim9 of 10 sources checked
    • cites [1] T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, 'Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,' arXiv preprint arXiv:1710.04615 , 2017. source

      The Appendix (Task Specification; page unavailable) describes varying object positions and orientations across several tasks, including a hammer placed within a 15 × 15 cm area at varying angles. The paper does not establish that this pose variation is well beyond that of typical prior work using behavioral cloning from human demonstrations. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [3] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, 'Bcz: Zero-shot task generalization with robotic imitation learning,' in Conference on Robot Learning . PMLR, 2022, pp. 991-1002. source

      Section 4 (p. 3) describes BC-Z as a demonstration-based manipulation system with objects in randomized poses, so it is a relevant prior-work example. Appendix B.2 (p. 14) says objects were only occasionally shuffled between demonstrations. The paper does not compare its pose variation with MimicGen’s or establish that MimicGen’s is well beyond it. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [4] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog et al. , 'Do as i can, not as i say: Grounding language in robotic affordances,' arXiv preprint arXiv:2204.01691 , 2022. source

      C.2 (p. 21) describes BC training using 68,000 human-teleoperated demonstrations, so this paper is an example of prior work using BC from human demonstrations. C.3 (p. 21) says object positions were randomized during evaluation, but the copy does not quantify pose variation or compare it with MimicGen's tasks. It therefore does not establish the claimed ‘well beyond’ comparison. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , 'Rt-1: Robotics transformer for real-world control at scale,' arXiv preprint arXiv:2212.06817 , 2022. source

      The Data collection at scale section describes randomized scenes and human demonstrations, while Robustness compares RT-1’s task and dataset diversity with prior work. Neither passage assesses pose variation in MimicGen’s tasks or compares it with typical behavioral-cloning work using human demonstrations. Searches for pose variation and object-placement variation found no such comparison in our copy; page numbers were not provided. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [6] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, 'Bridge data: Boosting generalization of robotic skills with cross-domain datasets,' arXiv preprint arXiv:2109.13396 , 2021. source

      The Related Work section (p. 2) compares prior datasets by numbers of tasks and domains, not by pose variation. The dataset description (p. 4) describes position randomization in Bridge Data, but our copy does not compare pose variation in prior behavioral-cloning work with the tasks demonstrated by MimicGen (searched: pose variation, object positions, behavioral cloning). (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın, 'What matters in learning from offline human demonstrations for robot manipulation,' in Conference on Robot Learning (CoRL) , 2021. source
    • cites [30] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, 'One-shot visual imitation learning via meta-learning,' in Conference on robot learning . PMLR, 2017, pp. 357-368. source
    • cites [33] A. Mandlekar, D. Xu, R. Mart´ ın-Mart´ ın, S. Savarese, and L. Fei-Fei, 'Learning to generalize across long-horizon tasks from human demonstrations,' arXiv preprint arXiv:2003.06085 , 2020. source

      The Real World Robot Manipulation Experiments section (p. 7) describes two starting configurations for the PandaKitchen task, but does not quantify pose variation or compare it with prior behavioral-cloning work. Searches for “pose variation” and varying object positions found no passage establishing that comparison. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
    • cites [55] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu, 'Learning and retrieval from prior data for skill-based imitation learning,' in Conference on Robot Learning (CoRL) , 2022. source

      Section 5.2 describes behavioral cloning on target-task demonstrations as a baseline, and the Target Tasks section describes randomized object placements in tasks demonstrated by humans. This makes the paper relevant as an example of prior work involving BC and pose variation. It does not compare that variation with MimicGen’s tasks or establish that MimicGen’s variation is ‘well beyond’ it. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [56] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, 'Calvin: A benchmark for languageconditioned policy learning for long-horizon robot manipulation tasks,' IEEE Robotics and Automation Letters , vol. 7, no. 3, pp. 7327-7334, 2022. source
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.
  • Interpretationcitation
    § C FAQ · p. 17

    MimicGen datasets allow full compatibility with a wide spectrum of offline policy learning algorithms.

    “while MimicGen datasets allow full compatibility with a wide spectrum of offline policy learning algorithms [57].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [57] S. Levine, A. Kumar, G. Tucker, and J. Fu, 'Offline reinforcement learning: Tutorial, review, and perspectives on open problems,' arXiv preprint arXiv:2005.01643 , 2020. source

      Section 2.2 (p. 8) discusses using datasets with various offline RL algorithms, but cautions that not all methods are effective. A search for MimicGen found no passage about its datasets or their compatibility with offline policy-learning algorithms, so the source does not establish the specific claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Premisecitation
    § C FAQ · p. 17

    It can be difficult to generate plausible interactions without prior knowledge of physics or causal dependencies, especially for new scenes, objects, or robots.

    “however, it can be difficult to generate plausible interactions without prior knowledge of physics [35] or causal dependencies [41,42], especially for new scenes, objects, or robots.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [35] P. Mitrano and D. Berenson, 'Data augmentation for manipulation,' arXiv preprint arXiv:2205.02886 , 2022. source

      The Assumptions section (page unavailable) says valid augmentations generally rely on expert knowledge or heuristics, and specifies physical assumptions required by this method. This supports the difficulty of generating plausible interactions without prior physical knowledge, but the paper does not specifically establish that the difficulty is greater for new scenes, objects, or robots. The causal-dependencies clause has separate citations in the citing sentence. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [41] S. Pitis, E. Creager, and A. Garg, 'Counterfactual data augmentation using locally factored dynamics,' Advances in Neural Information Processing Systems , vol. 33, pp. 3976-3990, 2020. source

      Section 3 (p. 6) says generating counterfactual data generally requires either a domain expert’s approximate causal model or local causal structure learned from data. This supports the importance of causal dependencies, but the paper does not establish that generating plausible interactions is especially difficult for new scenes, objects, or robots; it also presents learning those dependencies as an alternative to prior knowledge. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [42] S. Pitis, E. Creager, A. Mandlekar, and A. Garg, 'Mocoda: Model-based counterfactual data augmentation,' arXiv preprint arXiv:2210.11287 , 2022. source

      Section 4 (p. 6) says generating augmented transitions requires known causal parent sets, and Section 2.2 (p. 4) calls discovering dynamics factorization an open problem. This supports the difficulty of generating transitions without causal-dependency information, but the paper does not specifically establish that plausible interactions are especially difficult for new scenes, objects, or robots. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 25
  • May combine two claims. 40
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § J.2 Offline Data Augmentation · p. 27

    Several works have used offline data augmentation to increase the dataset size for learning policies.

    “Several works have used offline data augmentation to increase the dataset size for learning policies [7, 35-45].”
    Quote in the paperNot a substantial claim9 of 10 sources checked
    • cites [7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın, 'What matters in learning from offline human demonstrations for robot manipulation,' in Conference on Robot Learning (CoRL) , 2021. source

      Section E.3 describes random crops applied to image observations when they are fed to the network, and Section 4.3 reports that this randomization improves learned policy performance. The paper therefore supports using augmentation in offline policy learning, but it does not say the augmentation was performed offline to increase the dataset size; Section 4.6 studies dataset size by sampling subsets of collected demonstrations. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [35] P. Mitrano and D. Berenson, 'Data augmentation for manipulation,' arXiv preprint arXiv:2205.02886 , 2022. source

      The Cluttered Planar Pushing section reports increasing the training data from 3,000 to 75,000 time steps through augmentation. But the paper describes training a dynamics model and a constraint checker, not learning policies; that part of the cited claim is unsupported. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [36] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, 'Reinforcement learning with augmented data,' arXiv preprint arXiv:2004.14990 , 2020. source

      Section 2.2 (p. 2) identifies several works using data augmentation in reinforcement learning, and the Introduction (p. 1) says RAD trains agents on multiple augmented views of an input. Neither passage establishes that those works augmented data offline to increase the size of a policy-learning dataset; the dataset-size example in Appendix A.1 (p. 13) concerns supervised image classification. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [37] I. Kostrikov, D. Yarats, and R. Fergus, 'Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,' arXiv preprint arXiv:2004.13649 , 2020. source

      Section 3.1 (p. 3) says this work augments images when they are sampled from the replay buffer during SAC training, repeating the transformation on every sampling. That contradicts the characterization of this work as using offline augmentation to increase a dataset’s size, although it does use augmentation for policy learning. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [38] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, 'Visual imitation made easy,' arXiv e-prints , pp. arXiv-2008, 2020.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [39] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin, 'A framework for efficient robotic manipulation,' arXiv preprint arXiv:2012.07975 , 2020. source

      The Introduction and Data Augmentation section support that this paper uses data augmentation to learn policies. But the Introduction calls the policy-training augmentations online, and the Minimal Collection of Demonstrations section says augmentation is applied during training; neither describes offline augmentation to increase a dataset's size. No page numbers are available in our copy. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [40] S. Sinha, A. Mandlekar, and A. Garg, 'S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics,' in Conference on Robot Learning . PMLR, 2022, pp. 907-917. source
    • cites [41] S. Pitis, E. Creager, and A. Garg, 'Counterfactual data augmentation using locally factored dynamics,' Advances in Neural Information Processing Systems , vol. 33, pp. 3976-3990, 2020. source
    • cites [42] S. Pitis, E. Creager, A. Mandlekar, and A. Garg, 'Mocoda: Model-based counterfactual data augmentation,' arXiv preprint arXiv:2210.11287 , 2022. source
    • cites [43] Z. Mandi, H. Bharadhwaj, V. Moens, S. Song, A. Rajeswaran, and V. Kumar, 'Cacti: A framework for scalable multi-task multi-scene visual imitation learning,' arXiv preprint arXiv:2212.05711 , 2022. source

  • May combine two claims.

    Findingown data
    § 6.2 Comparing MimicGen to using more human data · p. 6

    On Kitchen, BC-RNN achieves 54.7% on 10 source demos, 100.0% on generated D0 data, and 76.0% on D1 variant, while BUDS achieved 72.0% on D0.

    “On Kitchen, BUDS achieved 72.0% ( D 0 ), while BC-RNN achieves 54.7% on our 10 source demos, 100.0% on our generated D 0 data, and 76.0% on the D 1 variant, where all objects move in wider regions.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [53] Y. Zhu, P. Stone, and Y. Zhu, 'Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation,' IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 4126-4133, 2022. source

  • May combine two claims.

    Findingown data
    § 6.2 Comparing MimicGen to using more human data · p. 6

    On Hammer Cleanup, BC-RNN achieves 59.3% on 10 source demos, 100.0% on generated 1000 D0 demos, and 62.7% on D1 variant, while BUDS achieved 68.6% on D0.

    “On Hammer Cleanup, BUDS achieved 68.6% ( D 0 ), while BC-RNN achieves 59.3% on our 10 source demos, 100.0% on our generated 1000 D 0 demos, and 62.7% on the D 1 variant where both the hammer and drawer move substantially.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [53] Y. Zhu, P. Stone, and Y. Zhu, 'Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation,' IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 4126-4133, 2022. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    Several recent works propose imitation learning methods that replay previous human demonstrations.

    “In fact, several recent works build on this intuition and propose imitation learning methods that replay previous human demonstrations [8-11].”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [8] B. Wen, W. Lian, K. Bekris, and S. Schaal, 'You only demonstrate once: Category-level manipulation from single visual demonstration,' in Robotics: Science and Systems (RSS) , 2022. source
    • cites [9] E. Johns, 'Coarse-to-fine imitation learning: Robot manipulation from a single demonstration,' in 2021 IEEE international conference on robotics and automation (ICRA) . IEEE, 2021, pp. 4613-4619. source
    • cites [10] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns, 'Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning,' in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2022, pp. 86148621. source
    • cites [11] N. Di Palo and E. Johns, 'Learning multi-stage tasks with one demonstration via self-replay,' in Conference on Robot Learning . PMLR, 2022, pp. 1180-1189. source

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    The RT-1 data collection effort spanned nearly 1.5 years and produced policies that can rearrange, cleanup, and retrieve objects with a 97% success rate across a handful of kitchens.

    “The nearly 1.5-year data collection effort from RT-1 [5] spans several human operators, months, kitchens, and robot arms to produce policies that can rearrange, cleanup, and retrieve objects with a 97% success rate across a handful of kitchens.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , 'Rt-1: Robotics transformer for real-world control at scale,' arXiv preprint arXiv:2212.06817 , 2022. source

      The Data section reports demonstrations collected over 17 months, and the Robustness section describes kitchen tasks involving restocking, tidying, and fetching objects. But its 97% success rate applies to more than 200 seen instructions, not to performance across kitchens; Table III reports 70% overall success in realistic kitchen generalization scenarios. (LLM, conf=0.93)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    A dataset of over 20,000 trajectories enables generalization to tasks with modest changes in objects and goals.

    “For example, [3] showed that a dataset of over 20,000 trajectories enables generalization to tasks with modest changes in objects and goals.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [3] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, 'Bcz: Zero-shot task generalization with robotic imitation learning,' in Conference on Robot Learning . PMLR, 2022, pp. 991-1002. source

  • May combine two claims.

    Premisecitation
    § D Limitations · p. 19

    A large number of robot learning works and benchmarks use quasi-static tasks, making the system broadly applicable.

    “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3-7,14,18,19,22,30,33,51, 55, 56, 64-66], making the system broadly applicable.”
    Quote in the paperSubstantial claim8 of 10 sources checked
    • cites [1] T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, 'Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,' arXiv preprint arXiv:1710.04615 , 2017. source
    • cites [3] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, 'Bcz: Zero-shot task generalization with robotic imitation learning,' in Conference on Robot Learning . PMLR, 2022, pp. 991-1002. source
    • cites [4] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog et al. , 'Do as i can, not as i say: Grounding language in robotic affordances,' arXiv preprint arXiv:2204.01691 , 2022. source
    • cites [5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , 'Rt-1: Robotics transformer for real-world control at scale,' arXiv preprint arXiv:2212.06817 , 2022. source
    • cites [6] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, 'Bridge data: Boosting generalization of robotic skills with cross-domain datasets,' arXiv preprint arXiv:2109.13396 , 2021. source

      The Related Work section (p. 2) surveys robot-learning datasets, and the dataset description (pp. 3–4) describes this paper’s kitchen tasks. Neither characterizes those tasks or other benchmarks as quasi-static, so our copy does not establish the attributed reason for the system’s broad applicability (searched: quasi-static tasks, static and dynamic manipulation). (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites [7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın, 'What matters in learning from offline human demonstrations for robot manipulation,' in Conference on Robot Learning (CoRL) , 2021. source
    • cites [14] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke et al. , 'Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,' arXiv preprint arXiv:1806.10293 , 2018. source

      Related Work (p. 3) says current grasping systems typically predict a grasp pose and execute an open-loop plan, supporting the point that static approaches are common. But the paper does not establish that a large number of robot-learning benchmarks use quasi-static tasks; it presents QT-Opt as dynamic, closed-loop grasping that responds to disturbances (Discussion, p. 8). (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [18] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, 'Rlbench: The robot learning benchmark & learning environment,' IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3019-3026, 2020. source
    • cites [19] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani et al. , 'Transporter networks: Rearranging the visual world for robotic manipulation,' arXiv preprint arXiv:2010.14406 , 2020. source

      Section 4.1 (page unavailable) describes Ravens as a benchmark of ten discrete-time tabletop manipulation tasks using position-controlled, two-pose motion primitives. This supports citing the paper as an example of a robot-learning benchmark with such tasks, but the copy does not call them quasi-static or establish how many works and benchmarks use quasi-static tasks. The sentence’s claim that this makes the citing authors’ system broadly applicable is their own inference. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
    • cites [22] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox, 'Imitating task and motion planning with visuomotor transformers,' arXiv preprint arXiv:2305.16309 , 2023. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § C FAQ · p. 17

    Replay-based imitation learning methods are promising for learning manipulation tasks using a handful of demonstrations.

    “Replay-based imitation learning methods are promising for learning manipulation tasks using a handful of demonstrations [8-11, 46-48]”
    Quote in the paperNot a substantial claim7 of 7 sources checked
    • cites [8] B. Wen, W. Lian, K. Bekris, and S. Schaal, 'You only demonstrate once: Category-level manipulation from single visual demonstration,' in Robotics: Science and Systems (RSS) , 2022. source
    • cites [9] E. Johns, 'Coarse-to-fine imitation learning: Robot manipulation from a single demonstration,' in 2021 IEEE international conference on robotics and automation (ICRA) . IEEE, 2021, pp. 4613-4619. source
    • cites [10] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns, 'Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning,' in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2022, pp. 86148621. source
    • cites [11] N. Di Palo and E. Johns, 'Learning multi-stage tasks with one demonstration via self-replay,' in Conference on Robot Learning . PMLR, 2022, pp. 1180-1189. source
    • cites [46] V. Vosylius and E. Johns, 'Where to start? transferring simple skills to complex environments,' arXiv preprint arXiv:2212.06111 , 2022. source

      The Experimental Setup (p. 5) describes learning grasping and placing skills through behavioural cloning, supporting the imitation-learning and manipulation portions of the claim. It does not describe the method as replay-based, and Appendix A.1 (p. 12) says each policy used 1,000 expert demonstrations, not a handful. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [47] A. Chenu, O. Serris, O. Sigaud, and N. Perrin-Gilbert, 'Leveraging sequentiality in reinforcement learning from a single demonstration,' arXiv preprint arXiv:2211.04786 , 2022. source
    • cites [48] J. Liang, B. Wen, K. Bekris, and A. Boularias, 'Learning sensorimotor primitives of sequential manipulation tasks from visual demonstrations,' in 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2022, pp. 8591-8597. source

      The Abstract says the framework learns robot manipulation tasks from a few visual demonstrations, and the Evaluation reports data-efficient learning from demonstration trajectories. However, the Learning Approach describes policy training and inference from demonstrations, not a replay-based imitation method; that characterization is unsupported by this paper. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § C FAQ · p. 17

    Offline data augmentation can be effective for generating larger dataset for robot manipulation.

    “Offline data augmentation can be effective for generating larger dataset for robot manipulation [7, 35-45]”
    Quote in the paperNot a substantial claim9 of 10 sources checked
    • cites [7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın, 'What matters in learning from offline human demonstrations for robot manipulation,' in Conference on Robot Learning (CoRL) , 2021. source

      Section 4.3 reports that pixel-shift image randomization improves robot-manipulation policy performance, and Appendix E.3 describes applying random crops when images are fed to the network. But the paper does not show offline augmentation generating a larger dataset; Section 4.6 studies dataset size using subsets of collected demonstrations instead. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [35] P. Mitrano and D. Berenson, 'Data augmentation for manipulation,' arXiv preprint arXiv:2205.02886 , 2022. source
    • cites [36] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, 'Reinforcement learning with augmented data,' arXiv preprint arXiv:2004.14990 , 2020. source

      Section 5.3 (p. 8) reports that data augmentation improves performance on reinforcement-learning benchmarks. It does not show that offline augmentation generates a larger dataset for robot manipulation; the dataset-size example in A.1 (p. 13) concerns supervised image classification instead. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [37] I. Kostrikov, D. Yarats, and R. Fergus, 'Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,' arXiv preprint arXiv:2004.13649 , 2020. source

      Section 3.1 (p. 3) reports that image augmentation reduced overfitting in DeepMind control tasks. But it applies fresh transformations whenever images are sampled from a replay buffer, rather than generating a larger dataset offline, and the paper does not establish the claimed effect for robot manipulation. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [38] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, 'Visual imitation made easy,' arXiv e-prints , pp. arXiv-2008, 2020.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [39] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin, 'A framework for efficient robotic manipulation,' arXiv preprint arXiv:2012.07975 , 2020. source

      The “Minimal Collection of Demonstrations” section says the authors use random crops of stored demonstrations during encoder pre-training, supporting offline augmentation of robot-manipulation data. It does not say they generate a larger dataset; it describes augmentation during subsequent RL training as occurring while the robot interacts with its environment. Our copy provides no page numbers for these passages. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [40] S. Sinha, A. Mandlekar, and A. Garg, 'S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics,' in Conference on Robot Learning . PMLR, 2022, pp. 907-917. source
    • cites [41] S. Pitis, E. Creager, and A. Garg, 'Counterfactual data augmentation using locally factored dynamics,' Advances in Neural Information Processing Systems , vol. 33, pp. 3976-3990, 2020. source

      The Experiments section (p. 8) reports that enlarging a HER agent’s dataset with CoDA approximately doubled sample efficiency in FetchPush and Slide2 manipulation tasks. It separately tests augmentation of a fixed, offline dataset in Pong (p. 8); the manipulation experiments collect new environment data during training, so they do not establish effectiveness in an offline robot-manipulation setting. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [42] S. Pitis, E. Creager, A. Mandlekar, and A. Garg, 'Mocoda: Model-based counterfactual data augmentation,' arXiv preprint arXiv:2210.11287 , 2022. source
    • cites [43] Z. Mandi, H. Bharadhwaj, V. Moens, S. Song, A. Rajeswaran, and V. Kumar, 'Cacti: A framework for scalable multi-task multi-scene visual imitation learning,' arXiv preprint arXiv:2212.05711 , 2022. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § E Full Related Work · p. 21

    RoboTurk is a system for crowdsourcing task demonstrations from human operators using smartphone-based teleoperation and video streams provided in web browsers.

    “RoboTurk [2,23-26] is a system for crowdsourcing task demonstrations from human operators using smartphone-based teleoperation and video streams provided in web browsers.”
    Quote in the paperNot a substantial claim5 of 5 sources checked
    • cites [2] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei, 'RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation,' in Conference on Robot Learning , 2018. source
    • cites [23] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei, 'Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity,' arXiv preprint arXiv:1911.04052 , 2019. source
    • cites [24] A. Mandlekar, D. Xu, R. Mart´ ın-Mart´ ın, Y . Zhu, L. Fei-Fei, and S. Savarese, 'Human-in-theloop imitation learning using remote teleoperation,' arXiv preprint arXiv:2012.06733 , 2020. source
    • cites [25] A. Tung, J. Wong, A. Mandlekar, R. Mart´ ın-Mart´ ın, Y. Zhu, L. Fei-Fei, and S. Savarese, 'Learning multi-arm manipulation through collaborative teleoperation,' arXiv preprint arXiv:2012.06738 , 2020. source
    • cites [26] J. Wong, A. Tung, A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Mart´ ın-Mart´ ın, 'Error-aware imitation learning from teleoperation data for mobile manipulation,' in Conference on Robot Learning . PMLR, 2022, pp. 1367-1378. source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § E Full Related Work · p. 21

    Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”
    Quote in the paperNot a substantial claim8 of 8 sources checked
    • cites [71] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, 'Datasetgan: Efficient labeled data factory with minimal human effort,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 10 14510 155. source
    • cites [72] D. Li, H. Ling, S. W. Kim, K. Kreis, S. Fidler, and A. Torralba, 'Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 21 330-21 340. source
    • cites [73] A. Kar, A. Prakash, M.-Y. Liu, E. Cameracci, J. Yuan, M. Rusiniak, D. Acuna, A. Torralba, and S. Fidler, 'Meta-sim: Learning to generate synthetic datasets,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 4551-4560. source
    • cites [74] J. Devaranjan, A. Kar, and S. Fidler, 'Meta-sim2: Unsupervised learning of scene structure for synthetic data generation,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16 . Springer, 2020, pp. 715-733. source
    • cites [75] S. W. Kim, J. Philion, A. Torralba, and S. Fidler, 'Drivegan: Towards a controllable highquality neural simulation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 5820-5829. source
    • cites [76] D. Paschalidou, A. Kar, M. Shugrina, K. Kreis, A. Geiger, and S. Fidler, 'Atiss: Autoregressive transformers for indoor scene synthesis,' Advances in Neural Information Processing Systems , vol. 34, pp. 12 013-12 026, 2021. source
    • cites [77] S. Tan, K. Wong, S. Wang, S. Manivasagam, M. Ren, and R. Urtasun, 'Scenegen: Learning to generate realistic traffic scenes,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 892-901. source

      Citation [78] is attached to the claim about benchmarking motion planning. The Introduction discusses using simulated traffic scenes to test self-driving vehicles generally, but the Sim2Real Evaluation (§4.4; page unavailable) describes generating scenes to train and evaluate a perception model. Searches for motion-planning benchmarks found no passage establishing that use. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [78] C. Chamzas, C. Quintero-Pena, Z. Kingston, A. Orthey, D. Rakita, M. Gleicher, M. Toussaint, and L. E. Kavraki, 'Motionbenchmaker: A tool to generate and benchmark motion planning datasets,' IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 882-889, 2021. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § E Full Related Work · p. 21

    Prior work has attempted to develop systems that can selectively query humans for demonstrations when they are needed, in order to reduce human operator time and burden.

    “Prior work has also attempted to develop systems that can selectively query humans for demonstrations when they are needed, in order to reduce human operator time and burden [67-70].”
    Quote in the paperNot a substantial claim3 of 4 sources checked
    • cites [67] J. Zhang and K. Cho, 'Query-efficient imitation learning for end-to-end autonomous driving,' arXiv preprint arXiv:1605.06450 , 2016. source
    • cites [68] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. Novoseller, and K. Goldberg, 'Lazydagger: Reducing context switching in interactive imitation learning,' in 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE) . IEEE, 2021, pp. 502-509. source
    • cites [69] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg, 'Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning,' arXiv preprint arXiv:2109.08273 , 2021. source
    • cites [70] S. Dass, K. Pertsch, H. Zhang, Y. Lee, J. J. Lim, and S. Nikolaidis, 'Pato: Policy assisted teleoperation for scalable robot data collection,' arXiv preprint arXiv:2212.04708 , 2022. source

  • May combine two claims.

    Interpretationcitation
    § J.1 Replay-Based Imitation Learning · p. 27

    MimicGen uses a similar mechanism to replay-based methods to generate datasets, allowing full compatibility with a wide spectrum of offline policy learning algorithms.

    “By contrast, MimicGen uses a similar mechanism to generate datasets -this allows full compatibility with a wide spectrum of offline policy learning algorithms [57].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [57] S. Levine, A. Kumar, G. Tucker, and J. Fu, 'Offline reinforcement learning: Tutorial, review, and perspectives on open problems,' arXiv preprint arXiv:2005.01643 , 2020. source

      Section 6.1 (p. 29) describes using replay-buffer data as a dataset for offline reinforcement learning, and Section 2.2 (p. 8) says that, in principle, off-policy algorithms can learn from a fixed dataset. The paper does not discuss MimicGen or establish that its generated datasets are fully compatible with a wide spectrum of offline policy-learning algorithms; Section 2.2 cautions that not all such algorithms are effective offline. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • The authors state this as an assumption: Tasks consist of a known sequence of object-centric subtasks.
    • The authors state this as an assumption: Access to metrics that allow the end of each subtask to be detected automatically.
    • The authors state this as an assumption: The sequence of object-centric subtasks is known.

  • Classified as probably not a substantial claim.

    Premisecitation
    § I Different Demonstrators · p. 26

    The robomimic study found significant difficulty in learning from datasets produced by less experienced operators.

    “the robomimic study [7] found that there can be significant difficulty in learning from datasets produced by less experienced operators.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın, 'What matters in learning from offline human demonstrations for robot manipulation,' in Conference on Robot Learning (CoRL) , 2021. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § E Full Related Work · p. 21

    Several related efforts also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.

    “Several related efforts [3-6, 27] also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.”
    Quote in the paperNot a substantial claim5 of 5 sources checked
    • cites [3] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, 'Bcz: Zero-shot task generalization with robotic imitation learning,' in Conference on Robot Learning . PMLR, 2022, pp. 991-1002. source
    • cites [4] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog et al. , 'Do as i can, not as i say: Grounding language in robotic affordances,' arXiv preprint arXiv:2204.01691 , 2022. source
    • cites [5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , 'Rt-1: Robotics transformer for real-world control at scale,' arXiv preprint arXiv:2212.06817 , 2022. source

      The Data section reports approximately 130,000 robot demonstrations collected over 17 months, and “Data collection at scale” describes human operators providing them. The paper specifies a fleet of 13 robots, but these passages do not establish that a large number of human operators were involved. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [6] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, 'Bridge data: Boosting generalization of robotic skills with cross-domain datasets,' arXiv preprint arXiv:2109.13396 , 2021. source

      The data collection protocol (p. 4) reports 7,200 demonstrations, and the robotic system overview (p. 3) describes a user controlling the robot to collect them. Our copy does not say that a large number of human operators participated or specify an extended collection period. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [27] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, 'Interactive language: Talking to robots in real time,' arXiv preprint arXiv:2210.06407 , 2022. source

  • May combine two claims.

    Findingown data
    § H Real Robot Results · p. 25

    Diffusion Policy trained on real-world Stack MimicGen data achieved 76% success across 50 evaluations, significantly improving over BC-RNN's 36%.

    “We trained Diffusion Policy [100], a recent state-of-the-art imitation learning model, on our real-world Stack dataset. The new agent achieved a success rate of 76% across 50 evaluations - a significant improvement over the 36% success rate achieved by BC-RNN.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [100] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, 'Diffusion policy: Visuomotor policy learning via action diffusion,' arXiv preprint arXiv:2303.04137 , 2023. source
    • The authors state this as an assumption: Pose estimates at the start of each object-centric subtask are available.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § E Full Related Work · p. 21

    Some works have collected large datasets using pre-programmed demonstrators in simulation.

    “Some works have collected large datasets using pre-programmed demonstrators in simulation [18-22]”
    Quote in the paperNot a substantial claim2 of 5 sources checked
    • cites [18] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, 'Rlbench: The robot learning benchmark & learning environment,' IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3019-3026, 2020. source
    • cites [19] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani et al. , 'Transporter networks: Rearranging the visual world for robotic manipulation,' arXiv preprint arXiv:2010.14406 , 2020. source
    • cites [20] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, 'Vima: General robot manipulation with multimodal prompts,' arXiv preprint arXiv:2210.03094 , 2022. source
    • cites [21] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao et al. , 'Maniskill2: A unified benchmark for generalizable manipulation skills,' arXiv preprint arXiv:2302.04659 , 2023. source
    • cites [22] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox, 'Imitating task and motion planning with visuomotor transformers,' arXiv preprint arXiv:2305.16309 , 2023. source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § H Real Robot Results · p. 24

    Recent sim2real efforts have been very promising; several works have transferred policies trained via imitation learning from sim to real.

    “Recent sim2real efforts have been very promising - several works [61, 95-98] have been able to transfer policies trained via imitation learning from sim to real.”
    Quote in the paperNot a substantial claim5 of 5 sources checked
    • cites [61] M. Khansari, D. Ho, Y. Du, A. Fuentes, M. Bennice, N. Sievers, S. Kirmani, Y. Bai, and E. Jang, 'Practical imitation learning in the real world via task consistency loss,' arXiv preprint arXiv:2202.01862 , 2022. source
    • cites [95] E. Valassakis, N. Di Palo, and E. Johns, 'Coarse-to-fine for sim-to-real: Sub-millimetre precision across wide task spaces,' in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 5989-5996. source
    • cites [96] P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid, 'Instruction-driven history-aware policies for robotic manipulations,' in Conference on Robot Learning . PMLR, 2023, pp. 175-187. source
    • cites [97] H. Ha, P. Florence, and S. Song, 'Scaling up and distilling down: Language-guided robot skill acquisition,' arXiv preprint arXiv:2307.14535 , 2023. source
    • cites [98] A. Agarwal, A. Kumar, J. Malik, and D. Pathak, 'Legged locomotion in challenging terrains using egocentric vision,' in Conference on Robot Learning . PMLR, 2023, pp. 403-415. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation
    § H Real Robot Results · p. 24

    MimicGen is entirely complementary to domain randomization techniques.

    “Furthermore, MimicGen is entirely complementary to domain randomization techniques [99], which could also be applied to assist in transferring policies to the real world.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [99] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, 'Domain randomization for transferring deep neural networks from simulation to the real world,' in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2017, pp. 23-30. source

  • May combine two claims.

    Findingreasoning
    § 7 Limitations · p. 8

    MimicGen only filters data generation attempts based on task success, so generated datasets can be biased.

    “MimicGen only filters data generation attempts based on task success, so generated datasets can be biased (Appendix R).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § C FAQ · p. 17

    MimicGen can perform tasks with non-trivial contact-rich manipulation, including Gear Assembly with 1mm insertion tolerance and Picture Frame Assembly requiring alignment of 4 holes with 4mm tolerance each.

    “This includes tasks with non-trivial contactrich manipulation (Gear Assembly has 1mm insertion tolerance , and Picture Frame Assembly needs alignment of 4 holes with 4mm tolerance each )”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • May combine two claims.

    Findingown data
    § 6.3 MimicGen Analysis · p. 7

    There is a large jump in performance from 200 to 1000 demos, but not much from 1000 to 5000, showing diminishing returns on generating more data.

    “In Fig. 4, we train agents on 200, 1000, and 5000 demos generated by MimicGen across several tasks. There is a large jump in performance from 200 to 1000, but not much from 1000 to 5000, showing that there can be diminishing returns on generating more data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2 Comparing MimicGen to using more human data · p. 6

    In this work, we are able to make effective use of just 10 human demonstrations to generate a set of 1000 demonstrations and learn proficient agents from them (76.0% and 58.7% low-dim, 53.3% and 50.7% image).

    “By contrast, in this work we are able to make effective use of just 10 human demonstrations to generate a set of 1000 demonstrations and learn proficient agents from them (76.0% and 58.7% low-dim, 53.3% and 50.7% image).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Applications of MimicGen · p. 6

    In the Mobile Kitchen task, MimicGen yields a gain from 2.0% to 46.7% (image) and 2.7% to 76.7% success rate (low-dim), showing it can be applied beyond static tabletop manipulation.

    “Applying MimicGen to mobile manipulation. In the Mobile Kitchen task MimicGen yields a gain from 2.0% to 46.7% (image, Fig. 4) and 2.7% to 76.7% success rate (low-dim, Table Q.1 in Appendix), highlighting that our method can be applied to tasks beyond static tabletop manipulation.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 6.4 Real Robot Evaluation · p. 8

    MimicGen can be applied to real-world robot arms and tasks, with generation success rates of 82.3% for Stack and 52.1% for Coffee, and trained agents achieving 36% and 14% success respectively, while source agents had 0% success.

    “We validate that MimicGen can be applied to real-world robot arms and tasks. We collect 10 source demonstrations for each task in narrow regions of the workspace ( D 0 ) and then generate demonstrations (200 for Stack, 100 for Coffee) for large regions of the workspace ( D 1 ) (see Fig. 5). The generation success rate was 82.3% for Stack (243 attempts) and 52.1% for Coffee (192 attempts), showing”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 7 Limitations · p. 8

    MimicGen uses linear interpolation between human segments, which does not guarantee collision-free motion, and can potentially hurt agent performance.

    “MimicGen uses linear interpolation between human segments (Appendix N.2), which does not guarantee collision-free motion, and can potentially hurt agent performance (Appendix H).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Applications of MimicGen · p. 5

    MimicGen data vastly improves agent performance on the source task, with substantial improvement across all tasks compared to agents trained on small source datasets.

    “MimicGen data vastly improves agent performance on the source task.”
    • Also stated in: § 6.1 Applications of MimicGen · p. 5 · plain
      “Comparing the performance of agents trained on the small source datasets vs. those trained on D 0 datasets generated by MimicGen, we see that there is substantial improvement across all our tasks (see Fig. 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Applications of MimicGen · p. 5

    In several cases, certain objects in the 10 source demonstrations never moved, but data was generated and policies were trained on regimes where the objects move in substantial regions of the robot workspace.

    “In several cases, certain objects in the 10 source demonstrations never moved (the peg in Square, the tripod in Threading, the base in Three Piece Assembly, etc), but data was generated (and policies consequently were trained) on regimes where the objects move in substantial regions of the robot workspace.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingnone given
    § E Full Related Work · p. 21

    MimicGen can work for a large variety of manipulation tasks and behaviors beyond standard pick-and-place tasks.

    “we show that MimicGen can be applied to a broad range of tasks.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • May combine two claims.

    Findingown data
    § 6.1 Applications of MimicGen · p. 6

    MimicGen can use demonstrations from inexperienced human operators and different teleoperation devices, with comparable performance to Fig. 4.

    “MimicGen can use demonstrations from inexperienced human operators and different teleoperation devices. Surprisingly, policies trained on these MimicGen datasets have comparable performance to those in Fig. 4. See Appendix I for the full set of results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2 Comparing MimicGen to using more human data · p. 7

    In most cases, agent performance is similar on 200 MimicGen demos and 200 human demos, despite the 200 MimicGen demos being generated from just 10 human demos, showing a small number of human demos can be as effective (or even more) than a large number when used with MimicGen.

    “In most cases, agent performance is similar, despite the 200 MimicGen demos being generated from just 10 human demos - a small number of human demos can be as effective (or even more) than a large number of them when used with MimicGen.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Applications of MimicGen · p. 6

    MimicGen can generate data for different objects, with policies trained on unseen mugs achieving 90.7% and 75.3% success.

    “MimicGen can generate data for different objects. The source dataset in the Mug Cleanup task contains just one mug, but we generate demonstrations with MimicGen for an unseen mug ( O 1 ) and for a set of 12 mugs ( O 2 ). Policies trained on these datasets have substantial task success rates (90.7% and 75.3% respectively) (full results in Appendix G).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § D Limitations · p. 19

    MimicGen tracks a target end effector pose path on each data generation attempt, which allows data generation for robot arms with different dynamics.

    “Recall that on each data generation attempt, MimicGen tracks a target end effector pose path (Sec. 4.2) - this allows data generation for robot arms with different dynamics (Appendix F)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § D Limitations · p. 19

    MimicGen was presented with results on the Mobile Kitchen task, which requires mobile manipulation.

    “In Sec. 6.1, we presented results for MimicGen on the Mobile Kitchen task, which requires mobile manipulation (base and arm motion).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: MimicGen assumes that the robot does not move the mobile base and arm simultaneously.

  • May combine two claims.

    Findingown data
    § C FAQ · p. 17

    In simulation, while data generation rates can decrease with pose estimate errors, policies trained on the generated data maintained the same level of performance.

    “We further investigated tolerance to pose estimate errors in simulation (see Appendix U) and found that while data generation rates can decrease (so data collection will take longer), policies trained on the generated data maintained the same level of performance.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.

  • May combine two claims.

    Findingreasoning
    § D Limitations · p. 19

    This filtering does not prevent the generated datasets from being biased or having artifacts.

    “However, this does not prevent the generated datasets from being biased, or having artifacts (see discussion in Appendix R).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • May combine two claims.

    Findingown data
    § F Robot Transfer · p. 22

    Policy success rates are significantly higher and remarkably similar across robots, e.g., 80%-91% on Square D0 and 89%-98% on Threading D0.

    “the policy success rates are significantly higher and remarkably similar across robots (for example, 80%-91% on Square D 0 and 89%-98% on Threading D 0 -see the full imagebased agent results in Table F.2 and low-dim agent results in Table F.3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § H Real Robot Results · p. 24

    Real-world data collection success rates were 82.3% for Stack and 52.1% for Coffee.

    “While we saw a significantly high data collection success rate (82.3% for Stack, 52.1% for Coffee)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Pose estimates at the start of each object-centric subtask are available.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Findingnone given
    § H Real Robot Results · p. 24

    The current pose estimation pipeline works reasonably well.

    “Note that while the current pose estimation pipeline works reasonably well, our framework is not specific to certain types of perception methods.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Pose estimates at the start of each object-centric subtask are available.

  • May combine two claims.

    Findingown data
    § H Real Robot Results · p. 24

    Real-world policy success rates were much lower than simulation: 36% vs. 100% for Stack, and 14% vs. ~90% for Coffee.

    “we saw much lower policy success rate on these tasks than in simulation (36% vs. 100% for Stack, and 14% vs. ∼ 90%for Coffee)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Pose estimates at the start of each object-centric subtask are available.

  • May combine two claims.

    Findingown data
    § P Data Generation Success Rates · p. 40

    Gear Assembly has data generation rates of 46.9% (D0), 8.2% (D1), and 7.1% (D2) but policy success rates of 92.7% (D0), 76.0% (D1), and 64.0% (D2).

    “An extreme example is the Gear Assembly task which has data generation rates of 46 . 9% ( D 0 ), 8 . 2% ( D 1 ), and 7 . 1% ( D 2 ) but policy success rates of 92 . 7% ( D 0 ), 76 . 0% ( D 1 ), and 64 . 0% ( D 2 ).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § T Data Generation with Multiple Seeds · p. 44

    Data generation success rates have very tight variance (less than 1%) and do not deviate from reported data generation rates by more than 0.6% across 3 seeds.

    “The data generation success rates have very tight variance (less than 1%) and do not deviate from our reported data generation rates (Appendix P) by more than 0.6%.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § Q Low-Dim Policy Training Results · p. 41

    There is a large improvement across all tasks on the default distribution (D0) for low-dim agents trained on MimicGen datasets compared to source demos.

    “There is a large improvement across all tasks on the default distribution ( D 0 ) and agents are performant on the broader distributions ( D 1 , D 2 ).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § N.3 Reference Segment Selection · p. 36

    Just replaying the same task instances from the source human data with action noise is not sufficient, although it improves results over just using the source human data.

    “They also show that just replaying the same task instances from the source human data with action noise is not sufficient (although it does improve results over just using the source human data).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § O Policy Training Details · p. 39

    Using an increased learning rate (1e-3 instead of 1e-4) for policies trained on low-dim observations speeds up policy convergence on large datasets.

    “with the exception of an increased learning rate (1e-3 instead of 1e-4) for policies trained on low-dim observations, as we found it to speed up policy convergence on large datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § U Tolerance to Pose Estimation Error · p. 45

    MimicGen is fully functional in real-world settings and can operate with minimal assumptions (e.g., no special tags or pose trackers) by using pose estimation methods.

    “In the main text, we demonstrated that MimicGen is fully functional in real-world settings and can operate with minimal assumptions (e.g. no special tags or pose trackers) by using pose estimation methods”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § T Data Generation with Multiple Seeds · p. 44

    Mean policy success rates are extremely close to reported low-dim results (less than 2% deviation) across 3 seeds.

    “Furthermore, the mean policy success rates are extremely close to our reported results for low-dim agents in Table Q.1 (less than 2% deviation).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 6.2 Comparing MimicGen to using more human data · p. 7

    MimicGen can easily generate more demonstrations to improve performance, unlike the time-consuming nature of collecting more human data.

    “MimicGen can also easily generate more demonstrations to improve performance (see Sec. 6.3), unlike the time-consuming nature of collecting more human data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction · p. 2

    These replay-based methods make assumptions about specific tasks and algorithms that limit their applicability.

    “While promising, these methods make assumptions about specific tasks and algorithms that limit their applicability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 6.3 MimicGen Analysis · p. 7

    Both the initial set of source demos provided to MimicGen and how segments from these demos are chosen during each generation attempt can matter.

    “This indicates that both the initial set of source demos provided to MimicGen ( D src), and how segments from these demos are chosen during each generation attempt ( τ i for each subtask, see Sec. 4.1) can matter.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.4 Real Robot Evaluation · p. 8

    The lower numbers than from simulation might be due to the larger number of interpolation steps used in the real world for hardware safety (50 total instead of 5).

    “The lower numbers than from simulation might be due to the larger number of interpolation steps we used in the real world for hardware safety (50 total instead of 5) - these motions are difficult for the agent to imitate since there is little association between the intermediate motion and observations ( see Appendix H for more experiments and discussion ).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § C FAQ · p. 17

    MimicGen is easy to implement and apply in practice, since only a small number of assumptions are needed.

    “Additionally, in contrast to many offline data augmentation methods, MimicGen is easy to implement and apply in practice, since only a small number of assumptions are needed (see Sec. 3).”
    • Also stated in: § J.2 Offline Data Augmentation · p. 27 · plain
      “In contrast to many offline data augmentation methods, MimicGen is easy to implement and apply in practice, since only a small number of assumptions are needed (see Sec. 3).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Tasks consist of a known sequence of object-centric subtasks.
    • The authors state this as an assumption: Access to metrics that allow the end of each subtask to be detected automatically.
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.
    • The authors state this as an assumption: The sequence of object-centric subtasks is known.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § E Full Related Work · p. 21

    MimicGen only needs an operator to collect a few minutes of demonstrations at the start of the process.

    “In contrast, MimicGen only needs an operator to collect a few minutes of demonstrations at the start of the process.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § C FAQ · p. 17

    The interpolation segments are unaware of the geometry in the scene and consist of naive linear interpolation, so these segments might result in unintended collisions.

    “One reason is that the interpolation segments are unaware of the geometry in the scene and consist of naive linear interpolation (see Appendix N.2), so these segments might result in unintended collisions.”
    • Also stated in: § D Limitations · p. 19 · plain
      “However, this method is not aware of scene geometry, and consequently can result in unintended collisions if objects happen to be in the way of the straight line path.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § C FAQ · p. 17

    The way source segments are transformed does not consider arm kinematics, so the end effector poses where segments start might be difficult to reach.

    “Another is that the way source segments are transformed do not consider arm kinematics, so the end effector poses where segments start might be difficult to reach.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § C FAQ · p. 17

    Certain source dataset motions might be easier for the controller to track than others.

    “A third reason is that certain source dataset motions might be easier for the controller to track than others.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).
    • The authors state this as an assumption: MimicGen assumes that the sequence of object-centric subtasks does not change.
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.
    • The authors state this as an assumption: MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

  • May combine two claims.

    Interpretationown data
    § C FAQ · p. 17

    MimicGen is compatible with pose estimation methods and has some tolerance to pose error.

    “Thus, MimicGen is compatible with pose estimation methods and has some tolerance to pose error.”
    • Also stated in: § D Limitations · p. 19 · boosted
      “we demonstrated that we can run MimicGen in the real world, using pose estimation methods (Sec. 6.4 and Appendix H), and has some tolerance to errors in pose estimates (Appendix U).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § H Real Robot Results · p. 24

    MimicGen framework is not specific to certain types of perception methods.

    “our framework is not specific to certain types of perception methods.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Pose estimates at the start of each object-centric subtask are available.

  • May combine two claims.

    Interpretationreasoning
    § J.2 Offline Data Augmentation · p. 27

    MimicGen generates datasets with new objects that are a critical part of the manipulation task, adapting behavior to new contexts.

    “By contrast, MimicGen generates datasets with new objects that are a critical part of the manipulation task - it seeks to generate data by adapting behavior to new contexts.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Tasks consist of a known sequence of object-centric subtasks.
    • The authors state this as an assumption: Access to metrics that allow the end of each subtask to be detected automatically.
    • The authors state this as an assumption: The sequence of object-centric subtasks is known.

  • May combine two claims.

    Interpretationreasoning
    § J.2 Offline Data Augmentation · p. 27

    MimicGen generates new datasets through environment interaction by re-purposing existing human demonstrations, which automatically leads to physically-consistent data because generation is online.

    “Instead, MimicGen opts for generating new datasets through environment interaction by re-purposing existing human demonstrations - this automatically leads to physicallyconsistent data, since generation is online.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Tasks consist of a known sequence of object-centric subtasks.
    • The authors state this as an assumption: Access to metrics that allow the end of each subtask to be detected automatically.
    • The authors state this as an assumption: The sequence of object-centric subtasks is known.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § E Full Related Work · p. 21

    It can be difficult to scale pre-programmed demonstrator approaches up to more complex tasks.

    “however, it can be difficult to scale these approaches up to more complex tasks”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § K.1 How Tasks can be broken up into Object-Centric Subtasks · p. 28

    A wide variety of tasks can be broken down into object-centric subtasks.

    “We also emphasize that a wide variety of tasks can be broken down into object-centric subtasks (e.g. Assumption 2 applies to a wide variety of tasks, especially those that are commonly considered in the robot learning community).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Tasks consist of a known sequence of object-centric subtasks.
    • The authors state this as an assumption: The sequence of object-centric subtasks is known.

  • May combine two claims.

    Interpretationreasoning
    § R Bias and Artifacts in Generated Data · p. 42

    MimicGen only checks for successful task completion when deciding whether to accept a generated trajectory, which means there might be undesirable behaviors such as collisions between the robot and non-task-relevant parts of the world.

    “Another reason is that MimicGen only checks for a successful task completion when deciding whether to accept a generated trajectory. This means that there might be undesirable behaviors such as collisions between the robot and certain parts of the world (including objects that are not task-relevant).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § U Tolerance to Pose Estimation Error · p. 45

    MimicGen can be tolerant to noisy object pose estimation and is suitable for real-world data collection.

    “This shows that MimicGen can be tolerant to noisy object pose estimation, and is suitable for real-world data collection.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § U Tolerance to Pose Estimation Error · p. 45

    The data generation process has some tolerance to pose error and can operate without having access to perfect pose estimates.

    “Consequently, the data generation process has some tolerance to pose error and can operate without having access to perfect pose estimates.”
    Quote in the paperSubstantial claim
65 of 65

  • The authors state this as an assumption.

    The action space consists of delta-pose commands for an end-effector controller and a gripper open/close command.

    “Assumption 1: delta end effector pose action space. The action space A consists of delta-pose commands for an end-effector controller and a gripper open/close command.”

  • The authors state this as an assumption.

    Tasks consist of a known sequence of object-centric subtasks.

    “Assumption 2: tasks consist of a known sequence of object-centric subtasks. Let O = { o 1 , ..., o K } be the set of objects in a task M . As in Di Palo et al. [11], we assume that tasks consist of a sequence of object-centric subtasks ( S 1 ( o S 1 ) , S 2 ( o S 2 ) , ..., S M ( o S M )) , where the manipulation in each subtask S i ( o S i ) is relative to a single object's coordinate frame ( o S”
    “we assume that tasks consist of a known sequence of object-centric subtasks”
    “As in Di Palo et al. [11], we assume that tasks consist of a sequence of object-centric subtasks”

  • The authors state this as an assumption.

    Object poses can be observed at the start of each subtask during data collection.

    “Assumption 3: object poses can be observed at the start of each subtask during data collection. Weassume that we can observe the pose of the relevant object o S i at the start of each subtask S i ( o S i ) during data collection (not, however, during policy deployment).”

  • The authors state this as an assumption.

    Access to metrics that allow the end of each subtask to be detected automatically.

    “In this work, to parse source demonstrations into segments for each subtask, we assume access to metrics that allow the end of each subtask to be detected automatically (see Appendix K for full details).”
    “We assume access to metrics that allow the end of each subtask to be detected automatically.”

  • The authors state this as an assumption.

    The motion in a source subtask segment is relative to the pose of the object at the start of the segment.

    “Since this motion is assumed to be relative to the pose of the object o S i (frame O 0 with pose T O 0 W ) at the start of the segment, we will transform τ i according to the new pose of the corresponding object in the current scene (frame O ′ 0 with pose T O ′ 0 W ) so that the relative poses between the target pose frame and the object frame are preserved at each timestep”

  • The authors state this as an assumption.

    Novel objects come from the same category.

    “MimicGen was demonstrated on quasi-static tasks with rigid objects, and novel objects were assumed to come from the same category.”

  • The paper relies on this without support.

    Large portions of human demonstration datasets contain similar manipulation skills applied in different contexts, so trajectories can be repurposed in new contexts.

    “Large portions of these datasets may contain similar manipulation skills applied in different contexts or situations. For example, human operators may demonstrate very similar robot trajectories to grasp a mug, regardless of its location on one countertop or another. Re-purposing these trajectories in new contexts can be a way to generate diverse data without much human effort.”

  • The authors state this as an assumption.

    MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask).

    “MimicGen assumes knowledge of the object-centric subtasks in a task (which object is involved at each subtask)”

  • The authors state this as an assumption.

    MimicGen assumes that the sequence of object-centric subtasks does not change.

    “and also assumes that this sequence of subtasks does not change (Assumption 2, Sec. 3).”

  • The authors state this as an assumption.

    During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask.

    “During data generation, at the start of each object-centric subtask, MimicGen requires an object pose estimate of the reference object for that subtask (Assumption 3, Sec 3).”

  • The authors state this as an assumption.

    MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object.

    “MimicGen assumes each task is composed of a sequence of subtasks that are each relative to exactly one object (Assumption 2, Sec. 3).”

  • The authors state this as an assumption.

    MimicGen assumes aligned canonical coordinate frames for all objects in a category.

    “We also assumed aligned canonical coordinate frames for all objects in a category”
    “We also assumed aligned canonical coordinate frames for all objects in a category”

  • The authors state this as an assumption.

    MimicGen assumes that objects are well-described by their poses (e.g., rigid bodies, not soft objects).

    “and that the objects are well-described by their poses (e.g. rigid bodies, not soft objects).”
    “and that the objects are well-described by their poses (e.g. rigid bodies, not soft objects).”

  • The authors state this as an assumption.

    MimicGen assumes that the robot does not move the mobile base and arm simultaneously.

    “First, it assumes that the robot does not move the mobile base and arm simultaneously.”

  • The paper relies on this without support.

    Replay-based agents generally conform to a specific policy architecture.

    “First, replay-based agents generally conform to a specific policy architecture”

  • The authors state this as an assumption.

    The sequence of object-centric subtasks is known.

    “We assume this sequence is known.”

  • The authors state this as an assumption.

    Aligned canonical coordinate frames for all objects in a category.

    “We also assumed aligned canonical coordinate frames for all objects in a category”

  • The authors state this as an assumption.

    Objects are well-described by their poses (e.g., rigid bodies, not soft objects).

    “and that the objects are welldescribed by their poses (e.g. rigid bodies, not soft objects).”

  • The authors state this as an assumption.

    Pose estimates at the start of each object-centric subtask are available.

    “MimicGen requires pose estimates at the start of each object-centric subtask (Assumption 3, Sec. 3).”

  • The authors state this as an assumption.

    The action space consists of delta-pose commands for an end effector controller.

    “We assume that the action space A consists of delta-pose commands for an end effector controller (Assumption 1, Sec. 3).”

  • The authors state this as an assumption.

    Actions are 7-dimensional, with the first 3 components being desired translation, the next 3 desired delta rotation, and the final component gripper open/close.

    “As in [7], we assume that actions are 7-dimensional, where the first 3 components are the desired translation from the current end effector position, the next 3 components represent the desired delta rotation from the current end effector rotation, and the final component is the gripper open/close action.”

  • The authors state this as an assumption.

    A source segment that contains mobile base actions can be split into three contiguous sub-segments: manipulator actions, mobile base actions, and manipulator actions.

    “If a segment does contain mobile base actions we assume that the segment can be split into three contiguous sub-segments: (1) a sub-segment involving manipulator actions, (2) a subsequent sub-segment involving mobile base actions, and (3) a final sub-segment involving manipulator actions.”

  • The authors state this as an assumption.

    The source subtask segment consists of target poses for the end effector controller.

    “We have chosen a source subtask segment consisting of target poses for the end effector controller (Assumption 1, Sec. 3)”
23 of 23

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RoboCasa supports room-scale scenes, unlike other frameworks that include mobile manipulation in smaller portions of a room.

    “RoboCasa also supports room-scale scenes, unlike other frameworks that include mobile manipulation in smaller portions of a room [10, 34].”

    From RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024) · cites this paper as [34]

    The Experiment Setup (p. 5) identifies MimicGen’s Mobile Kitchen task as mobile manipulation, and Tasks and Task Variants (p. 33) describes movement among a counter, stove, and sink. The paper gives small object-initialization regions but does not establish that its scene occupies only a smaller portion of a room. It also does not address RoboCasa’s room-scale scenes. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • MimicGen can automatically synthesize rich datasets from a seed set of human demonstrations by adapting them to new settings.

    “MimicGen can automatically synthesize rich datasets from a seed set of human demonstrations by adapting them to new settings.”

    From RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024) · cites this paper as [34]

  • RoboCasa integrates MimicGen to generate 100K additional trajectories for atomic tasks.

    “To augment our datasets, we extend MimicGen [34] to generate 100K additional trajectories for our atomic tasks.”

    From RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024) · cites this paper as [34]

  • MimicGen data generation attempts are not always successful; it uses rejection sampling to keep only successful attempts.

    “MimicGen data generation attempts are not always successful. In practice, it employs a rejection sampling scheme only to keep generation attempts that lead to task success.”

    From RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024) · cites this paper as [34]

  • RoboCasa provides large-scale datasets of task demonstrations through human teleoperation and MimicGen.

    “we provide large-scale datasets of task demonstrations through a combination of human teleoperation and the MimicGen system [34]”

    From RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024) · cites this paper as [34]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Segment-transform equation reverses the object-pose transformation

    Minor

    The stated requirement is to preserve the controller target's pose relative to the object. The transform shown instead applies the inverse change in object pose; solving the relative-pose equations in Appendix M gives T_O′W (T_OW)−1 T_CW, not the equation printed in Section 4.2.

  • Real-robot source region is assigned opposite colors

    Minor

    Figure 5 identifies blue as the source reset distribution and orange as the generated distribution. Section 6.4 instead calls the orange regions the regions where the source data came from.

  • Three Piece Assembly names the wrong insertion target

    Minor

    The task description says the second piece is inserted into the first, but its subtask list says to place it into piece 2 itself.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [75]S. W. Kim, J. Philion, A. Torralba, and S. Fidler, 'Drivegan: Towards a controllable highquality neural simulation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 5820-5829.
  2. [85]M. Ester, H.-P. Kriegel, J. Sander, X. Xu et al. , 'A density-based algorithm for discovering clusters in large spatial databases with noise.' in kdd , vol. 96, no. 34, 1996, pp. 226-231.
  3. [103]L. Chen, R. Paleja, and M. Gombolay, 'Learning from suboptimal demonstration via selfsupervised reward regression,' in Conference on robot learning . PMLR, 2021, pp. 12621277.

Cited by

Sorted by citations

Related

Authors
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.17596
arXiv
2310.17596
OpenAlex
W4387994700
Cited by
5
References
112
Citations by year
2025: 1 · 2024: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
imitation learning, human demonstrations, robot learning, data augmentation, coffee preparation, robot manipulators, long-horizon tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review