CoChat
Sign Up
PreprintCited by 94Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

Open access
Ebert, Frederik, Yang, Yanlai, Schmeckpeper, Karl, Bucher, Bernadette, Georgakis, Georgios, Daniilidis, Kostas, Finn, Chelsea, Levine, Sergey·2021·arXiv
DOI 10.48550/arxiv.2109.13396arXiv 2109.13396
Open this paper in CoChat — free accountOpen PDF

Abstract

Robot learning holds the promise of learning policies that generalize broadly. However, such generalization requires sufficiently diverse datasets of the task of interest, which can be prohibitively expensive to collect. In other fields, such as computer vision, it is common to utilize shared, reusable datasets, such as ImageNet, to overcome this challenge, but this has proven difficult in robotics. In this paper, we ask: what would it take to enable practical data reuse in robotics for end-to-end skill learning? We hypothesize that the key is to use datasets with multiple tasks and multiple domains, such that a new user that wants to train their robot to perform a new task in a new domain can include this dataset in their training process and benefit from cross-task and cross-domain generalization. To evaluate this hypothesis, we collect a large multi-domain and multi-task dataset, with 7,200 demonstrations constituting 71 tasks across 10 environments, and empirically study how this data can improve the learning of new tasks in new environments. We find that jointly training with the proposed dataset and 50 demonstrations of a never-before-seen task in a new domain on average leads to a 2x improvement in success rate compared to using target domain data alone. We also find that data for only a few tasks in a new domain can bridge the domain gap and make it possible for a robot to perform a variety of prior tasks that were only seen in other domains. These results suggest that reusing diverse multi-task and multi-domain datasets, including our open-source dataset, may pave the way for broader robot generalization, eliminating the need to re-collect data for each new robot learning project.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2109.13396

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several related efforts also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.

    “Several related efforts [3-6, 27] also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [6]

    The data collection protocol (p. 4) reports 7,200 demonstrations, and the robotic system overview (p. 3) describes a user controlling the robot to collect them. Our copy does not say that a large number of human operators participated or specify an extended collection period. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • A large number of robot learning works and benchmarks use quasi-static tasks, making the system broadly applicable.

    “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3-7,14,18,19,22,30,33,51, 55, 56, 64-66], making the system broadly applicable.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [6]

    The Related Work section (p. 2) surveys robot-learning datasets, and the dataset description (pp. 3–4) describes this paper’s kitchen tasks. Neither characterizes those tasks or other benchmarks as quasi-static, so our copy does not establish the attributed reason for the system’s broad applicability (searched: quasi-static tasks, static and dynamic manipulation). (LLM, conf=0.70)

    Experimental — accuracy not yet measured

  • The tasks MimicGen was demonstrated on have pose variation well beyond typical prior works using BC from human demos.

    “The tasks also have pose variation well beyond typical prior works using BC from human demos [1,3-7,30,33,55,56].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [6]

    The Related Work section (p. 2) compares prior datasets by numbers of tasks and domains, not by pose variation. The dataset description (p. 4) describes position randomization in Bridge Data, but our copy does not compare pose variation in prior behavioral-cloning work with the tasks demonstrated by MimicGen (searched: pose variation, object positions, behavioral cloning). (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. [1][1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' Advances in neural information processing systems , vol. 25, pp. 1097-1105, 2012. -
  2. [2][2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018. - · arXiv
  3. [3][3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in Conference on Computer Vision and Pattern Recognition , 2009. -
  4. [4][4] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, 'Gradient surgery for multi-task learning,' arXiv preprint arXiv:2001.06782 , 2020. - · arXiv
  5. [5][5] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, 'Mt-opt: Continuous multi-task robotic reinforcement learning at scale,' arXiv preprint arXiv:2104.08212 , 2021. - · arXiv
  6. [6][6] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, 'Robonet: Large-scale multi-robot learning,' arXiv preprint arXiv:1910.11215 , 2019. - · arXiv
  7. [7][7] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, 'One-shot visual imitation learning via meta-learning,' in Conference on Robot Learning . PMLR, 2017, pp. 357-368. -
  8. [8][8] Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, 'One-shot imitation learning,' arXiv preprint arXiv:1703.07326 , 2017. - · arXiv
  9. [9][9] J. Ho and S. Ermon, 'Generative adversarial imitation learning,' arXiv preprint arXiv:1606.03476 , 2016. - · arXiv
  10. [10][10] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine, 'One-shot imitation from observing humans via domain-adaptive meta-learning,' arXiv preprint arXiv:1802.01557 , 2018. - · arXiv
  11. [11][11] Y. Liu, A. Gupta, P. Abbeel, and S. Levine, 'Imitation from observation: Learning to imitate behaviors from raw video via context translation,' in International Conference on Robotics and Automation (ICRA) , 2018. -
  12. [12][12] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, 'Time-contrastive networks: Self-supervised learning from video,' in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 1134-1141. -
  13. [13][13] A. Ghadirzadeh, X. Chen, W. Yin, Z. Yi, M. Bjorkman, and D. Kragic, 'Human-centered collaborative robots with deep reinforcement learning,' IEEE Robotics and Automation Letters , 2020. -
  14. [14][14] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine, 'Model-based visual planning with self-supervised functional distances,' arXiv preprint arXiv:2012.15373 , 2020. - · arXiv
  15. [15][15] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, 'Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,' in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 5628-5635. -
  16. [16][16] P. Sharma, L. Mohan, L. Pinto, and A. Gupta, 'Multiple interactions made easy (mime): Large scale demonstrations data for imitation,' in Conference on robot learning . PMLR, 2018, pp. 906-915. -
  17. [17][17] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. FeiFei, 'Roboturk: A crowdsourcing platform for robotic skill learning through imitation,' in Conference on Robot Learning , 2018. -
  18. [18][18] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei, 'Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity,' arXiv:1911.04052 , 2019. - · arXiv
  19. [19][19] L. Pinto and A. Gupta, 'Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,' in international conference on robotics and automation (ICRA) . IEEE, 2016. -
  20. [20][20] C. Finn and S. Levine, 'Deep visual foresight for planning robot motion,' in 2017 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2017, pp. 2786-2793. -
  21. [21][21] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, 'Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,' The International Journal of Robotics Research , vol. 37, no. 4-5, pp. 421-436, 2018. -
  22. [22][22] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. , 'Scalable deep reinforcement learning for vision-based robotic manipulation,' in Conference on Robot Learning . PMLR, 2018, pp. 651-673. -
  23. [23][23] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine, 'Visual foresight: Model-based deep reinforcement learning for vision-based robotic control,' arXiv preprint arXiv:1812.00568 , 2018. - · arXiv
  24. [24][24] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, 'Tossingbot: Learning to throw arbitrary objects with residual physics,' IEEE Transactions on Robotics , vol. 36, no. 4, pp. 1307-1319, 2020. -
  25. [25][25] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, 'Visual imitation made easy,' arXiv e-prints , pp. arXiv-2008, 2020. -
  26. [26][26] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Conference on Computer Vision and Pattern Recognition , 2016. -
  27. [27][27] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel, 'Deep spatial autoencoders for visuomotor learning,' in International Conference on Robotics and Automation (ICRA) , 2016. -
  28. [28][28] S. Levine, C. Finn, T. Darrell, and P. Abbeel, 'End-to-end training of deep visuomotor policies,' The Journal of Machine Learning Research , vol. 17, no. 1, pp. 1334-1373, 2016.

Cited by

Sorted by citations

Authors
Ebert, F., Yang, Y., Schmeckpeper, K., Bucher, B., Georgakis, G., Daniilidis, K., Finn, C., & Levine, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2109.13396
arXiv
2109.13396
OpenAlex
W3204665606
Cited by
94
References
0
Citations by year
2026: 15 · 2025: 33 · 2024: 27 · 2023: 13 · 2022: 5 · 2021: 1
FWCI
10.78
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • Robotics: Science and Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
robot learning, domain generalization, cross-task generalization, multi-domain dataset, data reuse, robot demonstration, open-source dataset, domain gap

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review