CoChat
Sign Up
PreprintCited by 28Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Open access
Zhan, Albert, Zhao, Ruihan, Pinto, Lerrel, Abbeel, Pieter, Laskin, Michael·2020·arXiv
DOI 10.48550/arxiv.2012.07975arXiv 2012.07975
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances in unsupervised representation learning significantly improved the sample efficiency of training Reinforcement Learning policies in simulated environments. However, similar gains have not yet been seen for real-robot reinforcement learning. In this work, we focus on enabling data-efficient real-robot learning from pixels. We present Contrastive Pre-training and Data Augmentation for Efficient Robotic Learning (CoDER), a method that utilizes data augmentation and unsupervised learning to achieve sample-efficient training of real-robot arm policies from sparse rewards. While contrastive pre-training, data augmentation, demonstrations, and reinforcement learning are alone insufficient for efficient learning, our main contribution is showing that the combination of these disparate techniques results in a simple yet data-efficient method. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels, such as reaching, picking, moving, pulling a large object, flipping a switch, and opening a drawer in just 30 minutes of mean real-world training time. We include videos and code on the project website: https://sites.google.com/view/efficient-robotic-manipulation/home

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.07975

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Offline data augmentation can be effective for generating larger dataset for robot manipulation.

    “Offline data augmentation can be effective for generating larger dataset for robot manipulation [7, 35-45]”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [39]

    The “Minimal Collection of Demonstrations” section says the authors use random crops of stored demonstrations during encoder pre-training, supporting offline augmentation of robot-manipulation data. It does not say they generate a larger dataset; it describes augmentation during subsequent RL training as occurring while the robot interacts with its environment. Our copy provides no page numbers for these passages. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Offline data augmentation can be complementary to MimicGen.

    “In fact, this can be complementary to MimicGen- we leverage pixel shift randomization [7, 36-39] when training image-based agents on MimicGen data.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [39]

    The Introduction (page unavailable) describes training a pixel-based robotic agent with augmented images from both demonstrations and online experience. The “Minimal Collection of Demonstrations” section specifies random cropping, not pixel-shift randomization. The paper supports using image augmentation alongside demonstration data, but does not establish that the cited method is complementary to MimicGen specifically. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Several works have used offline data augmentation to increase the dataset size for learning policies.

    “Several works have used offline data augmentation to increase the dataset size for learning policies [7, 35-45].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [39]

    The Introduction and Data Augmentation section support that this paper uses data augmentation to learn policies. But the Introduction calls the policy-training augmentations online, and the Minimal Collection of Demonstrations section says augmentation is applied during training; neither describes offline augmentation to increase a dataset's size. No page numbers are available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 54

  1. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, p. 529, 2015.
  2. C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse et al., "Dota 2 with large scale deep reinforcement learning," arXiv preprint arXiv:1912.06680, 2019. · arXiv
  3. A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell, "Agent57: Outperforming the atari human benchmark," in International Conference on Machine Learning, 2020.
  4. J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel et al., "Mastering atari, go, chess and shogi by planning with a learned model," arXiv preprint arXiv:1911.08265, 2019. · arXiv
  5. M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srini- vas, "Reinforcement learning with augmented data," arXiv preprint arXiv:2004.14990, 2020. · arXiv
  6. D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, "Dream to control: Learning behaviors by latent imagination," in International Conference on Learning Representations, 2020.
  7. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-End Training of Deep Visuomotor Policies," CoRR, vol. abs/1504.00702, 2015. [Online]. Available: http://arxiv.org/abs/1504.00702 · arXiv
  8. D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke et al., "Qt- opt: Scalable deep reinforcement learning for vision-based robotic manipulation," arXiv preprint arXiv:1806.10293, 2018. · arXiv
  9. J. Mahler, F. T. Pokorny, B. Hou, M. Roderick, M. Laskey, M. Aubry, K. Kohlhoff, T. Kröger, J. Kuffner, and K. Goldberg, "Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards," in ICRA, 2016.
  10. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. Mc- Grew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray et al., "Learning dexterous in-hand manipulation," arXiv preprint arXiv:1808.00177, 2018. · arXiv
  11. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforce- ment learning," arXiv preprint arXiv:1509.02971, 2015. · arXiv
  12. Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq et al., "Deepmind control suite," arXiv preprint arXiv:1801.00690, 2018. · arXiv
  13. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welin- der, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, "Hindsight experience replay," in NeurIPS, 2017.
  14. T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, "Deep imitation learning for complex manipulation tasks from vir- tual reality teleoperation," 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8, 2018.
  15. S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, "Visual Imitation Made Easy," CoRR, vol. abs/2008.04899, 2020. [Online]. Available: https://arxiv.org/abs/2008.04899 · arXiv
  16. P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, "Implicit behavioral cloning," in Conference on Robot Learning. PMLR, 2022, pp. 158- 168.
  17. A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani et al., "Transporter networks: Rearranging the visual world for robotic manipulation," arXiv preprint arXiv:2010.14406, 2020. · arXiv
  18. P. Florence, L. Manuelli, and R. Tedrake, "Self-supervised correspon- dence in visuomotor policy learning," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 492-499, 2019.
  19. I. Kostrikov, D. Yarats, and R. Fergus, "Image augmentation is all you need: Regularizing deep reinforcement learning from pixels," arXiv preprint arXiv:2004.13649, 2020. · arXiv
  20. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, "The arcade learning environment: An evaluation platform for general agents," Journal of Artificial Intelligence Research, vol. 47, pp. 253-279, 2013.
  21. T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel et al., "Soft actor-critic algorithms and applications," arXiv preprint arXiv:1812.05905, 2018. · arXiv
  22. R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant mapping," in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), vol. 2. IEEE, 2006, pp. 1735-1742.
  23. A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," arXiv preprint arXiv:1807.03748, 2018. · arXiv
  24. Z. Wu, Y. Xiong, S. Yu, and D. Lin, "Unsupervised feature learn- ing via non-parametric instance-level discrimination," arXiv preprint arXiv:1805.01978, 2018. · arXiv
  25. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum con- trast for unsupervised visual representation learning," arXiv preprint arXiv:1911.05722, 2019. · arXiv
  26. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple frame- work for contrastive learning of visual representations," in Interna- tional Conference on Machine Learning, 2020.
  27. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum con- trast for unsupervised visual representation learning," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  28. O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord, "Data-efficient image recognition with contrastive predictive coding," arXiv preprint arXiv:1905.09272, 2019. · arXiv
  29. F. Schroff, D. Kalenichenko, and J. Philbin, "Facenet: A unified embedding for face recognition and clustering," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815-823.
  30. X. Wang and A. Gupta, "Unsupervised learning of visual representa- tions using videos," in ICCV, 2015.
  31. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, "Openai gym," arXiv preprint arXiv:1606.01540, 2016. · arXiv
  32. R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine, "Vision- Based Multi-Task Manipulation for Inexpensive Robots Using End- To-End Learning from Demonstration," CoRR, vol. abs/1707.02920, 2017. [Online]. Available: http://arxiv.org/abs/1707.02920 · arXiv
  33. P. R. Florence, L. Manuelli, and R. Tedrake, "Self-Supervised Correspondence in Visuomotor Policy Learning," IEEE Robotics Autom. Lett., vol. 5, no. 2, pp. 492-499, 2020. [Online]. Available: https://doi.org/10.1109/LRA.2019.2956365 · DOI
  34. T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, "Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning," in Conference on Robot Learning (CoRL), 2019. [Online]. Available: https://arxiv.org/abs/1910.10897 · arXiv
  35. S. Ross, G. J. Gordon, and D. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning," in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, ser. JMLR Proceedings, G. J. Gordon, D. B. Dunson, and M. Dudík, Eds., vol. 15. JMLR.org, 2011, pp. 627-635. [Online]. Available: http://proceedings.mlr.press/v15/ross11a/ross11a.pdf
  36. D. Pomerleau, "ALVINN: an autonomous land vehicle in a neural network," in Advances in Neural Information Processing Systems 1, [NIPS Conference, Denver, Colorado, USA, 1988], D. S. Touretzky, Ed. Morgan Kaufmann, 1988, pp. 305-313. [Online]. Available: http://papers.nips.cc/paper/ 95-alvinn-an-autonomous-land-vehicle-in-a-neural-network
  37. M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, "End to End Learning for Self-Driving Cars," CoRR, vol. abs/1604.07316, 2016. [Online]. Available: http://arxiv.org/abs/1604.07316 · arXiv
  38. A. Giusti, J. Guzzi, D. C. Ciresan, F. He, J. P. Rodriguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. D. Caro, D. Scaramuzza, and L. M. Gambardella, "A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots," IEEE Robotics Autom. Lett., vol. 1, no. 2, pp. 661-667, 2016. [Online]. Available: https://doi.org/10.1109/LRA.2015.2509024 · DOI · Paper page
  39. J. Nakanishi, J. Morimoto, G. Endo, G. Cheng, S. Schaal, and M. Kawato, "Learning from demonstration and adaptation of biped locomotion," Robotics Auton. Syst., vol. 47, no. 2-3, pp. 79-91, 2004. [Online]. Available: https://doi.org/10.1016/j.robot.2004.03.003 · DOI
  40. M. Kalakrishnan, J. Buchli, P. Pastor, and S. Schaal, "Learning locomotion over rough terrain using terrain templates," in 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 11-15, 2009, St. Louis, MO, USA. IEEE, 2009, pp. 167-172. [Online]. Available: https://doi.org/10.1109/IROS.2009.5354701 · DOI
  41. S. Ross, G. Gordan, and A. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning," in arXiv preprint arXiv:1011.0686, 2010. · arXiv
  42. A. Y. Ng and S. J. Russell, "Algorithms for inverse reinforcement learning," in Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 -July 2, 2000, P. Langley, Ed. Morgan Kaufmann, 2000, pp. 663-670.
  43. P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforcement learning," in Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004), Banff, Alberta, Canada, July 4-8, 2004, ser. ACM International Conference Proceeding Series, C. E. Brodley, Ed., vol. 69. ACM, 2004. [Online]. Available: https://doi.org/10.1145/1015330.1015430 · DOI
  44. P. Sharma, L. Mohan, L. Pinto, and A. Gupta, "Multiple interactions made easy (mime): Large scale demonstrations data for imitation," arXiv preprint arXiv:1810.07121, 2018. · arXiv
  45. C. Finn and S. Levine, "Deep visual foresight for planning robot motion," in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 2786-2793.
  46. H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar, "Dexter- ous manipulation with deep reinforcement learning: Efficient, general, and low-cost," in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 3651-3657.
  47. S. James, K. Wada, T. Laidlow, and A. J. Davison, "Coarse-to- fine q-attention: Efficient learning for visual robotic manipulation via discretisation," arXiv preprint arXiv:2106.12534, 2021. · arXiv
  48. A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel, "Overcoming exploration in reinforcement learning with demonstrations," in 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018. IEEE, 2018, pp. 6292-6299. [Online]. Available: https://doi.org/10.1109/ICRA.2018.8463162 · DOI · Paper page
  49. A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations," arXiv preprint arXiv:1709.10087, 2017. · arXiv
  50. M. Vecerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. A. Riedmiller, "Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards," CoRR, vol. abs/1707.08817, 2017. [Online]. Available: http://arxiv.org/abs/1707.08817 · arXiv
  51. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
  52. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, "Autoaugment: Learning augmentation strategies from data," in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019, pp. 113-123.
  53. P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine, "Time-contrastive networks: Self-supervised learning from video," in 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018. IEEE, 2018, pp. 1134-1141. [Online]. Available: https: //doi.org/10.1109/ICRA.2018.8462891 · DOI
  54. S. Pirk, M. Khansari, Y. Bai, C. Lynch, and P. Sermanet, "Online Object Representations with Contrastive Learning," CoRR, vol. abs/1906.04312, 2019. [Online]. Available: http://arxiv.org/abs/1906. 04312 · arXiv

Cited by

Sorted by citations

Authors
Zhan, A., Zhao, R., Pinto, L., Abbeel, P., & Laskin, M.
Affiliations
  • University of California, Berkeley (US)
  • New York University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
4040-4047
ISSN
2153-0858 (ISSN-L)
DOI
10.48550/arxiv.2012.07975
arXiv
2012.07975
OpenAlex
W4312288659
Cited by
28
References
2
Citations by year
2026: 5 · 2025: 5 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 9
FWCI
1.94
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Amazon Web Services
  • Open Philanthropy Project
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
contrastive pre-training, data augmentation, unsupervised feature learning, sample efficiency, sparse rewards, robotic arm manipulation, demonstration-based learning, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review