CoChat
Sign Up
PreprintCited by 49Robot Manipulation and LearningReinforcement Learning in RoboticsTeleoperation and Haptic Systems

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Open access
Tung, Albert, Wong, Josiah, Mandlekar, Ajay, Martín-Martín, Roberto, Zhu, Yuke, Fei-Fei, Li, Savarese, Silvio·2020·arXiv
DOI 10.48550/arxiv.2012.06738arXiv 2012.06738
Open this paper in CoChat — free accountOpen PDF

Abstract

Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -- asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks. Using MART, we collected demonstrations for five novel two and three-arm tasks from several geographically separated users. From our data we arrived at a critical insight: most multi-arm tasks do not require global coordination throughout its full duration, but only during specific moments. We show that learning from such data consequently presents challenges for centralized agents that directly attempt to model all robot actions simultaneously, and perform a comprehensive study of different policy architectures with varying levels of centralization on our tasks. Finally, we propose and evaluate a base-residual policy framework that allows trained policies to better adapt to the mixed coordination setting common in multi-arm manipulation, and show that a centralized policy augmented with a decentralized residual model outperforms all other models on our set of benchmark tasks. Additional results and videos at https://roboturk.stanford.edu/multiarm .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.06738

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RoboTurk is a system for crowdsourcing task demonstrations from human operators using smartphone-based teleoperation and video streams provided in web browsers.

    “RoboTurk [2,23-26] is a system for crowdsourcing task demonstrations from human operators using smartphone-based teleoperation and video streams provided in web browsers.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [25]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 39

  1. P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse rein- forcement learning", in Proceedings of the twenty-first international conference on Machine learning, 2004, p. 1.
  2. --, "Inverse reinforcement learning", in Encyclopedia of machine learning, Springer, 2011, pp. 554-558.
  3. F. Amadio, A. Colomé, and C. Torras, "Exploiting symmetries in reinforcement learning of bimanual robotic tasks", IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 1838-1845, 2019.
  4. K. Bogert and P. Doshi, "Multi-robot inverse reinforcement learning under occlusion with interactions", in Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, Citeseer, 2014, pp. 173-180.
  5. L. Busoniu, R. Babuska, and B. De Schutter, "A comprehensive survey of multiagent reinforcement learning", IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 38, no. 2, pp. 156-172, 2008.
  6. S. Chernova and M. Veloso, "Multiagent collaborative task learning through imitation", in Proceedings of the fourth International Symposium on Imitation in Animals and Artifacts, 2007, pp. 74-79.
  7. R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta, "Efficient bimanual manipulation using learned task schemas", in 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 1149-1155.
  8. --, "Intrinsic Motivation for Encouraging Synergistic Behavior", arXiv preprint arXiv:2002.05189, 2020. · arXiv
  9. A. Das, S. Kottur, J. M. Moura, S. Lee, and D. Batra, "Learning cooperative visual dialog agents with deep reinforcement learning", in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2951-2960.
  10. A. Edsinger and C. C. Kemp, "Two arms are better than one: A behavior based control system for assistive bimanual manipulation", in Recent progress in robotics: Viable robotic service to human, Springer, 2007, pp. 345-355.
  11. J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, "Learning to communicate with deep multi-agent reinforcement learning", in Advances in neural information processing systems, 2016, pp. 2137- 2145.
  12. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. White- son, "Counterfactual multi-agent policy gradients", arXiv preprint arXiv:1705.08926, 2017. · arXiv
  13. E. Gribovskaya and A. Billard, "Combining dynamical systems control and programming by demonstration for teaching discrete bimanual coordination tasks to a humanoid robot", in 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI), IEEE, 2008, pp. 33-40.
  14. J. Jiang and Z. Lu, "Learning attentional communication for multi- agent cooperation", in Advances in neural information processing systems, 2018, pp. 7254-7264.
  15. O. Khatib, "A unified approach for motion and force control of robot manipulators: The operational space formulation", IEEE Journal on Robotics and Automation, vol. 3, no. 1, pp. 43-53, 1987.
  16. D. P. Kingma and M. Welling, "Auto-encoding variational bayes", arXiv preprint arXiv:1312.6114, 2013. · arXiv
  17. O. Kroemer, C. Daniel, G. Neumann, H. Van Hoof, and J. Peters, "Towards learning hierarchical skills for multi-phase manipulation tasks", in 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2015, pp. 1503-1510.
  18. M. Laghi, M. Maimeri, M. Marchand, C. Leparoux, M. Catalano, A. Ajoudani, and A. Bicchi, "Shared-autonomy control for intuitive bimanual tele-manipulation", in 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), IEEE, 2018, pp. 1-9.
  19. H. M. Le, Y. Yue, P. Carr, and P. Lucey, "Coordinated multi-agent imitation learning", arXiv preprint arXiv:1703.03121, 2017. · arXiv
  20. J. I. Lipton, A. J. Fay, and D. Rus, "Baxter's homunculus: Virtual reality spaces for teleoperation in manufacturing", IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 179-186, 2017.
  21. R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, "Multi-agent actor-critic for mixed cooperative-competitive environments", in Advances in neural information processing systems, 2017, pp. 6379-6390.
  22. A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei, "Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity", arXiv preprint arXiv:1911.04052, 2019. · arXiv
  23. A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox, "IRIS: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data", in IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 4414-4420.
  24. A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei, "Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations", arXiv preprint arXiv:2003.06085, 2020. · arXiv
  25. A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei, "RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation", in Conference on Robot Learning, 2018.
  26. M. J. Matarić, "Reinforcement learning in the multi-robot domain", in Robot colonies, Springer, 1997, pp. 73-83.
  27. Ö. Örün and Y. Akbulut, "Effect of multitasking, physical envi- ronment and electroencephalography use on cognitive load and retention", Computers in Human Behavior, vol. 92, pp. 216-229, 2019.
  28. P. Peng, Y. Wen, Y. Yang, Q. Yuan, Z. Tang, H. Long, and J. Wang, "Multiagent bidirectionally-coordinated nets: Emergence of human- level coordination in learning to play starcraft combat games", arXiv preprint arXiv:1703.10069, 2017. · arXiv
  29. D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network", in Advances in neural information processing systems, 1989, pp. 305-313.
  30. J. Silvério, L. Rozo, S. Calinon, and D. G. Caldwell, "Learning biman- ual end-effector poses from demonstrations using task-parameterized dynamical systems", in 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, 2015, pp. 464-470.
  31. C. Smith, Y. Karayiannidis, L. Nalpantidis, X. Gratal, P. Qi, D. V. Dimarogonas, and D. Kragic, "Dual arm manipulation-A survey", Robotics and Autonomous systems, vol. 60, no. 10, pp. 1340-1353, 2012.
  32. J. Song, H. Ren, D. Sadigh, and S. Ermon, "Multi-agent generative adversarial imitation learning", in Advances in neural information processing systems, 2018, pp. 7461-7472.
  33. A. Šošić, W. R. KhudaBukhsh, A. M. Zoubir, and H. Koeppl, "Inverse reinforcement learning in swarm systems", arXiv preprint arXiv:1602.05450, 2016. · arXiv
  34. S. Sukhbaatar, R. Fergus, et al., "Learning multiagent communication with backpropagation", in Advances in neural information processing systems, 2016, pp. 2244-2252.
  35. M. Tan, "Multi-agent reinforcement learning: Independent vs. coop- erative agents", in Proceedings of the tenth international conference on machine learning, 1993, pp. 330-337.
  36. E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control", in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, pp. 5026-5033.
  37. T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, "Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation", arXiv preprint arXiv:1710.04615, 2017. · arXiv
  38. Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martín, "robosuite: A Modular Simulation Framework and Benchmark for Robot Learning", in arXiv preprint arXiv:2009.12293, 2020. · arXiv
  39. R. Zollner, T. Asfour, and R. Dillmann, "Programming by demonstra- tion: dual-arm manipulation tasks for humanoid robots", in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, vol. 1, 2004, pp. 479-484.

Cited by

Sorted by citations

Authors
Tung, A., Wong, J., Mandlekar, A., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S.
Affiliations
  • Stanford University (US)
  • The University of Texas at Austin (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
9212-9219
ISSN
1050-4729 (ISSN-L)
DOI
10.48550/arxiv.2012.06738
arXiv
2012.06738
OpenAlex
W3112445903
Cited by
49
References
0
Citations by year
2026: 2 · 2025: 15 · 2024: 6 · 2023: 13 · 2022: 7 · 2021: 6
FWCI
11.92
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • U.S. Department of Defense
  • Toyota Research Institute
  • National Defense Science and Engineering Graduate
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
Keywords
imitation learning, policy architecture

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review