CoChat
Sign Up
PreprintCited by 180Robot Manipulation and LearningAI-based Problem Solving and PlanningRobotic Path Planning Algorithms

RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

Open access
Mandi, Zhao, Jain, Shreeya, Song, Shuran·2023·arXiv
DOI 10.48550/arxiv.2307.04738arXiv 2307.04738
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning. Robots are equipped with LLMs to discuss and collectively reason task strategies. They then generate sub-task plans and task space waypoint paths, which are used by a multi-arm motion planner to accelerate trajectory planning. We also provide feedback from the environment, such as collision checking, and prompt the LLM agents to improve their plan and waypoints in-context. For evaluation, we introduce RoCoBench, a 6-task benchmark covering a wide range of multi-robot collaboration scenarios, accompanied by a text-only dataset for agent representation and reasoning. We experimentally demonstrate the effectiveness of our approach -- it achieves high success rates across all tasks in RoCoBench and adapts to variations in task semantics. Our dialog setup offers high interpretability and flexibility -- in real world experiments, we show RoCo easily incorporates human-in-the-loop, where a user can communicate and collaborate with a robot agent to complete tasks together. See project website https://project-roco.github.io for videos and code.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.04738

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RocoBench is a benchmark with six tasks evaluating multi-agent collaboration across diverse scenarios, emphasizing communication and coordination strategies to assess adaptability and generalization in cooperative robotics.

    “RocoBench [89] is a benchmark with six tasks evaluating multi-agent collaboration across diverse scenarios, emphasizing communication and coordination strategies to assess adaptability and generalization in cooperative robotics.”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [89]

  • Mandi et al. (2023) introduces RoCo, a multirobot collaboration approach using LLMs for high-level communication and low-level path planning, with experimental results demonstrating adaptability and success in collaborative tasks.

    “[Mandi et al., 2023] introduces RoCo, a novel approach for multirobot collaboration that uses LLMs for high-level communication and low-level path planning.Each robotic arm is equipped with an LLM, cooperating with inverse kinematics and collision checking.Experimental results demonstrate the adaptability and success of RoCo in collaborative tasks.”

    From Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024) · cites this paper as Mandi et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. · arXiv
  2. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. R. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, 2022.
  3. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policis: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753, 2022. · arXiv
  4. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022. · arXiv
  5. H. Wang, G. Gonzalez-Pumariega, Y. Sharma, and S. Choudhury. Demo2code: From sum- marizing demonstrations to synthesizing code via extended chain-of-thought. arXiv preprint arXiv:2305.16744, 2023. · arXiv
  6. J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658, 2023. · arXiv
  7. S. Huang, Z. Jiang, H. Dong, Y. Qiao, P. Gao, and H. Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176, 2023. · arXiv
  8. K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153, 2023. · arXiv
  9. Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan. Autotamp: Autoregressive task and motion planning with llms as translators and checkers. arXiv preprint arXiv:2306.06531, 2023. · arXiv
  10. Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247, 2023. · arXiv
  11. X. Zhang, Y. Zhu, Y. Ding, Y. Zhu, P. Stone, and S. Zhang. Visually grounded task and motion planning for mobile manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 1925-1931. IEEE, 2022.
  12. Y. Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. " no, to the right"- online language corrections for robotic manipulation via shared autonomy. arXiv preprint arXiv:2301.02555, 2023. · arXiv
  13. M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023. · arXiv
  14. W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647, 2023. · arXiv
  15. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. · arXiv
  16. C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.06407, 2022. · arXiv
  17. T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736, 2022. · arXiv
  18. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
  19. X. Zhao, M. Li, C. Weber, M. B. Hafez, and S. Wermter. Chat with the environment: Interactive multimodal perception using large language models. arXiv preprint arXiv:2303.08268, 2023. · arXiv
  20. M. Kwon, H. Hu, V. Myers, S. Karamcheti, A. Dragan, and D. Sadigh. Toward grounded social reasoning. arXiv preprint arXiv:2306.08651, 2023. · arXiv
  21. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022. · arXiv
  22. J. Andreas. Language models as agent models. arXiv preprint arXiv:2212.01681, 2022. · arXiv
  23. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society. ArXiv, abs/2303.17760, 2023. · arXiv
  24. V. Nair, E. Schumacher, G. Tso, and A. Kannan. Dera: Enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:2303.17071, 2023. · arXiv
  25. T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, Z. Tu, and S. Shi. En- couraging divergent thinking in large language models through multi-agent debate. ArXiv, abs/2305.19118, 2023. · arXiv
  26. Y. Koga and J.-C. Latombe. On multi-arm manipulation planning. Proceedings of the 1994 IEEE International Conference on Robotics and Automation, pages 945-952 vol.2, 1994.
  27. A. Dobson and K. E. Bekris. Planning representations and algorithms for prehensile multi-arm manipulation. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6381-6386. IEEE, 2015.
  28. H. Ha, J. Xu, and S. Song. Learning a decentralized multi-arm motion planner. In Conference on Robotic Learning (CoRL), 2020.
  29. J. Yakey, S. LaValle, and L. Kavraki. Randomized path planning for linkages with closed kinematic chains. Robotics and Automation, IEEE Transactions on, 17:951 -958, 01 2002. doi:10.1109/70.976030. · DOI
  30. Z. Xian, P. Lertkultanon, and Q.-C. Pham. Closed-chain manipulation of large objects by multi-arm robotic systems. IEEE Robotics and Automation Letters, 2(4):1832-1839, 2017.
  31. M. Dogar, A. Spielberg, S. Baker, and D. Rus. Multi-robot grasp planning for sequential assembly operations. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 193-200, 2015. doi:10.1109/ICRA.2015.7138999. · DOI
  32. E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033, 2012. doi:10.1109/IROS.2012.6386109. · DOI · Paper page
  33. S. Dasari, A. Gupta, and V. Kumar. Learning dexterous manipulation from exemplar object trajectories and pre-grasps. In IEEE International Conference on Robotics and Automation 2023, 2023.

Cited by

Sorted by citations

Authors
Mandi, Z., Jain, S., & Song, S.
Affiliations
Columbia University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
286-299
ISSN
1050-4729 (ISSN-L)
DOI
10.48550/arxiv.2307.04738
arXiv
2307.04738
OpenAlex
W4401416363
Cited by
180
References
0
Citations by year
2026: 63 · 2025: 83 · 2024: 33 · 2023: 1
FWCI
112.25
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2143601, 2037101, 2132519)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotic Path Planning Algorithms
Keywords
multi-robot collaboration, large language models, robot dialogue, task planning, path planning, trajectory planning, collision detection, in-context learning, benchmark, human-robot interaction, LLM agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review