Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios
The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
Developing a safe and efficient collision-avoidance policy for multiple robots is challenging in the decentralized scenarios where each robot generates its paths with limited observation of other robots’ states and intentions. Prior distributed multi-robot collision-avoidance systems often require frequent inter-robot communication or agent-level features to plan a local collision-free action, which is not robust and computationally prohibitive. In addition, the performance of these methods is not comparable with their centralized counterparts in practice. In this article, we present a decentralized sensor-level collision-avoidance policy for multi-robot systems, which shows promising results in practical applications. In particular, our policy directly maps raw sensor measurements to an agent’s steering commands in terms of the movement velocity. As a first step toward reducing the performance gap between decentralized and centralized methods, we present a multi-scenario multi-stage training framework to learn an optimal policy. The policy is trained over a large number of robots in rich, complex environments simultaneously using a policy-gradient-based reinforcement-learning algorithm. The learning algorithm is also integrated into a hybrid control framework to further improve the policy’s robustness and effectiveness. We validate the learned sensor-level collision-3avoidance policy in a variety of simulated and real-world scenarios with thorough performance evaluations for large-scale multi-robot systems. The generalization of the learned policy is verified in a set of unseen scenarios including the navigation of a group of heterogeneous robots and a large-scale scenario with 100 robots. Although the policy is trained using simulation data only, we have successfully deployed it on physical robots with shapes and dynamics characteristics that are different from the simulated agents, in order to demonstrate the controller’s robustness against the simulation-to-real modeling error. Finally, we show that the collision-avoidance policy learned from multi-robot navigation tasks provides an excellent solution for safe and effective autonomous navigation for a single robot working in a dense real human crowd. Our learned policy enables a robot to make effective progress in a crowd without getting stuck. More importantly, the policy has been successfully deployed on different types of physical robot platforms without tedious parameter tuning. Videos are available at https://sites.google.com/view/hybridmrca .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Deep learning methods have been successfully used in a wide variety of applications, including computer vision (He et al., 2016; Krizhevsky et al., 2012) and nature language processing (Amodei et al., 2016; Graves et al., 2013).”
MethodDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Social force model for pedestrian dynamics
Helbing, D., & Molnár, P. (1995) · Physical Review E · doi:10.1103/physreve.51.4282 · Cited by 6,845
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Run policy p u for T i time-steps, collecting f o ti , r ti , a ti g , where t 2 ½ 0 , T i 6: Estimate advantages using GAE (Schulman et al., 2015b) (Bengio et al., 2009), we propose a two-stage training process, which accelerates the policy to converge to a satisfactory solution, and gets higher…”
The dynamic window approach to collision avoidance
Fox, D., Burgard, W., & Thrun, S. (1997) · IEEE Robotics & Automation Magazine · doi:10.1109/100.580977 · Cited by 3,771
Social LSTM: Human Trajectory Prediction in Crowded Spaces
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.110 · Cited by 3,483
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 2×
“Tobin et al. (2017) introduced domain randomization to train a robust and transferable policy for manipulation tasks.”
BackgroundInfluentialBehavior-based formation control for multirobot teams
Balch, T., & Arkin, R. C. (1998) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.736776 · Cited by 3,112
Proceedings of 1995 IEEE International Conference on Robotics and Automation
(1995) · Proceedings of 1995 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1995.525251 · Cited by 2,830
Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., & Alahi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00240 · Cited by 2,466
Reciprocal n-Body Collision Avoidance
van den Berg, J., Guy, S. J., Lin, M., & Manocha, D. (2011) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-642-19457-3_1 · Cited by 1,858
Reciprocal Velocity Obstacles for real-time multi-agent navigation
van den Berg, J., Lin, M., & Manocha, D. (2008) · 2008 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2008.4543489 · Cited by 1,722
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 1×
“Although DRL algorithms have been successfully applied to mobile robot motion planning, they have mainly focused on a discrete action space (Zhang et al., 2016a; Zhu et al., 2017) or on small-scale problems (Chen et al., 2017a,b; Kahn et al., 2017; Tai et al., 2017).”
MethodConflict-based search for optimal multi-agent pathfinding
Sharon, G., Stern, R., Felner, A., & Sturtevant, N. R. (2015) · Artificial Intelligence · doi:10.1016/j.artint.2014.11.006 · Cited by 1,242
Proceedings of the 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems
(2001) · Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems Expanding the Societal Role of Robotics in the the Next Millennium (Cat No 01CH37180) IROS-01 · doi:10.1109/iros.2001.977143 · Cited by 1,197
Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Gini, M. L., Shehory, O. M., Itō, T., & Jonker, C. M. (2013) · Cited by 703
Emergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657 · cited 9×
“This parallel PPO algorithm can be easily scaled to a large-scale multi-robot system with hundreds of robots in a decentralized fashion since each robot in the team serves as an independent worker collecting data.”
BackgroundMethodInfluentialDecentralized non-communicating multiagent collision avoidance with deep reinforcement learning
Chen, Y. F., Liu, M., Everett, M., & How, J. P. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989037 · Cited by 645
Cooperative manipulation and transportation with aerial robots
Michael, N., Fink, J., & Kumar, V. (2011) · Autonomous Robots · doi:10.1007/s10514-010-9205-0 · Cited by 637
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618
Cooperative Manipulation and Transportation with Aerial Robots
Michael, N., Fink, J., & Kumar, V. (2010) · Robotics · doi:10.7551/mitpress/8727.003.0002 · Cited by 467
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Zhang, T., Kahn, G., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487175 · Cited by 452 · cited 3×
“DNNs have also been integrated with model predictive control to control robotic systems (Lenz et al., 2015; Zhang et al., 2016b).”
MethodThe Hybrid Reciprocal Velocity Obstacle
Snape, J., Berg, J. V. D., Guy, S. J., & Manocha, D. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2120810 · Cited by 448
Stone, P., & Veloso, M. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00025-9 · Cited by 436
A depth space approach to human-robot collision avoidance
Flacco, F., Kroger, T., De Luca, A., & Khatib, O. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6225245 · Cited by 428
Pfeiffer, M., Schaeuble, M., Nieto, J., Siegwart, R., & Cadena, C. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989182 · Cited by 413
Learning monocular reactive UAV control in cluttered natural environments
Ross, S., Melik-Barkhudarov, N., Shankar, K. S., Wendel, A., Dey, D., Bagnell, J. A., & Hebert, M. (2013) · 2013 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2013.6630809 · Cited by 403
Optimal Multirobot Path Planning on Graphs: Complete Algorithms and Effective Heuristics
Yu, J., & LaValle, S. M. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2593448 · Cited by 331
Alonso-Mora, J., Baker, S., & Rus, D. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917719333 · Cited by 307
Fast, On-line Collision Avoidance for Dynamic Vehicles Using Buffered Voronoi Cells
Zhou, D., Wang, Z., Bandyopadhyay, S., & Schwager, M. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2656241 · Cited by 286
Optimal Reciprocal Collision Avoidance for Multiple Non-Holonomic Robots
Alonso-Mora, J., Breitenmoser, A., Rufli, M., Beardsley, P., & Siegwart, R. (2013) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-642-32723-0_15 · Cited by 280
Chen, J., Sun, D., Yang, J., & Chen, H. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364909104290 · Cited by 258
Schwartz, J. T., & Sharir, M. (1983) · The International Journal of Robotics Research · doi:10.1177/027836498300200304 · Cited by 248
Sun, D., Wang, C., Shang, W., & Feng, G. (2009) · IEEE Transactions on Robotics · doi:10.1109/tro.2009.2027384 · Cited by 228
Pedestrian Behavior Understanding and Prediction with Deep Neural Networks
Yi, S., Li, H., & Wang, X. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_16 · Cited by 199
C apt: Concurrent assignment and planning of trajectories for multiple robots
Turpin, M., Michael, N., & Kumar, V. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913515307 · Cited by 193
Object Detection and Tracking for Autonomous Navigation in Dynamic Environments
Ess, A., Schindler, K., Leibe, B., & Van Gool, L. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364910365417 · Cited by 191
Automatic transportation of biological cells with a robot-tweezer manipulation system
Hu, S., & Sun, D. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911413479 · Cited by 188
Generalized reciprocal collision avoidance
Bareiss, D., & van den Berg, J. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364915576234 · Cited by 176
Cooperative Collision Avoidance for Nonholonomic Robots
Alonso-Mora, J., Beardsley, P., & Siegwart, R. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2793890 · Cited by 173
Deep-Learned Collision Avoidance Policy for Distributed Multiagent Navigation
Long, P., Liu, W., & Pan, J. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2651371 · Cited by 159
Self-citationApplications of hybrid reachability analysis to robotic aerial vehicles
Gillula, J., Hoffmann, G. M., Huang, H., Vitus, M. P., & Tomlin, C. J. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364910387173 · Cited by 137
BRVO: Predicting pedestrian trajectories using velocity-space reasoning
Kim, S., Guy, S. J., Liu, W., Wilkie, D., Lau, R. W. H., Lin, M. C., & Manocha, D. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364914555543 · Cited by 118
Self-citationA hybrid control approach to action coordination for mobile robots
Egerstedt, M., & Xiaoming Hu, A. (2002) · Automatica · doi:10.1016/s0005-1098(01)00185-6 · Cited by 111
Reactive navigation in outdoor environments using potential fields
Haddad, H., Khatib, M., Lacroix, S., & Chatila, R. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1998.677268 · Cited by 108
A study on decentralized receding horizon control for decoupled systems
Keviczky, T., Borrelli, F., & Balas, G. J. (2004) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2004.1384629 · Cited by 107
Tang, S., Thomas, J. R., & Kumar, V. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917741532 · Cited by 68
Switching Rules for Decentralized Control with Simple Control Laws
Shucker, B., Murphey, T. D., & Bennett, J. K. (2007) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2007.4282736 · Cited by 25
Cited by
Deep reinforcement learning based mobile robot navigation: A review
Zhu, K., & Zhang, T. (2021) · Tsinghua Science and Technology · doi:10.26599/tst.2021.9010012 · Cited by 499
Bio-Inspired Collision Avoidance in Swarm Systems via Deep Reinforcement Learning
Na, S., Niu, H., Lennox, B., & Arvin, F. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3145346 · Cited by 64
Metadata
- Authors
- Fan, T., Long, P., Liu, W., & Pan, J.
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2020-05-31
- Volume
- 39
- Issue
- 7
- Pages
- 856-892
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W3031798691
- Cited by
- 332
- References
- 53
- FWCI
- 15.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.