A Sim-to-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain
Abstract
Robots that autonomously navigate real-world 3D cluttered environments need to safely traverse terrain with abrupt changes in surface normals and elevations. In this letter, we present the development of a novel sim-to-real pipeline for a mobile robot to effectively learn how to navigate real-world 3D rough terrain environments. The pipeline uses a deep reinforcement learning architecture to learn a navigation policy from training data obtained from the simulated environment and a unique combination of strategies to directly address the reality gap for such environments. Experiments in the real-world 3D cluttered environment verified that the robot successfully performed point-to-point navigation from arbitrary start and goal locations while traversing rough terrain. A comparison study between our DRL method, classical, and deep learning-based approaches showed that our method performed better in terms of success rate, and cumulative travel distance and time in a 3D rough terrain environment.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A comparison with some state of the art solutions for robot navigation highlights the better performance of the proposed approach in terms of success rate as well as cumulative travel distance, and time required for task execution.
“Moreover, a comparison with some state of the art solutions for robot navigation highlights the better performance of the proposed approach in terms of success rate as well as cumulative travel distance, and time required for task execution.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [41]
The resulting controller in [41], learned entirely in simulation, results in an effective application on the real-world environment.
“The resulting , learned entirely in simulation, results in an effective application on the real-world environment.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [41]
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
MeshLab: an Open-Source Mesh Processing Tool
Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., & Ranzuglia, G. (2008) · ISTI Open Portal · doi:10.2312/localchapterevents/italchap/italianchapconf2008/129-136 · Cited by 2,445 · see citing works
Introduction to Machine Learning
Yu, F. R., & He, Y. (2019) · Springer briefs in electrical and computer engineering · doi:10.1007/978-3-030-10546-4_1 · Cited by 1,619 · see citing works
Deep Recurrent Q-Learning for Partially Observable MDPsPreprint
Hausknecht, M., & Stone, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06527 · Cited by 1,224 · see citing works
Labbé, M., & Michaud, F. (2018) · J. Field Robotics · doi:10.1002/rob.21831 · Cited by 1,009 · see citing works
Tai, L., Paolo, G., & Liu, M. (2017) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros.2017.8202134 · Cited by 843 · see citing works
An evaluation of the RGB-D SLAM system
Endres, F., Hess, J., Engelhard, N., Sturm, J., Cremers, D., & Burgard, W. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225199 · Cited by 743 · see citing works
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735 · see citing works
Introduction to Machine Learning
Ayodele, T. O. (2010) · InTech eBooks · doi:10.5772/9394 · Cited by 655 · see citing works
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618 · see citing works
Robust adversarial reinforcement learning
Pinto, L., Davidson, J., Sukthankar, R., & Gupta, A. (2017) · arXiv · doi:10.48550/arxiv.1703.02702 · Cited by 507 · cited 1× · see citing works
“Sim-to-real has been used in a number of robotic applications including aerial vehicles [18], manipulators [19], and mobile robots [20]–[22].”
BackgroundNiroui, F., Zhang, K., Kashino, Z., & Nejat, G. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2891991 · Cited by 389 · see citing works
Self-citationProbabilistic Terrain Mapping for Mobile Robots With Uncertain Localization
Fankhauser, P., Bloesch, M., & Hutter, M. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2849506 · Cited by 313 · see citing works
Terrain traversability analysis methods for unmanned ground vehicles: A survey
Papadakis, P. (2013) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2013.01.006 · Cited by 302 · see citing works
Robotic Urban Search and Rescue: A Survey from the Control Perspective
Liu, Y., & Nejat, G. (2013) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-013-9822-x · Cited by 273 · see citing works
Self-citationDiscontinuous Automatic Control
Flugge-Lotz, I. (1953) · Princeton University Press eBooks · doi:10.1515/9781400877218 · Cited by 190 · see citing works
Krüsi, P., Furgale, P., Bosse, M. C., & Siegwart, R. (2016) · Journal of Field Robotics · doi:10.1002/rob.21700 · Cited by 178 · see citing works
Long-Range Indoor Navigation With PRM-RL
Francis, A. G., Jr., Faust, A., Chiang, H.-T. L., Hsu, J., Kew, J. C., Fišer, M., & Lee, T.-W. E. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.2975428 · Cited by 172 · see citing works
Learning from Demonstration for Autonomous Navigation in Complex Unstructured Terrain
Silver, D., Bagnell, J. A., & Stentz, A. (2010) · The International Journal of Robotics Research · doi:10.1177/0278364910369715 · Cited by 142 · see citing works
Deep Reinforcement Learning for Safe Local Planning of a Ground Vehicle in Unknown Rough Terrain
Josef, S., & Degani, A. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3011912 · Cited by 136 · see citing works
Multirobot Cooperative Learning for Semiautonomous Control in Urban Search and Rescue Applications
Liu, Y.-G., & Nejat, G. (2016) · J. Field Robotics · doi:10.1002/rob.21597 · Cited by 116 · see citing works
Self-citationLearning Ground Traversability From Simulations
Chavéz-García, R. O., Guzzi, J., Gambardella, L. M., & Giusti, A. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2801794 · Cited by 107 · see citing works
Doroodgar, B., Liu, Y., & Nejat, G. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2314294 · Cited by 102 · see citing works
Self-citationRobot Navigation of Environments with Unknown Rough Terrain Using deep Reinforcement Learning
Zhang, K., Niroui, F., Ficocelli, M., & Nejat, G. (2018) · IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) · doi:10.1109/ssrr.2018.8468643 · Cited by 86 · see citing works
Self-citation3D path planning and execution for search and rescue ground robots
Colas, F., Mahesh, S., Pomerleau, F., Liu, M., & Siegwart, R. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696431 · Cited by 85 · see citing works
Zhou, S., Xi, J., McDaniel, M. W., et al. (2012) · Journal of Field Robotics · doi:10.1002/rob.21417 · Cited by 81 · see citing works
Geometric and visual terrain classification for autonomous mobile navigation
Schilling, F., Chen, X., Folkesson, J., et al. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206092 · Cited by 75 · see citing works
Virtual-to-Real: Learning to Control in Visual Semantic Segmentation
Hong, Z.-W., Chen, Y., Yang, H.-K., Su, S.-Y., Shann, T.-Y., Chang, Y.-H., Ho, B. H.-L., Tu, C.-C., Hsiao, T.-C., Hsiao, H.-W., et al. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/682 · Cited by 73 · see citing works
Leaving Flatland: Efficient real‐time three‐dimensional perception and motion planning
Rusu, R. B., Sundaresan, A., Morisset, B., et al. (2009) · Journal of Field Robotics · doi:10.1002/rob.20313 · Cited by 54 · see citing works
Autonomous Navigation in Complex Environments with Deep Multimodal Fusion Network
Nguyen, A., Nguyen, N. D., Tran, K., et al. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341494 · Cited by 50 · see citing works
Using robots in hazardous environments
Baudoin, Y., & Habib, M. K. (2011) · Woodhead Publishing Limited eBooks · doi:10.1533/9780857090201 · Cited by 31 · see citing works
Bayesian Generalized Kernel Inference for Terrain Traversability Mapping
Shan, T., Wang, J., Englot, B., et al. (2018) · Conference on Robot Learning · Cited by 31 · see citing works
Global UGV Path Planning on Point Cloud Maps Created by UAV
Fedorenko, R., Gabdullin, A., & Fedorenko, A. (2018) · IEEE International Conference on Intelligent Transportation Engineering (ICITE) · doi:10.1109/icite.2018.8492584 · Cited by 26 · see citing works
Stereo Vision Based SLAM Issues and Solutions
Herath, D. C., Kodagoda, S., & Dissanayake, G. (2007) · doi:10.5772/5006 · Cited by 15 · see citing works
Kinetic model of a skid steered robot
Šolc, F., & Sembera, J. (2008) · International Conference on Signal Processing · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“lation [54] PPO Trotting and galloping of quadruped [56] A3C Marble maze game with robotic arm [57] PPO Rubik’s cube with robotic hand [46] QT-Opt Rvision-based control task of a robotic arm for grasping [61] PPO Climb and descend stairs with bipedal robot [41] A3C Wheeled mobile platform navigation”
BackgroundMethodInfluential
- Authors
- Hu, H., Zhang, K., Tan, A. H., Ruan, M., Agia, C. G., & Nejat, G.
- Affiliations
- University of Toronto (CA)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2021-10
- Volume
- 6
- Issue
- 4
- Pages
- 6569-6576
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W3176880374
- Cited by
- 93
- References
- 35
- Citations by year
- 2026: 11 · 2025: 20 · 2024: 24 · 2023: 17 · 2022: 18 · 2021: 3
- FWCI
- 5.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Canada Research Chairs
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotic Path Planning Algorithms
- Keywords
- sim-to-real transfer, deep reinforcement learning, robot navigation, uneven terrain, cluttered environments, mobile robots, reality gap, 3D terrain, point-to-point navigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
