The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
A Survey of Deep RL and IL for Autonomous Driving Policy Learning
Open accessAbstract
Autonomous driving (AD) agents generate driving policies based on online perception results, which are obtained at multiple levels of abstraction, e.g., behavior planning, motion planning and control. Driving policies are crucial to the realization of safe, efficient and harmonious driving behaviors, where AD agents still face substantial challenges in complex scenarios. Due to their successful application in fields such as robotics and video games, the use of deep reinforcement learning (DRL) and deep imitation learning (DIL) techniques to derive AD policies have witnessed vast research efforts in recent years. This paper is a comprehensive survey of this body of work, which is conducted at three levels: First, a taxonomy of the literature studies is constructed from the system perspective, among which five modes of integration of DRL/DIL models into an AD architecture are identified. Second, the formulations of DRL/DIL models for conducting specified AD tasks are comprehensively reviewed, where various designs on the model state and action spaces and the reinforcement learning rewards are covered. Finally, an in-depth review is conducted on how the critical issues of AD applications regarding driving safety, interaction with other traffic participants and uncertainty of the environment are addressed by the DRL/DIL models. To the best of our knowledge, this is the first survey to focus on AD policy learning using DRL/DIL, which is addressed simultaneously from the system, task-driven and problem-driven perspectives. We share and discuss findings, which may lead to the investigation of various topics in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 2×
“Below, we focus on policy learning for AD applications, while readers are suggested to refer [27], [29], [31] for comprehensive discussions on the general challenges of DRL/DIL.”
BackgroundA Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · cited 3×
“Simultaneous lateral and longitudinal control has received wide attention, especially in urban scenarios [63], [96]–[99].”
BackgroundSimple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · cited 1×
“TD3 [54] builds on Double Q-learning and takes the minimum value between a pair of critics to limit overestimation.”
BackgroundDeep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
A Survey on Deep Transfer Learning
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., & Liu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01424-7_27 · Cited by 3,165
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Stanley: The robot that won the DARPA Grand Challenge
Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J., Fong, P., Gale, J., Halpenny, M., Hoffmann, G., et al. (2006) · Journal of Field Robotics · doi:10.1002/rob.20147 · Cited by 2,237 · cited 2×
“Digital Object Identifier 10.1109/TITS.2021.3134702 Defense Advanced Research Projects Agency (DARPA) Grand Challenges [8].”
BackgroundInfluentialSoft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · cited 2×
“Soft actor critic (SAC) [53] adds an entropy term to the reward function to encourage better exploration.”
BackgroundGenerative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · cited 1×
“Intrinsic curiosity module (ICM) [205] transforms high dimensional observations into the visual feature space, which is learned with a self-supervised inverse dynamics model.”
Background(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 1×
“replay [41], which maintains a memory that stores transitions (st , at , st+1, rt+1), increases the sample efficiency.”
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2977374 · Cited by 1,306
Imitation Learning: A Survey of Learning Methods
Hussein, A., Gaber, M. M., Elyan, E., & Jayne, C. (2018) · ACM Computing Surveys · doi:10.1145/3054912 · Cited by 1,060
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., & Shah, A. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793742 · Cited by 694 · cited 1×
“, highdimensional state spaces), which have increased in popularity for complex locomotion [20], robotics [21] and autonomous driving [22]–[24] tasks.”
BackgroundDistributional Reinforcement Learning With Quantile Regression
Dabney, W., Rowland, M., Bellemare, M., & Munos, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11791 · Cited by 612
Learning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · cited 2×
“Advantage actor critic (A2C) [52], which”
Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
Shalev-Shwartz, S., Shammah, S., & Shashua, A. (2016) · arXiv · doi:10.48550/arxiv.1610.03295 · Cited by 509 · cited 1×
“[160] decompose the double-merge problem into learnable DRL policies and non-learnable hard constraints, which offers better interpretability than a pure endto-end framework.”
An algorithmic perspective on imitation learning
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., & Peters, J. (2018) · Foundations and Trends in Robotics · doi:10.1561/2300000053 · Cited by 405 · cited 3×
“Below, we focus on policy learning for AD applications, while readers are suggested to refer [27], [29], [31] for comprehensive discussions on the general challenges of DRL/DIL.”
BackgroundDeep Bayesian Active Learning with Image Data
Ghahramani, Z., Gal, Y., & Islam, R. (2017) · PMLR · doi:10.17863/cam.11070 · Cited by 392
Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning
Isele, D., Rahimi, R., Cosgun, A., Subramanian, K., & Fujimura, K. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461233 · Cited by 379
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Decoupling Representation Learning from Reinforcement Learning
Stooke, A., Lee, K., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2009.08319 · Cited by 123
VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control
Zhang, J., Tai, L., Yun, P., Xiong, Y., Liu, M., Boedecker, J., & Burgard, W. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2894216 · Cited by 109
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · cited 2×
“References [4] addresses the deep learning techniques for AD with a focus on perception and control, while [32] addresses control only.”
BackgroundDiversity-Driven Exploration Strategy for Deep Reinforcement Learning
Hong, Z.-W., Shann, T.-Y., Su, S.-Y., Chang, Y.-H., & Lee, C.-Y. (2018) · arXiv · doi:10.48550/arxiv.1802.04564 · Cited by 77
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Cited by
End-to-End Autonomous Driving: Challenges and Frontiers
Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., & Li, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3435937 · Cited by 610
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey
Chib, P. S., & Singh, P. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3318070 · Cited by 301
Huang, Z., Sheng, Z., Ma, C., & Chen, S. (2024) · Communications in Transportation Research · doi:10.1016/j.commtr.2024.100127 · Cited by 88
Metadata
- Authors
- Zhu, Z., & Zhao, H.
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2022-09
- Volume
- 23
- Issue
- 9
- Pages
- 14043-14065
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W3121045039
- Cited by
- 237
- References
- 42
- FWCI
- 10.42
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.