Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Artificial Intelligence (AI) shows promising applications for the perception and planning tasks in autonomous driving (AD) due to its superior performance compared to conventional methods. However, highly complex AI systems exacerbate the existing challenge of safety assurance of AD. One way to mitigate this challenge is to utilize explainable AI (XAI) techniques. To this end, we present the first comprehensive systematic literature review of explainable methods for safe and trustworthy AD. We begin by analyzing the requirements for AI in the context of AD, focusing on three key aspects: data, model, and agency. We find that XAI is fundamental to meeting these requirements. Based on this, we explain the sources of explanations in AI and describe a taxonomy of XAI. We then identify five key contributions of XAI for safe and trustworthy AI in AD, which are interpretable design, interpretable surrogate models, interpretable monitoring, auxiliary explanations, and interpretable validation. Finally, we propose a conceptual modular framework called SafeX to integrate the reviewed methods, enabling explanation delivery to users while simultaneously ensuring the safety of AI models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“While this efficiency versus transparency trade-off is some-times acceptable (arguably even non-existent [105]), highly complex safety-relevant systems such as AD cannot fully rely on black box systems, as they are not readily certifiable for safety.”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718
Probabilistic Principal Component Analysis
Tipping, M. E., & Bishop, C. M. (1999) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00196 · Cited by 3,679
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., & Darrell, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00271 · Cited by 2,641
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
Robust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · cited 1×
“In addition, adversarial perturbations can fool deep learning-based neural networks [117], leading to implausible results.”
Background1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
Hinton, G. E., Krizhevsky, A., & Wang, S. D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-21735-7_6 · Cited by 1,360 · cited 1×
“Martinez et al. [23] developed an interpretable latent space in the DNN by using capsule networks [142] to predict eye fixations in AD scenarios and contextual conditions.”
MethodTechniques for Interpretable Machine Learning
Du, M., Liu, N., & Hu, X. (2018) · arXiv · doi:10.48550/arxiv.1808.00033 · Cited by 1,261
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 1×
“To better understand stake-holder requirements and to adapt explanations to the varied socio-technical interactions of the real world, it is crucial to develop methods that are rooted in actual problems and not merely motivated by a vague sense of need for safety and trustworthiness.”
Autonomous vehicle perception: The technology of today and tomorrow
Van Brummelen, J., O’Brien, M., Gruyer, D., & Najjaran, H. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2018.02.012 · Cited by 869
Explainable artificial intelligence: an analytical review
Angelov, P. P., Soares, E. A., Jiang, R., Arnold, N. I., & Atkinson, P. M. (2021) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1424 · Cited by 843
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709 · cited 1×
“This is in addition to the countless ethical, social, and legal reasons why neural methods may also be suspect [106].”
BackgroundPerception, Planning, Control, and Coordination for Autonomous Vehicles
Pendleton, S., Andersen, H., Du, X., Shen, X., Meghjani, M., Eng, Y., Rus, D., & Ang, M. (2017) · Machines · doi:10.3390/machines5010006 · Cited by 657
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103473 · Cited by 652 · cited 2×
“XAI aims to provide human-understandable insights into the behaviour of the AI and the development of XAI meth-ods could be beneficial for different kinds of stakeholders [12].”
BackgroundStepin, I., Alonso, J. M., Catala, A., & Pereira-Farina, M. (2021) · IEEE Access · doi:10.1109/access.2021.3051315 · Cited by 376 · cited 1×
“To comprehensively explore the published methods, we have decided to perform a systematic literature review following the recommendations of Kitchenham and Charters [137] and the review methodology section of Stepin et al. [138].”
MethodExplanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355
Shapley values for feature selection: the good, the bad, and the axioms
Fryer, D., Strumke, I., & Nguyen, H. (2021) · IEEE Access · doi:10.1109/access.2021.3119110 · Cited by 347 · cited 1×
“Although this framework provides useful guidance, its application to AVs is not addressed and users’ queries are not considered.”
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
One Thousand and One Hours: Self-driving Motion Prediction Dataset
Houston, J., Zuidhof, G., Bergamini, L., Ye, Y., Chen, L., Jain, A., Omari, S., Iglovikov, V., & Ondruska, P. (2020) · arXiv · doi:10.48550/arxiv.2006.14480 · Cited by 292
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · cited 2×
“These models provide a way to shed some light on black box decision making but importance must not be conflated with actual explanations as they can often be altered without affecting the output prediction [126], [127].”
BackgroundExplainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227
Explainability of deep vision-based autonomous driving systems: Review and challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2021) · arXiv · doi:10.48550/arxiv.2101.05307 · Cited by 227
Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations
Sadat, A., Casas, S., Ren, M., Wu, X., Dhawan, P., & Urtasun, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58592-1_25 · Cited by 183
Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2019) · Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3308532.3329441 · Cited by 136 · cited 1×
“Second, XAI could help users calibrate their trust in automated systems in line with the actual capabilities of AVs [14], thereby preventing misuse.”
BackgroundExplainable Object-Induced Action Decision for Autonomous Vehicles
Xu, Y., Yang, X., Gong, L., Lin, H.-C., Wu, T.-Y., Li, Y., & Vasconcelos, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00954 · Cited by 129 · cited 1×
“Xu et al. [94] introduced the dataset BDD-OIA (object-induced actions) which extracted complicated scenarios from BDD-X and annotated them with new explanations focusing on objects which influence the decision.”
Requirements-driven test generation for autonomous vehicles with machine learning components
Tuncali, C. E., Fainekos, G., Prokhorov, D., Ito, H., & Kapinski, J. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2955903 · Cited by 129
Interpretable Goal Recognition in the Presence of Occluded Factors for Autonomous Vehicles
Hanna, J. P., Rahman, A., Fosong, E., Eiras, F., Dobre, M., Redford, J., Ramamoorthy, S., & Albrecht, S. V. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635903 · Cited by 21 · cited 1×
“Hanna et al. [32] extends this work by considering occluded factors in the environment.”
BackgroundSelf-citationInterpretable Goal-based Prediction and Planning for Autonomous Driving
Albrecht, S. V., Brewitt, C., Wilhelm, J., Gyevnar, B., Eiras, F., Dobre, M., & Ramamoorthy, S. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560849 · Cited by 12 · cited 1×
“2) Interpretable By Design – Planning & Prediction: Albrecht et al. [27] proposed an ante-hoc, interpretable, and intelligible prediction and planning system called IGP2 based on rational inverse planning and Monte Carlo Tree Search over high-level macro actions.”
BackgroundSelf-citation
Cited by
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 323
Mathew, D. E., Ebem, D. U., Ikegwu, A. C., Ukeoma, P. E., & Dibiaezue, N. F. (2025) · Neural Processing Letters · doi:10.1007/s11063-025-11732-2 · Cited by 168
McGrath, M. J., Duenser, A., Lacey, J., & Paris, C. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100200 · Cited by 51
Kabir, S., Hossain, M. S., & Andersson, K. (2025) · Algorithms · doi:10.3390/a18090556 · Cited by 44
“Such highlighted regions facilitate design-time debugging and runtime safety monitoring of self-driving cars [126].”
Metadata
- Authors
- Kuznietsov, A., Gyevnar, B., Wang, C., Peters, S., & Albrecht, S. V.
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2024-12
- Volume
- 25
- Issue
- 12
- Pages
- 19342-19364
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W4403390366
- Cited by
- 144
- References
- 38
- FWCI
- 6.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.