Explainable AI and Reinforcement Learning—A Systematic Review of Current Approaches and Trends
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Research into Explainable Artificial Intelligence (XAI) has been increasing in recent years as a response to the need for increased transparency and trust in AI. This is particularly important as AI is used in sensitive domains with societal, ethical, and safety implications. Work in XAI has primarily focused on Machine Learning (ML) for classification, decision, or action, with detailed systematic reviews already undertaken. This review looks to explore current approaches and limitations for XAI in the area of Reinforcement Learning (RL). From 520 search results, 25 studies (including 5 snowball sampled) are reviewed, highlighting visualization, query-based explanations, policy summarization, human-in-the-loop collaboration, and verification as trends in this area. Limitations in the studies are presented, particularly a lack of user studies, and the prevalence of toy-examples and difficulties providing understandable explanations. Areas for future study are identified, including immersive visualization, and symbolic representation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 1×
“Popularly, RL has been used successfully by the DeepMind team to produce agents capable of better than human-level performance in complex games like GO (Silver et al., 2016), and a suite of Atari games (Mnih et al., 2015).”
MethodSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 1×
“To examine the current state of the literature, a systematic literature review using a methodology adapted from Kitchenham et al. (2009) was performed.”
MethodGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294 · cited 1×
“A considerable amount of work has already gone into the concept of “saliency maps” which highlight areas of the input image that were of importance to the outcome, see Adebayo et al. (2018).”
Background(2019) · Cited by 467
Emergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424 · cited 1×
“A recent example of this was a game of hide-and-seek where agents learned strategies to exploit the physics system of the game to overcome what was intended by the developers to be walls that could not be passed (Baker et al., 2019).”
BackgroundSimplicity and probability in causal explanation
LOMBROZO, T. (2007) · Cognitive Psychology · doi:10.1016/j.cogpsych.2006.09.006 · Cited by 384 · cited 1×
“…is very little research in the XAI space on leverages existing work on how people “generate, select, present, and evaluate” (p. 4) explanations, such as the work by Lombrozo (2007) which describes how people prefer simpler and more general explanations over specific and more likely explanations.”
Improving Robot Controller Transparency Through Autonomous Policy Explanation
Hayes, B., & Shah, J. A. (2017) · Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020233 · Cited by 230
Miller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · cited 4×
“This lack of user testing across the reviewed papers is consistent with the findings in the Miller et al. (2017) review of XAI in Machine Learning.”
BackgroundResultInfluentialToward establishing trust in adaptive agents
Glass, A., McGuinness, D. L., & Wolverton, M. (2008) · Proceedings of the 13th international conference on Intelligent user interfaces · doi:10.1145/1378773.1378804 · Cited by 211 · cited 1×
“(Glass et al., 2008).”
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210
Challenges and opportunities: from big data to knowledge in AI 2.0
Zhuang, Y.-T., Wu, F., Chen, C., & Pan, Y.-H. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601883 · Cited by 210 · cited 1×
“A quote by Zhuang et al. (2017) sums up the current situation well: “ [...] people and computers can both play chess, it is far from clear whether they do it the same way.”
Background(2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01388-2 · Cited by 201
Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning
Fulton, N., & Platzer, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12107 · Cited by 188 · cited 1×
“As Fulton and Platzer (2018) point out, formal verification allows us to detect discrepancies between models and the physical system being controlled, which could lead to accidents.”
BackgroundIATK: An Immersive Analytics Toolkit
Cordeil, M., Cunningham, A., Bach, B., Hurter, C., Thomas, B. H., Marriott, K., & Dwyer, T. (2019) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2019.8797978 · Cited by 178 · cited 1×
“…by the addition of visualization techniques better designed for handling the temporal dimension of RL (such as the Immersive Analytics Toolkit by Cordeil et al. (2019) or TensorBoard graphs 4 ), as well as multi-modal, immersive forms of visualization such as virtual or augmented reality to…”
BackgroundDQNViz: A Visual Analytics Approach to Understand Deep Q-Networks
Wang, J., Gou, L., Shen, H., & Yang, H. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864504 · Cited by 167 · cited 5×
“Wang et al. (2018) provided a comprehensive yet highly application-specific visualization tool for Deep-Q Reinforcement Learning Networks called DQNViz , with the goal of identifying and extracting typical action/movement/reward patterns of agents.”
MethodInfluentialData Protection: A Practical Guide to UK and EU Law
Carey, P. (2004) · Cited by 160
Transparency and Explanation in Deep Reinforcement Learning Neural Networks
Iyer, R., Li, Y., Li, H., Lewis, M., Sundar, R., & Sycara, K. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278776 · Cited by 145 · cited 3×
“A more complex approach to visualizing the focus of a RL agent was presented by Iyer et al. (2018).”
BackgroundMethodEnabling robots to communicate their objectives
Huang, S. H., Held, D., Abbeel, P., & Dragan, A. D. (2019) · Autonomous Robots · doi:10.1007/s10514-018-9771-0 · Cited by 101 · cited 2×
“…with one paper using 3 experts (Wang et al., 2018), others with students (Iyer et al., 2018), n = 40; and Greydanus et al. (2018), n = 31, and three recruiting using Amazon Mechanical Turk 3 (Huang et al., 2019, n = 191; Madumal et al., 2020, n = 120; and Ehsan et al., 2019, n = 65 and n = 60).”
BackgroundMethodVerifying Deep-RL-Driven Systems
Kazak, Y., Barrett, C., Katz, G., & Schapira, M. (2019) · ACM SIGCOMM Workshop on Network Meets AI & ML (NetAI) · doi:10.1145/3341216.3342218 · Cited by 93
Explanation-Based Reward Coaching to Improve Human Performance via Reinforcement Learning
Tabrez, A., Agrawal, S., & Hayes, B. (2019) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673104 · Cited by 90 · cited 3×
“…(17 papers), or when they did, limited details of the testing were published, failing to describe where the participants were recruited from, how many were recruited, or if the participants were knowledgeable in Machine Learning (Pynadath et al., 2018; Tabrez and Hayes, 2019; Tabrez et al., 2019).”
BackgroundAmir, O., Doshi‐Velez, F., & Sarne, D. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09418-w · Cited by 63 · cited 2×
“Amiret al. (2019) proposed a conceptual framework for strategy summarization.”
BackgroundExplainable Reinforcement Learning through a Causal Lens
Madumal, P., Miller, T., Sonenberg, L., & Vetere, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5631 · Cited by 58
Dethise, A., Canini, M., & Kandula, S. (2019) · ACM SIGCOMM Workshop on Network Meets AI & ML (NetAI) · doi:10.1145/3341216.3342210 · Cited by 40 · cited 1×
“Finally, only four papers provided the reader with a link to the open-source repository of their code (Greydanus et al., 2018; Yang et al., 2018; Dethise et al., 2019; Sridharan and Meadows, 2019).”
BackgroundTowards a Theory of Explanations for Human–Robot Collaboration
Sridharan, M., & Meadows, B. (2019) · KI - Künstliche Intelligenz · doi:10.1007/s13218-019-00616-y · Cited by 39
Learn to Interpret Atari AgentsPreprint
Yang, Z., Bai, S., Zhang, L., & Torr, P. H. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11276 · Cited by 33 · cited 3×
“Finally, only four papers provided the reader with a link to the open-source repository of their code (Greydanus et al., 2018; Yang et al., 2018; Dethise et al., 2019; Sridharan and Meadows, 2019).”
BackgroundMethodTransparency Communication for Machine Learning in Human-Automation Interaction
Pynadath, D. V., Barnes, M. J., Wang, N., & Chen, J. Y. C. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-90403-0_5 · Cited by 26 · cited 15×
“In addition to providing tools to assist with trust and accountability, XAI assists with debugging and bias in Machine Learning.”
BackgroundInfluentialHow to Compete in the Age of Artificial Intelligence
Mohanty, S., & Vyas, S. (2018) · Apress eBooks · doi:10.1007/978-1-4842-3808-0 · Cited by 21
Semantic Predictive Control for Explainable and Efficient Policy Learning
Pan, X., Chen, X., Cai, Q.-Z., Canny, J., & Yu, F. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794437 · Cited by 18 · cited 3×
“” Looking at autonomous vehicles as an example, Pan et al. (2019), contributed Semantic Predictive Control (SPC) which learns to “predict the visual semantics of future states and possible events based upon visual inputs and an inferred sequence of future actions” (p.”
BackgroundInfluentialDeep Reinforcement Learning Monitor for Snapshot Recording
Dao, G., Mishra, I., & Lee, M. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00095 · Cited by 18 · cited 2×
“An extension to SBRL by Mishra et al. (2018) was V-SBRL which was applied to vision-based RL agents.”
MethodToward Robust Policy Summarization
Lage, I., Lifschitz, D., Doshi‐Velez, F., & Amir, O. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/fwkh4493 · Cited by 12
Mishra, I., Dao, G., & Lee, M. (2018) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2018.8628887 · Cited by 9
Stamper, J. C., & Moore, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23207-8_50 · Cited by 7 · cited 4×
“As Stamper and Moore (2019) point out, analysis of agents playing the Atari 2600 game Space Invaders exhibited similar decision-making behaviors to expert human players (e.g., keeping the invaders in a square formation, and destroying right-most enemies first to slow down the rate of advancement), however in other gam…”
BackgroundInfluentialSafe and Trustworthy Human-Robot Interaction
Araiza-Illan, D., & Eder, K. (2018) · Springer eBooks · doi:10.1007/978-94-007-6046-2_131 · Cited by 6
StarCraft II: A New Challenge for Reinforcement Learning
Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., et al. (2017) · arXiv · doi:10.48550/arxiv.1708.04782 · cited 1×
“…challenge is the fact that RL agents are generally trained without using training data (with the exception of where human-replay data is used, such as in Vinyals et al., 2017), and instead learning is facilitated by a feedback loop (observations) from performing actions within an environment.”
Cited by
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Weber, P., Carl, K. V., & Hinz, O. (2024) · Management Review Quarterly · doi:10.1007/s11301-023-00320-0 · Cited by 292
“Our research offers broad guidance for scientists to understand the growing emphasis on XAI in Finance research, which we also found in other domains (Wells and Bednarz 2021; Islam et al. 2022).”
Haque, A. B., Islam, A. K. M. N., & Mikalef, P. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 240
“Research on explainable AI has increased and has primarily focused on policy summarization, human collaboration, visualization, verification, etc. (Wells and Bednarz, 2021); however, research gaps exist in customized algorithms, user testing, and scalability (Wells and Bednarz, 2021).”
BackgroundResultInfluentialSeoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107441 · Cited by 239
Explainable Reinforcement Learning: A Survey and Comparative Review
Milani, S., Topin, N., Veloso, M., & Fang, F. (2024) · ACM Computing Surveys · doi:10.1145/3616864 · Cited by 224
“Second, existing surveys on XRL focus on delving deeper into a more limited set of papers [71, 121, 158] or focus on speciic applications [168].”
Background
Metadata
- Authors
- Wells, L., & Bednarz, T.
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2021-05-20
- Volume
- 4
- Pages
- 550030-550030
- ISSN
- 2624-8212 (ISSN-L)
- OpenAlex
- W3164011142
- Cited by
- 185
- References
- 36
- FWCI
- 3.01
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.