CoChat

From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

Open access

The DOI matches its Crossref record; 204 of the 208 references it cites were found in a registry or an index, and 4 are not checked yet; 17 of the 124 sources its claims cite were checked for support.

Abstract

The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practices of more than 300 papers published in the past 7 years at major AI and ML conferences that introduce an XAI method. We find that one in three papers evaluate exclusively with anecdotal evidence, and one in five papers evaluate with users. This survey also contributes to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. Our systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark, and compare new and existing XAI methods. The Co-12 categorization scheme and our identified evaluation methods open up opportunities to include quantitative metrics as optimization criteria during model training to optimize for accuracy and interpretability simultaneously.

Verified 2026-10-05 · 4 references not checked yet

Overview

CoChat Summary

TL;DR

This systematic review analyzes how explainable AI methods are evaluated, proposing 12 conceptual explanation-quality properties (Co-12) and cataloging quantitative evaluation methods. Across 606 screened papers from 2014–2020 at 12 major conferences, 312 introduced an XAI method, and the authors find that 33% evaluate only with anecdotal evidence, 58% use quantitative evaluation, and 22% use user studies.

Key findings

  • Of the 312 papers introducing an XAI method, 33% evaluated exclusively with anecdotal evidence, 58% applied quantitative evaluation, and 22% evaluated with human subjects in a user study; of those user studies, 23% used domain experts (application-grounded).
  • The Co-12 properties are proposed as a multi-faceted categorization scheme; the most frequently evaluated properties were Coherence and Output-completeness, followed by Correctness, Compactness, and Covariate complexity.
  • Most papers that quantitatively evaluated an XAI method evaluated only one or two Co-12 properties; 73 papers evaluated exactly one property, often Coherence.
  • Quantitative evaluation methods exist for each Co-12 property, but evaluation practice remains concentrated on Coherence, Completeness, Compactness, and Correctness; the authors also report that quantitative evaluation slightly increased over time while anecdotal-only evaluation decreased, with user studies remaining around 20%.

Methods

  • Systematic literature review: DBLP title keyword search (`explain* OR explanat* OR interpret*`) across 12 conferences (AAAI, IJCAI, NeurIPS, ICML, ICLR, CVPR, ICCV, ACL, WWW, ICDM, SIGKDD, SIGIR) for 2014–2020, conducted 4 May 2021; 606 papers found, 494 after excluding companion papers, 361 included after inclusion criteria, and 312 after filtering to papers introducing an XAI method.
  • Each included paper was categorized along six dimensions (data type, explanation type, explanation problem, predictive model type, task type, method type) and reviewed for evaluation practices using the Co-12 properties; user studies were classified as application-grounded or human-grounded following Doshi-Velez and Kim.
  • The review focuses on functionally grounded, quantitative evaluation methods without user studies, with user-study evaluation methods summarized in Supplementary Material; quantitative methods were clustered and named according to the Co-12 properties.

Limitations

  • The paper states that it focuses on functionally grounded evaluation and provides only a concise overview of user-study evaluation methods in Supplementary Material.
  • The paper states that it may be unreasonable to expect an XAI method to score well on all Co-12 properties, and that trade-offs are necessary: Coherence may contradict Correctness, and Completeness and Compactness may be diametrical opposites; application domain and practical feasibility determine which properties should be emphasized.
  • The text makes evident that the corpus is limited to 12 computer science conferences from 2014–2020 and uses title keyword search, so coverage may miss other venues, terms, or later work; the authors also note that their venue selection may have influenced the higher quantitative-evaluation statistics, and Figure 5 notes that a manually added category may be high precision but potentially low recall.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3583558

Cites

204 of 208 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

10 Valid · 3 Not valid · 63 Not checked · 97 Own results and interpretations

74 more claims

Create a free account to see all 173 claims and to verify the 107 references still open.

See all claims — free account
  • Probably not a substantial claim. 70
  • It may state two separate claims as one. 47

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 208 references

Research works printed but not linked: 102

  1. Tameem Adel, Zoubin Ghahramani, and Adrian Weller. 2018. Discovering interpretable representations for both deep generative and discriminative models. In Proceedings of the ICML, Vol. 80. PMLR.
  2. David Alvarez-Melis and Tommi S. Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2018/hash/3e9f0fc9b2f89e043bc6233994dfcf76-Abstract.html.
  3. Marco Ancona, Cengiz Öztireli, and Markus H. Gross. 2019. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In Proceedings of the ICML, Vol. 97. PMLR.
  4. Marco Ancona, Cengiz Öztireli, Enea Ceolini, and Markus Gross. 2017. A unified view of gradient-based attribution methods for Deep Neural Networks. In Proceedings of the NIPS Workshop on Interpreting, Explaining and Visualizing Deep Learning.
  5. Christopher J. Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. 2020. Fairwashing explanations with off-manifold detergent. In Proceedings of the ICML, Vol. 119. PMLR.
  6. Vijay Arya Rachel K. E. Bellamy Pin-Yu Chen Amit Dhurandhar Michael Hind Samuel C. Hoffman Stephanie Houde Q Vera Liao Ronny Luss Aleksandra Mojsilović et al. 2019. One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. Retrieved from https://arXiv:1909.03012.
  7. Cher Bass, Mariana da Silva, Carole H. Sudre, Petru-Daniel Tudosiu, Stephen M. Smith, and Emma C. Robinson. 2020. ICAM: Interpretable classification via disentangled representations and feature attribution mapping. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/56f9f88906aebf4ad985aaec7fa01313-Abstract.html.
  8. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI Workshop on XAI.
  9. Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Cynthia Liu, Pin-Yu Chen, Shiyu Chang, and Luca Daniel. 2020. Proper network interpretability helps adversarial robustness in classification. In Proceedings of the ICML, Vol. 119. PMLR.
  10. Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. 2020. Concise explanations of neural networks using adversarial training. In Proceedings of the ICML, Vol. 119. PMLR. http://proceedings.mlr.press/v119/chalasani20a.html.
  11. Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. 2019. Explaining image classifiers by counterfactual generation. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=B1MXz20cYQ.
  12. Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. 2019. This looks like that: Deep learning for interpretable image recognition. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2019/hash/adf7ee2dcf142b0e11888e72b43fcb75-Abstract.html.
  13. Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. 2018. Learning to explain: An information-theoretic perspective on model interpretation. In Proceedings of the ICML, Vol. 80. PMLR.
  14. Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. 2019. L-shapley and C-shapley: Efficient model interpretation for structured data. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=S1E3Ko09F7.
  15. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2016/hash/7c9d0b1f96aebd7b5eca8c3edaa19ebb-Abstract.html.
  16. Michael Chromik and Martin Schuessler. 2020. A taxonomy for human subject evaluation of black-box explanations in XAI. In Proceedings of the ExSS-ATEC’20.
  17. Jonathan Crabbé, Yao Zhang, William R. Zame, and Mihaela van der Schaar. 2020. Learning outside the Black-Box: The pursuit of interpretable models. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/ce758408f6ef98d7c7a7b786eca7b3a8-Abstract.html.
  18. Mark W. Craven and Jude W. Shavlik. 1995. Extracting tree-structured representations of trained networks. In Proceedings of the NIPS. MIT Press. Retrieved from http://papers.nips.cc/paper/1152-extracting-tree-structured-representations-of-trained-networks.
  19. Arun Das and Paul Rad. 2020. Opportunities and challenges in explainable artificial intelligence (xai): A survey. Retrieved from https://arXiv:2006.11371.
  20. Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Pai-Shun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Proceedings of the NeurIPS.
  21. Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. 2019. Explanations can be manipulated and geometry is to blame. In Proceedings of the NeurIPS.
  22. Christian Etmann, Sebastian Lunz, Peter Maass, and Carola Schönlieb. 2019. On the connection between adversarial robustness and saliency map interpretability. In Proceedings of the ICML, Vol. 97. PMLR. Retrieved from http://proceedings.mlr.press/v97/etmann19a.html.
  23. Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. 2019. SOM-VAE: Interpretable discrete representation learning on time series. In Proceedings of the ICLR.
  24. Christopher Frye, Colin Rowat, and Ilya Feige. 2020. Asymmetric shapley values: Incorporating causal knowledge into model-agnostic explainability. In Proceedings of the NeurIPS.
  25. Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. 2019. Towards automatic concept-based explanations. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2019/hash/77d2afcb31f6493e350fca61764efb9a-Abstract.html.
  26. Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual visual explanations. In Proceedings of the ICML, Vol. 97. PMLR. Retrieved from http://proceedings.mlr.press/v97/goyal19a.html.
  27. Tian Guo, Tao Lin, and Nino Antulov-Fantulin. 2019. Exploring interpretable LSTM neural networks over multi-variable data. In Proceedings of the ICML, Vol. 97. PMLR.
  28. Wenbo Guo, Sui Huang, Yunzhe Tao, Xinyu Xing, and Lin Lin. 2018. Explaining deep learning models - a bayesian non-parametric approach. In Proceedings of the NeurIPS.
  29. Peter Hase, Harry Xie, and Mohit Bansal. 2021. The out-of-distribution problem in explainability and search methods for feature importance explanations. In Proceedings of the NeurIPS, Vol. 34. Curran Associates, Inc.
  30. Juyeon Heo, Sunghwan Joo, and Taesup Moon. 2019. Fooling neural network interpretations via adversarial model manipulation. In Proceedings of the NeurIPS.
  31. Jay Heo, Haebeom Lee, Saehoon Kim, Juho Lee, Kwang Joon Kim, Eunho Yang, and Sung Ju Hwang. 2018. Uncertainty-aware attention for reliable interpretation and prediction. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2018/hash/285e19f20beded7d215102b49d5c09a0-Abstract.html.
  32. Bernease Herman. 2017. The promise and peril of human evaluation for model interpretability. Retrieved from https://arXiv:1711.07414.
  33. Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. 2020. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. In Proceedings of the NeurIPS.
  34. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the NeurIPS, Vol. 30. Curran Associates, Inc.
  35. Robert R. Hoffman Shane T. Mueller Gary Klein and Jordan Litman. 2019. Metrics for explainable AI: Challenges and prospects. Retrieved from https://arXiv:1812.04608.
  36. Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In Proceedings of the NeurIPS.
  37. Lukas Hoyer, Mauricio Munoz, Prateek Katiyar, Anna Khoreva, and Volker Fischer. 2019. Grid saliency for context explanations of semantic segmentation. In Proceedings of the NeurIPS.
  38. Wei-Ning Hsu, Yu Zhang, and James R. Glass. 2017. Unsupervised learning of disentangled and interpretable representations from sequential data. In Proceedings of the NeurIPS.
  39. Aya Abdelsalam Ismail, Mohamed K. Gunady, Héctor Corrada Bravo, and Soheil Feizi. 2020. Benchmarking deep learning interpretability in time series predictions. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/47a3893cc405396a5c30d91320572d6d-Abstract.html.
  40. Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. 2020. Multi-objective molecule generation using interpretable substructures. In Proceedings of the ICML, Vol. 119. PMLR. Retrieved from http://proceedings.mlr.press/v119/jin20b.html.
  41. Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren. 2020. Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=BkxRRkSKwr.
  42. Been Kim, Oluwasanmi Koyejo, and Rajiv Khanna. 2016. Examples are not enough, learn to criticize! Criticism for interpretability. In Proceedings of the NeurIPS.
  43. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the ICML, Vol. 80. PMLR.
  44. Hyunjik Kim and Andriy Mnih. 2018. Disentangling by factorising. In Proceedings of the ICML, Vol. 80. PMLR.
  45. Wonjae Kim and Yoonho Lee. 2019. Learning dynamics of attention: Human prior for interpretable machine reasoning. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2019/hash/ae3539867aaeec609a4260c6feb725f4-Abstract.html.
  46. Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. 2018. Learning how to explain neural networks: PatternNet and pattern attribution. In Proceedings of the ICLR.
  47. Isaac Lage Emily Chen Jeffrey He Menaka Narayanan Been Kim Sam Gershman and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. Retrieved from https://arXiv:1902.00006.
  48. Isaac Lage, Andrew Slavin Ross, Samuel J. Gershman, Been Kim, and Finale Doshi-Velez. 2018. Human-in-the-loop interpretability prior. In Proceedings of the NeurIPS.
  49. Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. 2020. Robust and stable black box explanations. In Proceedings of the ICML, Vol. 119. PMLR.
  50. Himabindu Lakkaraju and Jure Leskovec. 2016. Confusions over time: An interpretable bayesian model to characterize trends in decision making. In Proceedings of the NeurIPS.
  51. Matthew L. Leavitt and Ari Morcos. 2020. Towards falsifiable interpretability research. Retrieved from https://arXiv:2010.12016.
  52. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the Text Summarization Branches Out. ACL.
  53. Yang Liu, Sujay Khandagale, Colin White, and Willie Neiswanger. 2021. Synthetic benchmarks for scientific research in explainable machine learning. In Proceedings of the NeurIPS Datasets and Benchmarks Track.
  54. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the NeurIPS.
  55. Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. 2020. Parameterized explainer for graph neural network. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/e37b08dd3015330dcbb5d6663667b8b8-Abstract.html.
  56. José Oramas M., Kaili Wang, and Tinne Tuytelaars. 2019. Visual explanation by interpretation: Improving visual feedback capabilities of deep neural networks. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=H1ziPjC5Fm.
  57. João Marques-Silva, Thomas Gerspacher, Martin C. Cooper, Alexey Ignatiev, and Nina Narodytska. 2020. Explaining naive bayes and other linear classifiers with polynomial time and delay. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/eccd2a86bae4728b38627162ba297828-Abstract.html.
  58. Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. In Proceedings of the IJCAI Workshop on Explainable Artificial Intelligence.
  59. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/c74956ffb38ba48ed6ce977af6727275-Abstract.html.
  60. Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. 2016. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Proceedings of the NeurIPS, Vol. 29. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2016/file/5d79099fcdf499f12b79770834c0164a-Paper.pdf.
  61. Weili Nie, Yang Zhang, and Ankit Patel. 2018. A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. In Proceedings of the ICML, Vol. 80. PMLR. Retrieved from http://proceedings.mlr.press/v80/nie18a.html.
  62. Matthew R. O’Shaughnessy, Gregory Canal, Marissa Connor, Christopher Rozell, and Mark A. Davenport. 2020. Generative causal explanations of black-box classifiers. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/3a93a609b97ec0ab0ff5539eb79ef33a-Abstract.html.
  63. Jung Yeon Park, Kenneth Theo Carr, Stephan Zheng, Yisong Yue, and Rose Yu. 2020. Multiresolution tensor learning for efficient and interpretable spatial analysis. In Proceedings of the ICML, Vol. 119. PMLR.
  64. Tejaswini Pedapati, Avinash Balakrishnan, Karthikeyan Shanmugam, and Amit Dhurandhar. 2020. Learning global transparent models consistent with local contrastive explanations. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/24aef8cb3281a2422a59b51659f1ad2e-Abstract.html.
  65. Vitali Petsiuk, Abir Das, and Kate Saenko. 2018. RISE: Randomized input sampling for explanation of black-box models. In Proceedings of the BMVC.
  66. Gregory Plumb, Maruan Al-Shedivat, Ángel Alexander Cabrera, Adam Perer, Eric P. Xing, and Ameet Talwalkar. 2020. Regularizing black-box models for improved interpretability. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/770f8e448d07586afbf77bb59f698587-Abstract.html.
  67. Gregory Plumb, Denali Molitor, and Ameet S. Talwalkar. 2018. Model agnostic supervised local explanations. In Proceedings of the NeurIPS.
  68. Gregory Plumb, Jonathan Terhorst, Sriram Sankararaman, and Ameet Talwalkar. 2020. Explaining groups of points in low-dimensional representations. In Proceedings of the ICML, Vol. 119. PMLR.
  69. Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. 2020. Explain your move: Understanding agent actions using specific and relevant feature attribution. In Proceedings of the ICLR.
  70. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2017/hash/dc6a7e655d7e5840e66733e9ee67cc69-Abstract.html.
  71. Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. 2020. Model agnostic multilevel explanations. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/426f990b332ef8193a61cc90516c1245-Abstract.html.
  72. Kaivalya Rawal and Himabindu Lakkaraju. 2020. Beyond individualized recourse: Interpretable and interactive summaries of actionable recourses. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/8ee7730e97c67473a424ccfeff49ab20-Abstract.html.
  73. Laura Rieger, Chandan Singh, W. James Murdoch, and Bin Yu. 2020. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In Proceedings of the ICML, Vol. 119. PMLR.
  74. Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. 2022. A consistent and efficient evaluation strategy for attribution methods. In Proceedings of the ICML, Vol. 162. PMLR.
  75. Benjamin Sanchez-Lengeling, Jennifer Wei, Brian Lee, Emily Reif, Peter Wang, Wesley Qian, Kevin McCloskey, Lucy Colwell, and Alexander Wiltschko. 2020. Evaluating attribution for graph neural networks. In Proceedings of the NeurIPS, Vol. 33. Curran Associates.
  76. Patrick Schwab and Walter Karlen. 2019. CXPlain: Causal explanations for model interpretation under uncertainty. In Proceedings of the NeurIPS.
  77. Wenxian Shi, Hao Zhou, Ning Miao, and Lei Li. 2020. Dispersed exponential family mixture VAEs for interpretable text generation. In Proceedings of the ICML, Vol. 119. PMLR.
  78. Chandan Singh, W. James Murdoch, and Bin Yu. 2019. Hierarchical interpretations for neural network predictions. In Proceedings of the ICLR.
  79. Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. 2020. Explanation by progressive exaggeration. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=H1xFWgrFPS.
  80. Leon Sixt, Maximilian Granz, and Tim Landgraf. 2020. When explanations lie: Why many modified BP attributions fail. In Proceedings of the ICML, Vol. 119. PMLR. http://proceedings.mlr.press/v119/sixt20a.html.
  81. Mukund Sundararajan and Amir Najmi. 2020. The many shapley values for model explanation. In Proceedings of the ICML, Vol. 119. PMLR. Retrieved from http://proceedings.mlr.press/v119/sundararajan20b.html.
  82. Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. 2020. Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=BkgnhTEtDS.
  83. Michael Tsang, Hanpeng Liu, Sanjay Purushotham, Pavankumar Murali, and Yan Liu. 2018. Neural interaction transparency (NIT): Disentangling learned interactions for improved interpretability. In Proceedings of the NeurIPS.
  84. Michael Tsang, Sirisha Rambhatla, and Yan Liu. 2020. How does this interaction affect me? Interpretable attribution for feature interactions. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/443dec3062d0286986e21dc0631734c9-Abstract.html.
  85. Ramakrishna Vedantam, Karan Desai, Stefan Lee, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2019. Probabilistic neural symbolic models for interpretable visual question answering. In Proceedings of the ICML, Vol. 97. PMLR.
  86. Andrey Voynov and Artem Babenko. 2020. Unsupervised discovery of interpretable directions in the GAN latent space. In Proceedings of the ICML, Vol. 119. PMLR. Retrieved from http://proceedings.mlr.press/v119/voynov20a.html.
  87. Minh N. Vu and My T. Thai. 2020. PGM-explainer: Probabilistic graphical model explanations for graph neural networks. In Proceedings of the NeurIPS.
  88. Pei Wang and Nuno Vasconcelos. 2019. Deliberative explanations: Visualizing network insecurities. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2019/hash/68053af2923e00204c3ca7c6a3150cf7-Abstract.html.
  89. Shengjie Wang, Tianyi Zhou, and Jeff A. Bilmes. 2019. Bias also matters: Bias attribution for deep neural network explanation. In Proceedings of the ICML, Vol. 97. PMLR.
  90. Tong Wang. 2018. Multi-value rule sets for interpretable classification with feature-efficient representations. In Proceedings of the NeurIPS.
  91. Tong Wang. 2019. Gaining free or low-cost interpretability with interpretable partial substitute. In Proceedings of the ICML, Vol. 97. PMLR. Retrieved from http://proceedings.mlr.press/v97/wang19a.html.
  92. Andrew G. Wilson, Christoph Dann, Chris Lucas, and Eric P. Xing. 2015. The human kernel. In Proceedings of the NeurIPS, Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/4462bf0ddbe0d0da40e1e828ebebeb11-Paper.pdf.
  93. Ning Xie Gabrielle Ras Marcel van Gerven and Derek Doran. 2020. Explainable deep learning: A field guide for the uninitiated. Retrieved from https://arXiv:2004.14545.
  94. Fan Yang Mengnan Du and Xia Hu. 2019. Evaluating explanation without ground truth in interpretable machine learning. Retrieved from https://arXiv:1907.06831.
  95. Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. 2019. On the (in)fidelity and sensitivity of explanations. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2019/hash/a7471fdc77b3435276507cc8f2dc2569-Abstract.html.
  96. Chih-Kuan Yeh, Been Kim, Sercan Ömer Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. 2020. On completeness-aware concept-based explanations in deep neural networks. In Proceedings of the NeurIPS. Retrieved from https://proceedings.neurips.cc/paper/2020/hash/ecb287ff763c169694f682af52c1f309-Abstract.html.
  97. Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. 2019. GNNExplainer: Generating explanations for graph neural networks. In Proceedings of the NeurIPS.
  98. Haizi Yu and Lav R. Varshney. 2017. Towards deep interpretability (MUS-ROVER II): Learning hierarchical representations of tonal music. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=ryhqQFKgl.
  99. Hao Zhang Jiayi Chen Haotian Xue and Quanshi Zhang. 2019. Towards a unified evaluation of explanation methods without ground truth. Retrieved from https://arXiv:1911.09017.
  100. Xin Zhang, Armando Solar-Lezama, and Rishabh Singh. 2018. Interpreting neural network judgments via minimal, stable, and symbolic corrections. In Proceedings of the NeurIPS.
  101. Yuan Zhao and Il Memming Park. 2016. Interpretable nonlinear dynamic modeling of neural trajectories. In Proceedings of the NeurIPS.
  102. Charles Y. Zheng, Francisco Pereira, Chris I. Baker, and Martin N. Hebart. 2019. Revealing interpretable object representations from human behavior. In Proceedings of the ICLR. OpenReview.net. Retrieved from https://openreview.net/forum?id=ryxSrhC9KX.

Cited by

Items 101–200 of 592

Metadata

Authors
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C.
Affiliations
  • University of Duisburg-Essen (DE)
  • University of Twente (NL)
  • University of Twente, the Netherlands
Venue
ACM Computing Surveys
Published
2023-07-13
Volume
55
Issue
13s
Pages
1-42
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4321786089
Cited by
590
References
202
Citations by year
2026: 184 · 2025: 195 · 2024: 157 · 2023: 45
FWCI
12.16
Type
journal article
Language
en
Open access
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (publishedVersion)
  • University of Twente Research Information (publishedVersion, other-oa)
  • University of Twente Research Information (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
Ziekenhuis Groep Twente
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
explainable AI, XAI evaluation, explanation quality metrics, interpretability, quantitative evaluation methods, anecdotal evidence, user evaluation, benchmarking, black-box models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.