Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Abstract
The most successful Machine Learning (ML) systems remain complex black boxes to end-users, and even experts are often unable to understand the rationale behind their decisions. The lack of transparency of such systems can have severe consequences or poor uses of limited valuable resources in medical diagnosis, financial decision-making, and in other high-stake domains. Therefore, the issue of ML explanation has experienced a surge in interest from the research community to application domains. While numerous explanation methods have been explored, there is a need for evaluations to quantify the quality of explanation methods to determine whether and to what extent the offered explainability achieves the defined objective, and compare available explanation methods and suggest the best explanation from the comparison for a specific task. This survey paper presents a comprehensive overview of methods proposed in the current literature for the evaluation of ML explanations. We identify properties of explainability from the review of definitions of explainability. The identified properties of explainability are used as objectives that evaluation metrics should achieve. The survey found that the quantitative metrics for both model-based and example-based explanations are primarily used to evaluate the parsimony/simplicity of interpretability, while the quantitative metrics for attribution-based explanations are primarily used to evaluate the soundness of fidelity of explainability. The survey also demonstrated that subjective measures, such as trust and confidence, have been embraced as the focal point for the human-centered evaluation of explainable systems. The paper concludes that the evaluation of ML explanations is a multidisciplinary research topic. It is also not possible to define an implementation of evaluation metrics, which can be applied to all explanation methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“They found that some properties (clarity, broadness, and completeness) are still in shortage of appropriate metrics, and so is the class of explanations that are example-based.”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [141]
In the citing paper
“It is not possible to come up with quantitative metrics that are applicable for all interpretability methods [169].”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [169]
In the citing paper
“They conclude their survey by stating that”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [141]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 2×
“One way to objectively evaluate explanations is to verify whether the explanation mechanism satisfies (or does not satisfy) certain axioms, or properties [34].”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“[39] explained an ensemble of models, such as neural networks with a single model effectively.”
BackgroundCausality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Explanation is also a domain-specific note, and there cannot be an all-purpose explanation [22], and different types of explanations might be useful.”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“The machine learning explainability has been reviewed thoroughly in recent years [10,14,17,23,28,31,32].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 4×
“…Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity…”
BackgroundInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 4×
“The advantages of application-grounded and human-grounded evaluations are that they can provide direct and strong evidence of success of explanations [9].”
BackgroundMethodInfluentialMethods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 3×
“…[80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity [87] Sensitivity-n [88] Mutual information [80] Example-basedexplanations Non-representativeness [80] Diversity [80]”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“[29] stated that interpretability and fidelity are both necessary components for explainability.”
Background(2009) · Cited by 1,900
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 1×
“While causability refers to the measurable extent to which an explanation to a human achieves a specified level of causal understanding [25], and it does refer to a human model.”
BackgroundSelf-citationMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 5×
“However, most of the work in the ML explanation research field focuses on creating new methods and techniques to improve explainability in prediction tasks and try to minimise the decrease in the prediction accuracy [17].”
BackgroundInfluentialCan we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 1×
“However, because of the black box nature of ML models [6,7], the deployment of ML algorithms, especially in high stake domains, such as medical diagnosis, criminal justice, financial decision-making, and other regulated safety critical domains, requires verification and testing for plausibility by…”
BackgroundUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 2×
“For example, feature importance [79] or influence [38,64] based explanation approaches belong to this category.”
BackgroundExplainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377 · cited 2×
“The challenge is that there is no ground truth when evaluating post-hoc explanations, as the real inner workings of the model is not known [60].”
BackgroundDARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · cited 1×
“The issue of ML explanation has experienced a significant surge in interest since the launch of the USA’s Defense Advanced Research Projects Agency (DARPA) initiative [12], from the international research community [13–16] and to various application domains in recent years [17–19], which can be demonstrated by the eno…”
BackgroundInfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 1,156 · cited 1×
“For example, Chen et al. [40] described a GAN-based approach to learn interpretable representations.”
BackgroundInterpretable Machine Learning
Molnar, C. (2020) · Cited by 1,104
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906 · cited 1×
“…Advanced Research Projects Agency (DARPA) initiative [12], from the international research community [13–16] and to various application domains in recent years [17–19], which can be demonstrated by the enormous and quickly growing research publications on ML explanations, as shown in Figure 1.”
BackgroundSelf-citationInterpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
How AI can be a force for good
Taddeo, M., & Floridi, L. (2018) · Science · doi:10.1126/science.aat5991 · Cited by 760 · cited 1×
“…medicine and personalised medicine; in agriculture, ML can detect crop disease, and spray pesticide to crops with pinpoint accuracy and can help to save our forest ecosystems [1–4]; and, in scientific research, ML can fuse heterogeneous bibliographic information to efficiently extract knowledge [5].”
BackgroundInterpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · cited 2×
“• Studies using proxy tasks to evaluate how well users are able to simulate the model’s decisions [6,70,71] or decision boundaries [72].”
BackgroundInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · cited 5×
“Model size [14,46] Runtime operation counts [81] Interaction strength [23,46] Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important…”
BackgroundMethodInfluentialA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 3×
“Model size [14,46] Runtime operation counts [81] Interaction strength [23,46] Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important…”
BackgroundMethodGenerating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534
Metrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518 · cited 1×
“For example, Hoffman et al. [74] presented metrics for explainable systems that are grounded in the subjective evaluation of a system (e.g., user trust, satisfaction, and understanding).”
BackgroundBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“• Studies using actual tasks to evaluate the performance of human and the system on the ML-informed decision-making tasks [68,69].”
MethodOn the (In)fidelity and Sensitivity for ExplanationsPreprint
Yeh, C., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09392 · Cited by 439 · cited 5×
“Sensitivity [84,85] is proposed to measure the degree to which the explanation is affected by insignificant perturbations from the test point.”
BackgroundInfluentialHuysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · cited 2×
“Qualitative metrics include asking about the usefulness of, satisfaction with, confidence in, and trust in provided explanations by means of interviews or questionnaires [3,63–65].”
BackgroundHolzinger, A., Carrington, A., & Müller, H. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00636-z · Cited by 406
Self-citationTowards Robust Interpretability with Self-Explaining Neural Networks
Alvarez-Melis, D., & Jaakkola, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 397 · cited 1×
“Example-based explanations provide a summarised overview of a model using representative examples or high-level concepts [80,90,91].”
BackgroundTowards better understanding of gradient-based attribution methods for Deep Neural Networks
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 368 · cited 3×
“…[80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity [87] Sensitivity-n [88] Mutual information [80] Example-basedexplanations Non-representativeness [80] Diversity [80]”
BackgroundMethodNguyen, A. T., Yosinski, J., & Clune, J. (2016) · arXiv.org · doi:10.48550/arxiv.1602.03616 · Cited by 365 · cited 1×
“For example, Nguyen et al. [35] introduced multifaceted feature visualisations to explain neural networks.”
BackgroundA Benchmark for Interpretability Methods in Deep Neural Networks
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 365 · cited 2×
“…[82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity [87] Sensitivity-n [88] Mutual information [80]…”
BackgroundToward Human-Understandable, Explainable AI
Hagras, H. (2018) · Computer · doi:10.1109/mc.2018.3620965 · Cited by 350 · cited 1×
“…interest since the launch of the USA’s Defense Advanced Research Projects Agency (DARPA) initiative [12], from the international research community [13–16] and to various application domains in recent years [17–19], which can be demonstrated by the enormous and quickly growing research…”
BackgroundCounterfactual Explanations for Machine Learning: A Review.Preprint
Verma, S., Dickerson, J. P., & Hines, K. (2020) · arXiv (Cornell University) · Cited by 332
Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow
Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mane, D., Fritz, D., Krishnan, D., Viegas, F. B., & Wattenberg, M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744878 · Cited by 331 · cited 1×
“For example, TensorFlow Graph Visualizer [59] is used to visualise data flow graphs of deep learning models in TensorFlow to help users understand, debug, and share the structure of their deep learning models.”
MethodProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · cited 2×
“Since human-centered evaluations employ experiments with end-users or lay humans to assess the quality of explanations, subjective measures, such as trust and confidence, have been embraced as the focal point for the evaluation of explainable systems [33,64,65,69,92,93].”
BackgroundMethodFrom Machine Learning to Explainable AI
Holzinger, A. (2018) · 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA) · doi:10.1109/disa.2018.8490530 · Cited by 306
Self-citationYlikoski, P., & Kuorikoski, J. (2010) · Philosophical Studies · doi:10.1007/s11098-008-9324-z · Cited by 273 · cited 1×
“…strength [23,46] Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85]…”
BackgroundOne Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268 · cited 9×
“Table 1 shows examples of associations between explanation types from Reference [43] and explanation technique categories from Reference [10].”
BackgroundMethodInfluentialModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702509 · Cited by 262
Goebel, R., Chander, A., Holzinger, K., Lecue, F., Akata, Z., Stumpf, S., Kieseberg, P., & Holzinger, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-99740-7_21 · Cited by 259 · cited 1×
“…since the launch of the USA’s Defense Advanced Research Projects Agency (DARPA) initiative [12], from the international research community [13–16] and to various application domains in recent years [17–19], which can be demonstrated by the enormous and quickly growing research publications…”
BackgroundSelf-citationExplanatory Preferences Shape Learning and Inference
Lombrozo, T. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.08.001 · Cited by 229 · cited 1×
“Lombrozo [30] showed that explanations considered good are simple and broad.”
BackgroundEnsembleMatrix: interactive visualization to support machine learning with multiple classifiers
Talbot, J., Lee, B., Kapoor, A., & Tan, D. S. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518895 · Cited by 226 · cited 1×
“EnsembleMatrix [52] allows users to visually ensemble multiple classifiers together and provides a summary visualisation of results of these multiple classifiers.”
BackgroundOn quantitative aspects of model interpretabilityPreprint
Nguyen, A.-P., & Martínez, M. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07584 · Cited by 179 · cited 8×
“…counts [81] Interaction strength [23,46] Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation…”
BackgroundMethodInfluentialQuantifying Interpretability and Trust in Machine Learning SystemsPreprint
Schmidt, P. S., & Bießmann, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08558 · Cited by 161 · cited 1×
“Based on approaches that quantify response times in user studies as well as agreement between human labels and model predictions [63,72], Schmidt and Biessmann [78] proposed that faster and more accurate decisions indicate intuitive understanding of explanations.”
BackgroundReview Study of Interpretation Methods for Future Interpretable Machine Learning
Mi, J.-X., Li, A.-D., & Zhou, L.-F. (2020) · IEEE Access · doi:10.1109/access.2020.3032756 · Cited by 158 · cited 1×
“The machine learning explainability has been reviewed thoroughly in recent years [10,14,17,23,28,31,32].”
BackgroundNeto, M. P., & Paulovich, F. V. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030354 · Cited by 143 · cited 1×
“Neto and Paulovich [55] presented a matrix-like visual metaphor to explain rule-based Random Forest models, where logic rules are rows, features are columns, and rule predicates are cells.”
BackgroundWeitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2019) · Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3308532.3329441 · Cited by 136 · cited 1×
“and preference (see Figure 4), which have been largely embraced as the focal point for the evaluation of explainable systems [33,64,65,73].”
BackgroundRüping, S. (2006) · Technische Universität Dortmund Eldorado (Technische Universität Dortmund) · doi:10.17877/de290r-8863 · Cited by 128 · cited 1×
“It is hard to formalise it [17,21].”
Human Evaluation of Models Built for Interpretability
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S. J., & Doshi‐Velez, F. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5280 · Cited by 118 · cited 1×
“• Studies using proxy tasks to evaluate how well users are able to simulate the model’s decisions [6,70,71] or decision boundaries [72].”
Machine Learning for Human Learners: Opportunities, Issues, Tensions and Threats
Webb, M. E., Fluck, A., Magenheim, J., Malyn-Smith, J., Waters, J., Deschênes, M., & Zagami, J. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-020-09858-2 · Cited by 116 · cited 1×
“Following these benefits and risks, References [43,44] identified six types of explanations as follows:”
BackgroundAssessing the Local Interpretability of Machine Learning ModelsPreprint
Slack, D., Friedler, S. A., Scheidegger, C., & Roy, C. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03501 · Cited by 107 · cited 2×
“Model size [14,46] Runtime operation counts [81] Interaction strength [23,46] Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important…”
BackgroundA survey on ethical principles of AI and implementations
Zhou, J., Chen, F., Berry, A., Reed, M., Zhang, S., & Savage, S. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308437 · Cited by 99 · cited 1×
“Of all these explanations, responsibility and fairness are two ethical principles of artificial intelligence, which are different from explainability [45].”
BackgroundSelf-citationBlack Box Explanation by Learning Image Exemplars in the Latent Feature Space
Guidotti, R., Monreale, A., Matwin, S., & Pedreschi, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46150-8_12 · Cited by 93
ViCE: visual counterfactual explanations for machine learning models
Gomez, O., Holter, S., Yuan, J., & Bertini, E. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377536 · Cited by 91 · cited 1×
“[58] proposed a visual analytics tool to generate counterfactual explanations by identifying the minimal set of changes needed to flip the model’s output.”
MethodUser Trust Dynamics: An Investigation Driven by Differences in System Performance
Yu, K., Berkovsky, S., Taib, R., Conway, D., Zhou, J., & Chen, F. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025219 · Cited by 90 · cited 2×
“Since human-centered evaluations employ experiments with end-users or lay humans to assess the quality of explanations, subjective measures, such as trust and confidence, have been embraced as the focal point for the evaluation of explainable systems [33,64,65,69,92,93].”
BackgroundMethodSelf-citationQuantifying Model Complexity via Functional Decomposition for Better Post-hoc Interpretability
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_17 · Cited by 86 · cited 1×
“It is argued that model complexity and explainability are deeply intertwined and reducing complexity can help to make the model explanation more reliable and compact [89].”
BackgroundThe European Legal Framework for Medical AI
Schneeberger, D., Stöger, K., & Holzinger, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_12 · Cited by 85 · cited 1×
“deployment of ML algorithms, especially in high stake domains, such as medical diagnosis, criminal justice, financial decision-making, and other regulated safety critical domains, requires verification and testing for plausibility by domain experts not only for safety but for legal reasons [8].”
BackgroundSelf-citationHutchison, D., Weikum, G., Vardi, M. Y., Tygar, D., Terzopoulos, D., Sudan, M., Steffen, B., Simoff, S. J., Pandu Rangan, C., Nierstrasz, O., et al. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-71080-6 · Cited by 83
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · arXiv · doi:10.48550/arxiv.1603.08507 · Cited by 64 · cited 1×
“For example, Hendricks et al. [41] proposed a visual explanation generation approach, which describes visual content in a specific image instance and constrains information to explain why an image instance belongs to a specific class.”
BackgroundThe Black Box, Unlocked: Predictability and Understandability in Military AI
United Nations Institute for Disarmament Research, & Holland Michel, A. (2020) · doi:10.37559/sectec/20/ai1 · Cited by 64 · cited 1×
“It is argued that two factors determine understandability of an ML system: the features of the ML system and the human’s capacity for understanding [62].”
BackgroundThe XAOS Metric – Understanding Visual Complexity as Measure of Usability
Stickel, C., Ebner, M., & Holzinger, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-16607-5_18 · Cited by 59 · cited 1×
“The name of causability was picked with reference to the already well-known concept of usability, which has been established in software engineering for years [26].”
BackgroundSelf-citationEffects of Uncertainty and Cognitive Load on User Trust in Predictive Decision Making
Zhou, J., Arshad, S. Z., Luo, S., & Chen, F. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-68059-0_2 · Cited by 59 · cited 1×
“Since human-centered evaluations employ experiments with end-users or lay humans to assess the quality of explanations, subjective measures, such as trust and confidence, have been embraced as the focal point for the evaluation of explainable systems [33,64,65,69,92,93].”
BackgroundSelf-citationMeasurable Decision Making with GSR and Pupillary Analysis for Intelligent User Interface
Zhou, J. J., Sun, J., Chen, F., Wang, Y., Taib, R., Khawaji, A., & Li, Z. (2015) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2687924 · Cited by 55 · cited 1×
“Quantitative metrics include measuring human-machine task performance in terms of accuracy, response time needed, likelihood to deviate, or ability to detect errors [3,66], and even physiological responses from humans during experimental tasks [67].”
BackgroundSelf-citationImproved Similarity Trees and their Application to Visual Data Classification
Paiva, J. G. S., Florian, L., Pedrini, H., Telles, G., & Minghim, R. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.212 · Cited by 52 · cited 1×
“The similarity tree visualisation has been developed to distinguish groups of interest within the data set in classifications [50].”
Gaining insights into support vector machine pattern classifiers using projection-based tour methods
Caragea, D., Cook, D., & Honavar, V. (2001) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/502512.502547 · Cited by 51 · cited 1×
“For example, different visualisation methods are used to examine specific values and show probabilities of picked objects visually for Naïve-Bayes [47] and Support Vector Machines (SVMs) [48].”
MethodLee, B., Kim, N., Kim, E.-S., Jang, K., Kang, M., Lim, J., Cho, J., & Lee, Y. (2020) · Forests · doi:10.3390/f11091000 · Cited by 41
Effects of Influence on User Trust in Predictive Decision Making
Zhou, J. J., Li, Z., Hu, H., Yu, K., Chen, F., Li, Z., & Wang, Y. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312962 · Cited by 39 · cited 4×
“Examples of subjective metrics are user trust, confidence, and preference (see Figure 4), which have been largely embraced as the focal point for the evaluation of explainable systems [33,64,65,73].”
BackgroundInfluentialSelf-citationMachine Learning and Knowledge Extraction
Holzinger, A., Kieseberg, P., Tjoa, A. M., & Weippl, E. R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29726-8 · Cited by 33
Self-citationMaking machine learning useable by revealing internal states update - a transparent approach
Zhou, J. J., Khawaja, M. A., Li, Z., Sun, J., Wang, Y., & Chen, F. (2016) · International Journal of Computational Science and Engineering · doi:10.1504/ijcse.2016.080214 · Cited by 27
Self-citationCan we Trust Machine Learning Results? Artificial Intelligence in Safety-Critical Decision Support.
Holzinger, K., Mak, K., Kieseberg, P., & Holzinger, A. (2018) · ERCIM news/ERCIM news online edition · Cited by 27
Self-citationDo I Trust a Machine? Differences in User Trust Based on System Performance
Yu, K., Berkovsky, S., Conway, D., Taib, R., Zhou, J. J., & Chen, F. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-90403-0_12 · Cited by 26 · cited 1×
“Yu et al. [76] investigated the trust dynamics corresponding to ML performance changes.”
BackgroundSelf-citationAHNG: Representation learning on attributed heterogeneous network
Liu, M., Liu, J., Chen, Y., Wang, M., Chen, H., & Zheng, Q. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.01.005 · Cited by 24 · cited 1×
“Some examples are: in human’s everyday life, ML systems can recognise objects in images, it can transcribe speech to text, it can translate between languages, it can recognise emotions in the images of faces or speech; in traveling, ML makes self-driving cars possible, ML systems enables drones to fly autonomously; in…”
BackgroundInterpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · cited 2×
“Model size [14,46] Runtime operation counts [81] Interaction strength [23,46] Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important…”
BackgroundChan, G. Y.-Y., Bertini, E., Nonato, L. G., Barr, B., & Silva, C. T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10614 · Cited by 20 · cited 1×
“[56] presented an interactive visual analytics system to construct an optimal global overview of the model and data behaviour by summarising the local explanations using information theory.”
MethodZhou, J. J., Bridon, C., Chen, F., Khawaji, A., & Wang, Y. (2015) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2702613.2732769 · Cited by 17 · cited 1×
“Zhou, et al. [65,75] investigated factors such as uncertainty and correlation that affect user confidence in ML-informed decision-making.”
BackgroundSelf-citationDiagnostic visualization for non-expert machine learning practitioners: A design study
Chen, D., Bellamy, R., Malkin, P. K., & Erickson, T. D. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739669 · Cited by 16 · cited 1×
“[54] proposed an interactive visualisation tool by combining ten state-of-the-art visualisation methods in ML (shaded confusion matrix, ManiMatrix, learning curve, learning curve of multiple models, McNemar Test matrix, EnsembleMatrix, Customized SmartStripes, Customized ModelTracker, confusion matrix with sub-categor…”
MethodGuo, Z., Ward, M. O., & Rundensteiner, E. A. (2011) · International Conference on Information Visualisation (IV) · doi:10.1109/iv.2011.21 · Cited by 8 · cited 1×
“For example, a visual interface named Nugget Browser [51] is introduced to allow users to interactively submit subgroup mining queries for discovering unique patterns dynamically.”
MethodCorrelation for user confidence in predictive decision making
Zhou, J. J., Arshad, S. Z., Yu, K., & Chen, F. (2016) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/3010915.3011004 · Cited by 8 · cited 4×
“Examples of subjective metrics are user trust, confidence, and preference (see Figure 4), which have been largely embraced as the focal point for the evaluation of explainable systems [33,64,65,73].”
BackgroundInfluentialSelf-citationMaking machine learning useable by revealing internal states update - a transparent approach
Chen, F., Sun, J., Wang, Y., et al. (2016) · International Journal of Computational Science and Engineering · doi:10.1504/ijcse.2016.10001041 · Cited by 8 · cited 3×
“Zhou et al. [6] revealed states of key internal variables of ML models with interactive visualisation to let users perceive what is going on inside a model.”
BackgroundMethodSelf-citationAn Interactive Bio-inspired Approach to Clustering and Visualizing Datasets
Erra, U., Frola, B., & Scarano, V. (2011) · International Conference on Information Visualisation (IV) · doi:10.1109/iv.2011.16 · Cited by 7 · cited 1×
“For example, visualisation has been used to help users to understand and guide the clustering process [49].”
MethodA Radial Visualisation for Model Comparison and Feature Identification
Zhou, J. J., Huang, W., & Chen, F. (2020) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis48177.2020.1028 · Cited by 4 · cited 1×
“Zhou et al. [57] presented a radial visualisation to compare machine learning models with a different number of features and identify important features directly from visualisation.”
BackgroundSelf-citation
Research works printed but not linked: 5
- Zhou, J., and Chen, F. (2018). Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent, Springer.
- Lage, I., and Doshi-Velez, F. (2020, January 17). Human-in-the-Loop Learning of Interpretable and Intuitive Representations. Proceedings of the ICML Workshop on Human Interpretability in Machine Learning, Vienna, Austria.
- Becker, B., Kohavi, R., and Sommerfield, D. (2001). Visualizing the Simple Bayesian Classifier. Information Visualization in Data Mining and Knowledge Discovery, Morgan Kaufmann Publishers Inc.
- Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Vaughan, J.W., and Wallach, H. (2019). Manipulating and Measuring Model Interpretability. arXiv.
- Yang, M., and Kim, B. (2019). Benchmarking Attribution Methods with Relative Feature Importance. arXiv, 1907.
Cited by
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
“A summary of many quantitative metrics for the evaluation of explainability properties (i.e., clarity, broadness, parsimony, completeness, and soundness) for different explanation types, is presented in the work of Zhou et al. [141].”
BackgroundLongo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“Furthermore, XAI has evolved into an exceedingly multidisciplinary, interdisciplinary, and transdisciplinary field.”
BackgroundSelf-citationNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
“[314] that presents a concise overview and discussion regarding evaluating XAI without a structured literature review, and the recent work of Vilone and Longo [263] that present a list of concepts related to explainability and discuss evaluation methods in combination with explainable AI methods.”
BackgroundInfluentialSalahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Metadata
- Authors
- Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A.
- Affiliations
- University of Technology Sydney (AU)
- Medical University of Graz (AT)
- Corresponding author
- Zhou, Jianlong
- Gandomi, Amir H.
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2021-03-04
- Volume
- 10
- Issue
- 5
- Pages
- 593
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W3133543405
- Cited by
- 606
- References
- 80
- Citations by year
- 2026: 75 · 2025: 138 · 2024: 179 · 2023: 109 · 2022: 84 · 2021: 20
- FWCI
- 34.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University of Technology Sydney
- Austrian Science Fund (P 32554)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable machine learning, explanation quality evaluation, evaluation metrics, explainability, model-based explanation, example-based explanations, interpretability, fidelity, human-centered evaluation, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.