The mythos of model interpretability
Open accessAbstract
In machine learning, the concept of interpretability is both important and slippery.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
As users of ML systems have grown beyond ML experts and the research community, the need to better interpret and understand them has grown, particularly in high-stakes decisions that affect individuals' health or wellbeing.
“As users of these systems have grown beyond ML experts and the research community, the need to better interpret and understand them has grown as well, particularly in the context of high-stakes decisions that affect individuals ' health or wellbeing (Lakkaraju et al., 2016; Lipton, 2018; Rudin, 2019).”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Lipton, 2018
The Introduction (page unavailable) says that as ML models enter medicine, criminal justice, and finance, humans’ inability to understand them becomes problematic. Section 2.5 links consequential algorithmic decisions to demands for interpretable models. The copy does not establish the claimed shift in users from ML experts and researchers to a broader group. (LLM, conf=0.83)
Experimental — accuracy not yet measuredThe explainable ML community has highlighted shortfalls including lack of consistent language and definitions, lack of clearly defined explainability goals and desiderata, and lack of consensus on metrics and methods for evaluating explanation quality.
“While this expansion of the field has yielded a rich body of methodological work, recently, the community has begun to highlight the shortfalls such as the lack of consistent language and definitions, the lack of clearly defined explainability goals, and desiderata; and the lack of consensus on metrics and methods of evaluating the quality of explanations (Doshi-Velez and Kim, 2017; Lipton, 2018;”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Lipton, 2018
The Contributions section says that interpretability has no agreed-upon meaning and calls for clearer problem formulation; Section 4.2 likewise urges authors to define their objectives and demonstrate that interpretations achieve them. The Evaluation Metric section explains why conventional predictive metrics may not capture interpretability goals, but the paper does not establish a lack of consensus on metrics and methods for evaluating explanation quality. (LLM, conf=0.84)
Experimental — accuracy not yet measuredPost-hoc explanation techniques are employed to explain existing models.
“Therefore, post-hoc explanation techniques are employed to explain existing models (Došilović et al., 2018; Lipton, 2018).”
From Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision task (2023) · cites this paper as Lipton, 2018
Explainability is not a monolithic concept and can play different roles in different applications.
“As explainability is not a monolithic concept and can play different roles in different applications (Lipton, 2018; Molnar, 2019), this process requires extensive domain/application-specific efforts.”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Lipton, 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · see citing works
(2009) · Cited by 1,900 · see citing works
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Hidden factors and hidden topics
McAuley, J., & Leskovec, J. (2013) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2507157.2507163 · Cited by 1,565 · see citing works
Inceptionism: Going Deeper into Neural Networks
Mordvintsev, A., Olah, C., & Tyka, M. (2015) · Cited by 569 · see citing works
Accurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564 · see citing works
Intelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547 · see citing works
Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · see citing works
Legibility and predictability of robot motion
Dragan, A. D., Lee, K. C. T., & Srinivasa, S. S. (2013) · doi:10.5555/2447556.2447672 · Cited by 346 · see citing works
An empirical study of machine learning techniques for affect recognition in human–robot interaction
Rani, P., Liu, C., Sarkar, N., & Vanman, E. (2006) · Pattern Analysis and Applications · doi:10.1007/s10044-006-0025-y · Cited by 313 · see citing works
Wang, H.-X., Fratiglioni, L., Frisoni, G. B., Viitanen, M., & Winblad, B. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009864 · Cited by 110 · see citing works
Case-based explanation of non-case-based learning methods.
Caruana, R., Kangarloo, H., Dionisio, J. D. N., Sinha, U., & Johnson, D. (1999) · PubMed Central · Cited by 92 · see citing works
Interactive and interpretable machine learning models for human machine collaboration
Kim, B. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 83 · see citing works
Learning From Explanations Using Sentiment and Advice in RL
Krening, S., Harrison, B. E., Feigh, K. M., Isbell, C. L., Riedl, M. O., & Thomaz, A. L. (2016) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2016.2628365 · Cited by 83 · see citing works
An empirical study of machine learning techniques for affect recognition in human-robot interaction
Liu, C., Rani, P., & Sarkar, N. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2005.1545344 · Cited by 67 · see citing works
(2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 34 · see citing works
Flagged first, then by citations
2 of the 27 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Doshi-Velez, Finale, Wallace, Byron, and Adams, Ryan. Graph-sparse lda: a topic model with structured sparsity. AAAI, 2015.
- Kim, Been, Rudin, Cynthia, and Shah, Julie A. The Bayesian Case Model: A generative approach for case- based reasoning and prototype classification. In NIPS, 2014.
- Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111-3119, 2013.
- Pearl, Judea. Causality. Cambridge university press, 2009.
- Ridgeway, Greg, Madigan, David, Richardson, Thomas, and O'Kane, John. Interpretable boosted naïve bayes classification. In KDD, 1998.
Cited by
Sorted by citations
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
A Comprehensive Survey on Transfer Learning
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3004555 · Cited by 6,678 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
“In other words, for the field to progress, it should be supported by a standalone research community who, at this stage of advancement, should mainly be engaged towards more formalism in terms of: (i) Systematic definitions: depending on their background, researchers use synonymously concepts that are semantically dif…”
BackgroundMethodInfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
The role of artificial intelligence in achieving the Sustainable Development Goals
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., Felländer, A., Langhans, S. D., Tegmark, M., & Fuso Nerini, F. (2020) · Nature Communications · doi:10.1038/s41467-019-14108-y · Cited by 3,294 · see citing works
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · see citing works
Recent advances and applications of machine learning in solid-state materials science
Schmidt, J., Marques, M. R. G., Botti, S., & Marques, M. A. L. (2019) · npj Computational Materials · doi:10.1038/s41524-019-0221-0 · Cited by 2,655 · see citing works
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · see citing works
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,062 · see citing works
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · see citing works
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · see citing works
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · see citing works
Investigating Ad Transparency Mechanisms in Social Media: A Case Study of Facebook's Explanations
Andreou, A., Venkatadri, G., Goga, O., Gummadi, K. P., Loiseau, P., & Mislove, A. E. (2018) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23191 · Cited by 1,505 · see citing works
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,486 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
“Orthogonal to the quest for better and more holistic ML models, explainable artificial intelligence (XAI) [18], [73], [112], [128], [159] has developed as a subfield of ML that seeks to augment the training process, the learned representations, and the decisions with human-interpretable explanations.”
BackgroundWild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,444 · see citing works
Explaining machine learning classifiers through diverse counterfactual explanations
Mothilal, R. K., Sharma, A., & Tan, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372850 · Cited by 1,233 · see citing works
Explainable artificial intelligence: A survey
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400040 · Cited by 1,147 · see citing works
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080 · see citing works
Machine learning & artificial intelligence in the quantum domain: a review of recent progress
Dunjko, V., & Briegel, H. J. (2018) · Reports on Progress in Physics · doi:10.1088/1361-6633/aab406 · Cited by 1,132 · see citing works
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063 · see citing works
Human-in-the-loop machine learning: a state of the art
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10246-w · Cited by 1,137 · see citing works
Speech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044 · see citing works
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · see citing works
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008 · see citing works
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983 · see citing works
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · see citing works
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923 · see citing works
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039 · see citing works
Jain, S., & Wallace, B. C. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1357 · Cited by 2,287 · see citing works
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882 · see citing works
“…a system is explicable (explainable and interpretable) it is inherently more transparent and therefore more accountable in terms of its decision-making properties and the extent to which they include human oversight and are fair, robust and justifiable (Binns et al. 2018; Cath 2018; Lipton 2016).”
BackgroundExplainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863 · see citing works
From DFT to machine learning: recent approaches to materials science–a review
Schleder, G. R., Padilha, A. C. M., Acosta, C. M., Costa, M., & Fazzio, A. (2019) · Journal of Physics: Materials · doi:10.1088/2515-7639/ab084b · Cited by 852 · see citing works
Ferrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875 · see citing works
Li, W., Huang, R., Li, J., Liao, Y., Chen, Z., He, G., Yan, R., & Gryllias, K. (2021) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2021.108487 · Cited by 837 · see citing works
An Introduction to Machine Learning
Badillo, S., Bánfai, B., Birzele, F., Davydov, I. I., Hutchinson, L. G., Kam‐Thong, T., Siebourg‐Polster, J., Steiert, B., & Zhang, J. D. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1796 · Cited by 836 · see citing works
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792 · see citing works
Artificial Intelligence and Black‐Box Medical Decisions: <i>Accuracy versus Explainability</i>
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801 · see citing works
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · see citing works
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783 · see citing works
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765 · see citing works
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800 · see citing works
Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges
Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., & Zhu, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32236-6_51 · Cited by 758 · see citing works
Machine learning for digital soil mapping: Applications, challenges and suggested solutions
Wadoux, A. M. J. ‐., Minasny, B., & McBratney, A. B. (2020) · Earth-Science Reviews · doi:10.1016/j.earscirev.2020.103359 · Cited by 752 · see citing works
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · see citing works
“We chose text classification because it is a popular task in natural language processing (NLP) that has been used in several previous studies on human-AI teaming [17, 24, 35, 43, 53, 70], requires little domain expertise, and is thus amenable to crowdsourcing.”
MethodLepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · see citing works
Interpretable 3D Human Action Analysis with Temporal Convolutional Networks
Kim, T. S., & Reiter, A. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.207 · Cited by 696 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · see citing works
“2Although we acknowledge this is an important challenge if explanations are to be more than just comforting stories, as cautioned by Lipton [41].”
BackgroundSerrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690 · see citing works
Explainable Recommendation a survey and new perspectives
Zhang, Y., & Chen, X. (2020) · Cited by 689 · see citing works
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · see citing works
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
Reinforcement Learning in Healthcare: A Survey
Yu, C., Liu, J., Nemati, S., & Yin, G. (2021) · ACM Computing Surveys · doi:10.1145/3477600 · Cited by 680 · see citing works
Born Again Neural NetworksPreprint
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04770 · Cited by 798 · see citing works
Self-citationMinimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1041-y · Cited by 632 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
“Existing works tend to provide just a general overview of the problem [68] highlighting unanswered questions and problems [28].”
BackgroundAntoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · see citing works
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103473 · Cited by 652 · see citing works
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607 · see citing works
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606 · see citing works
Machine learning and AI in marketing–Connecting computing power to human insights
Ma, L., & Sun, B. (2020) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2020.04.005 · Cited by 602 · see citing works
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665 · see citing works
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · see citing works
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · see citing works
Recent advances in decision trees: an updated survey
Costa, V. G., & Pedreira, C. E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10275-5 · Cited by 607 · see citing works
Longo, L., Brčić, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 648 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
Transfer learning: a friendly introduction
Hosna, A., Merry, E., Gyalmo, J., Alom, Z., Aung, Z., & Azim, M. A. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00652-w · Cited by 617 · see citing works
Making the Black Box More Transparent: Understanding the Physical Implications of Machine Learning
McGovern, A., Lagerquist, R., Gagne, D. J., Jergensen, G. E., Elmore, K. L., Homeyer, C. R., & Smith, T. M. (2019) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-18-0195.1 · Cited by 580 · see citing works
Eigen-CAM: Class Activation Map using Principal Components
Muhammad, M. B., & Yeasin, M. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206626 · Cited by 584 · see citing works
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561 · see citing works
On Interpretability of Artificial Neural Networks: A Survey
Fan, F., Xiong, J., Li, M., & Wang, G. (2021) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2021.3066428 · Cited by 573 · see citing works
Nanoparticle synthesis assisted by machine learning
Tao, H., Wu, T., Aldeghi, M., Wu, T., Aspuru‐Guzik, A., & Kumacheva, E. (2021) · Nature Reviews Materials · doi:10.1038/s41578-021-00337-5 · Cited by 558 · see citing works
Fink, O., Wang, Q., Svensén, M., Dersin, P., Lee, W.-J., & Ducoffe, M. (2020) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2020.103678 · Cited by 549 · see citing works
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545 · see citing works
Jan, Z. M., Ahamed, F., Mayer, W., Patel, N., Großmann, G., Stumptner, M., & Kuusk, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119456 · Cited by 586 · see citing works
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · see citing works
Huang, X., Kroening, D., Ruan, W., Sharp, J. J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 567 · see citing works
HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., & Mukherjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17745 · Cited by 585 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · see citing works
Artificial Intelligence and Marketing: Pitfalls and Opportunities
Bruyn, A. D., Viswanathan, V., Beh, Y. S., Brock, J. K., & von Wangenheim, F. (2020) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2020.04.007 · Cited by 571 · see citing works
Interpretable Machine Learning
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2021) · Queue · doi:10.1145/3511299 · Cited by 540 · see citing works
Explainable machine learning in materials science
Zhong, X., Gallagher, B. J., Liu, S., Kailkhura, B., Hiszpanski, A. M., & Han, T. Y.-J. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00884-7 · Cited by 526 · see citing works
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515 · see citing works
Salahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508 · see citing works
Reinforcement Learning based Recommender Systems: A Survey
Afsar, M. M., Crump, T., & Far, B. H. (2022) · ACM Computing Surveys · doi:10.1145/3543846 · Cited by 560 · see citing works
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 542 · see citing works
- Authors
- Lipton, Z. C.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Lipton, Zachary C.
- ORCID
- Lipton, Zachary C.
- Venue
- Communications of the ACM
- Published
- 2018-09-26
- Volume
- 61
- Issue
- 10
- Pages
- 36-43
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2439568532
- Cited by
- 5,427
- References
- 27
- Citations by year
- 2026: 803 · 2025: 918 · 2024: 633 · 2023: 657 · 2022: 575 · 2021: 664 · 2020: 521 · 2019: 395 · 2018: 190 · 2017: 62 · 2016: 8 · 2012: 1
- FWCI
- 249.23
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Communications of the ACM (publishedVersion)
- Queue (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- model interpretability, explainable AI, machine learning models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
