Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning
The DOI matches its Crossref record; 63 of the 65 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
Machine learning (ML) models are now routinely deployed in domains ranging from criminal justice to healthcare. With this newfound ubiquity, ML has moved beyond academia and grown into an engineering discipline. To that end, interpretability tools have been designed to help data scientists and machine learning practitioners better understand how ML models work. However, there has been little evaluation of the extent to which these tools achieve this goal. We study data scientists' use of two existing interpretability tools, the InterpretML implementation of GAMs and the SHAP Python package. We conduct a contextual inquiry (N=11) and a survey (N=197) of data scientists to observe how they use interpretability tools to uncover common issues that arise when building and evaluating ML models. Our results indicate that data scientists over-trust and misuse interpretability tools. Furthermore, few of our participants were able to accurately describe the visualizations output by these tools. We highlight qualitative themes for data scientists' mental models of interpretability tools. We conclude with implications for researchers and tool designers, and contextualize our findings in the social science literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further work has found that even data scientists “over-trust and misuse interpretability tools” and that few such experts were able to accurately describe visualisations output by interpretability tools.”
From The false hope of current approaches to explainable artificial intelligence in health care (2021) · cites this paper as [16]
In the citing paper
“However, the previous research discusses the issues of these feature-based explanations on overtrust in AI [29, 37, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [29]
In the citing paper
“researchers have discussed the importance of an AI explanation to communicate an AI output to a user [5, 20, 29] and realize human-AI collaboration [48] for a decision-making task [15, 42].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [29]
In the citing paper
“recent studies that showed explanations can mislead data scientists into overtrusting ML models for deployment [38].”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [38]
In the citing paper
“This finding is aligned with another research where data scientist practitioners overtrusted the XAI methods without a careful evaluation of their explanations [31].”
From AI Trust: Can Explainable AI Enhance Warranted Trust? (2023) · cites this paper as [31]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“It assigns each input feature an importance score for each prediction [44].”
BackgroundMethodInfluentialGrice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
(2011) · Cited by 14,317
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Grounded theory research: Procedures, canons, and evaluative criteria
Corbin, J. M., & Strauss, A. (1990) · Qualitative Sociology · doi:10.1007/bf00988593 · Cited by 11,133
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“Rudin [56] argues against the use of post-hoc explanation techniques for ML models deployed in high-stakes domains because they may not faithfully represent the models’ behavior.”
BackgroundMethodLightGBM: A Highly Efficient Gradient Boosting Decision Tree
(2017) · HAL · Cited by 10,512
Braun, V., & Clarke, V. (2012) · APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. · doi:10.1037/13620-004 · Cited by 10,299
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 3×
“Lipton [39] surveys different criteria for assessing interpretability, such as simulatability, as well as different goals that interpretability may be used to achieve.”
BackgroundShapley, L. S. (1988) · The Shapley Value · doi:10.1017/cbo9780511528446.003 · Cited by 4,954
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 4×
“Generating explanations in human-interpretable ways, and evaluating them as such, is at the top of the specificity-costliness taxonomy pyramid presented by Doshi-Velez and Kim [12].”
BackgroundInfluentialSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 1×
“For example, ML systems have been found to amplify biases in data and lead to unfair outcomes [3, 7, 28].”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“[18] provide an overview of different interpretability approaches.”
BackgroundFoundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799 · cited 1×
“[23] with respect to the interpretability tool.”
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Self-citationSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
Consistent Individualized Feature Attribution for Tree Ensembles
Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018) · arXiv · doi:10.48550/arxiv.1802.03888 · Cited by 1,430 · cited 1×
“was for three reasons: the InterpretML implementation of GAMs is based on gradient boosted decision trees, the SHAP Python package has a highly optimized routine for computing explanations for the predictions made by a LightGBM model [43], and LightGBM is widely used.”
MethodKim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · cited 1×
“, images [32, 57]), this was not the focus of our work.”
BackgroundMachine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023 · cited 1×
“With only six interviews, we cannot claim that this list is by any means exhaustive, but it is in line with other analyses of ML pitfalls [42].”
ResultDesigning Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Performance Consequences of Automation-Induced 'Complacency'
Parasuraman, R., Molloy, R., & Singh, I. L. (1993) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0301_1 · Cited by 875 · cited 2×
“Prior work on automation use classifies this as misuse [15, 49].”
BackgroundGeneralized Additive Models: Some Applications
Hastie, T., & Tibshirani, R. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478440 · Cited by 870
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711 · cited 1×
“[51] test the impact of two factors often thought to impact interpretability—the number of input features and the transparency of the model.”
BackgroundSelf-citationJAABA: Interactive machine learning for automatic annotation of animal behavior
Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S., & Branson, K. (2013) · Nature Methods · doi:10.1038/nmeth.2281 · Cited by 644 · cited 1×
“Several examples of iML systems now exist, for applications ranging from annotation of animal behavior [25], to academic citation review [58], to on-demand personalized group formations [2].”
BackgroundThe structure and function of explanations
Lombrozo, T. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.08.004 · Cited by 639 · cited 1×
“To that end, Miller [46, 45] and Lombrozo [38] propose simplicity, generality, and coherence as the main evaluation criteria for explanations.”
BackgroundFails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“…together from the start, with HCI methodologies applied at all stages of interpretability tool development: supporting need-finding studies (e.g., [8, 64]), designing tools that can be understood by users with different background (e.g., [37]), and undertaking user studies at each stage of tool…”
MethodPrinciples of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 2×
“Interactive machine learning systems leverage this by letting people ask follow-up questions while using the tool, or update values to see immediate changes [17, 22, 33].”
Background(2016) · neural information processing systems · Cited by 556
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · cited 1×
“Several examples of iML systems now exist, for applications including annotation of animal behavior [26], academic citation review [63], and on-demand personalized group formation [3].”
BackgroundUnequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · cited 1×
“ML models have been found to amplify societal biases in datasets and lead to unfair outcomes [4, 9, 29].”
BackgroundAlgorithms and their others: Algorithmic culture in context
Dourish, P. (2016) · Big Data & Society · doi:10.1177/2053951716665128 · Cited by 481 · cited 1×
“Dourish [14] adds scrutability as an important component of interactivity.”
BackgroundIntelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471 · cited 2×
“Bellotti and Edwards [6] were the first to consider intelligibility and accountability, providing guidelines for system designers.”
BackgroundCommon ground at the understanding of demonstrative reference
Clark, H. H., Schreuder, R., & Buttrick, S. (1983) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(83)90189-5 · Cited by 445 · cited 1×
“Finally, explanations under this goal-based approach must include grounding in some common demonstrative reference between the human and the explanation system [9, 43].”
BackgroundHow the mind explains behavior: folk explanations, meaning, and social interaction
(2005) · Choice Reviews Online · doi:10.5860/choice.42-6170 · Cited by 436 · cited 1×
“’s maxims of quality, quantity, relation, and manner [19] as core parts of a good conversation, and thus of explanation design [33, 44, 55].”
BackgroundOrange: From Experimental Machine Learning to Interactive Data Mining
Demšar, J., Zupan, B., Leban, G., & Curk, T. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30116-5_58 · Cited by 426 · cited 1×
“[18] to describe an approach where people are involved in an iterative process of training, using, and correcting an ML model, requiring interpretability for effective corrections [13].”
BackgroundGeneralized Additive Models; Some Applications
Hastie, T., & Tibshirani, R. (1985) · Lecture Notes in Statistics · doi:10.1007/978-1-4615-7070-7_8 · Cited by 407 · cited 1×
“GAMs are a learning method rooted in statistics that decompose the learned function into additive components that are functions of one variable each [21].”
MethodHow the Mind Explains Behavior: Folk Explanations, Meaning, and Social Interaction
Malle, B. F. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/3586.001.0001 · Cited by 356
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300356 · Cited by 261
Explainable AI for Designers: A Human-Centered Perspective on Mixed-Initiative Co-Creation
Zhu, J., Liapis, A., Risi, S., Bidarra, R., & Youngblood, G. M. (2018) · 2018 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2018.8490433 · Cited by 255 · cited 1×
“[60] provide domain specific results for challenges of designing iML systems for software developers and game designers, respectively.”
BackgroundThe challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241
Investigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 226 · cited 1×
“…with HCI methodologies applied at all stages of interpretability tool development: supporting need-finding studies (e.g., [8, 64]), designing tools that can be understood by users with different background (e.g., [37]), and undertaking user studies at each stage of tool development (e.g., [67]).”
MethodTell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225
Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models
Hohman, F., Head, A., Caruana, R., DeLine, R., & Drucker, S. M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225 · cited 4×
“, data scientists [22]) and the consequences of imperfect interpretation (e.”
BackgroundInfluentialSelf-citationInterpretable classification models for recidivism prediction
Zeng, J., Ustun, B., & Rudin, C. (2017) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12227 · Cited by 220 · cited 2×
“, simple point systems [25, 68], GAMs [10]) and post-hoc explanation techniques that are designed to make the predictions of “blackbox” models more interpretable (e.”
BackgroundMethodMiller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · cited 3×
“To that end, Miller [47, 48] and Lombrozo [41] propose simplicity, generality, and coherence as the main evaluation criteria for explanations.”
BackgroundA Transactional Model of Communication
Barnlund, D. C. (2017) · communication theory · doi:10.4324/9781315080918-5 · Cited by 211 · cited 1×
“Looking ahead, this communication lens prescribes transactional models of communication [4] that understand verbal, nonverbal, and behavioral cues for more human-centric design of interpretability tools.”
BackgroundInteracting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184 · cited 1×
“[56] study the viability of including rich human feedback in an ML-based email filtering system, finding lack of common language between humans and machines to be a key challenge.”
BackgroundRegroup: interactive machine learning for on-demand group creation in social networks
Amershi, S., Fogarty, J., & Weld, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207680 · Cited by 144
Lomborg, S., & Kapsch, P. H. (2020) · Media, Culture & Society · doi:10.1177/0163443719855301 · Cited by 142
Investigating statistical machine learning as a tool for software development
Patel, K., Fogarty, J., Landay, J. A., & Harrison, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357160 · Cited by 129 · cited 1×
“Patel et al. [53] and Zhu et al. [69] provide domain-specific challenges to designing iML systems for software developers and game designers, respectively.”
BackgroundMapping Machine Learning Advances from HCI Research to Reveal Starting Places for Design Innovation
Yang, Q., Banovic, N., & Zimmerman, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173704 · Cited by 125 · cited 1×
“To date, research on interpretability in the ML and HCI communities has evolved somewhat independently [66].”
BackgroundHuman Evaluation of Models Built for Interpretability
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S. J., & Doshi-Velez, F. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5280 · Cited by 110 · cited 1×
“[36] study two aspects of an explanation (length and complexity) via a wizard-of-oz approach in two domains, finding that longer explanations overload people’s cognitive abilities.”
BackgroundSimple Rules for Complex Decisions
Jung, J., Concannon, C., & Shroff, R. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2919024 · Cited by 87 · cited 2×
“, simple point systems [24, 59], GAMs [8]) and post-hoc explanation methods that help make blackbox models interpretable (e.”
BackgroundAttribution in conversational context: Effect of mutual knowledge on explanation‐giving
Slugoski, B. R., Lalljee, M., Lamb, R., & Ginsburg, G. P. (1993) · European Journal of Social Psychology · doi:10.1002/ejsp.2420230302 · Cited by 79 · cited 1×
“’s maxims of quality, quantity, relation, and manner [19] as core parts of a good conversation, and thus of explanation design [33, 44, 55].”
BackgroundGoal‐Based Explanation Evaluation
Leake, D. B. (1991) · Cognitive Science · doi:10.1207/s15516709cog1504_2 · Cited by 53 · cited 1×
“Leake’s goal-based approach to explanations further includes metrics such as timeliness of the explanation in providing opportunity to deal better with the predicted event, knowability and the features responsible for “knowing,” and independence of individual explanations [35].”
BackgroundDzindolet, M. T., Beck, H. P., Pierce, L. G., & Dawe, L. A. (2001) · doi:10.21236/ada393966 · Cited by 31 · cited 2×
“Previous work categorizes such over-use as misuse [17, 52].”
BackgroundGoal-based explanation evaluation
Leake, D. B. (1991) · Cognitive Science · doi:10.1016/0364-0213(91)80017-y · Cited by 12
Research works printed but not linked: 11
- David Alvarez-Melis, Hal Daumé, III, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Weight of Evidence as a Basis for Human-Oriented Explanations. arXiv preprint arXiv:1910.13503 (2019). · arXiv
- Julia Angwin, Jeff Larson, Surya Mattu, and Kirchner Lauren. 2016. Machine Bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica, May 23 (2016), 2016. http://www.propublica.org/article/machine-bias-risk -assessments-in-criminal-sentencing
- 10.1007/978--3--540--30116--5_58 · doi:10.1007/978--3--540--30116--5_58
- Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O'Brien, Stuart Schieber, James Waldo, David Weinberger, and Alexandra Wood. 2017. Accountability of AI under the law: The role of explanation. arXiv preprint arXiv:1711.01134 (2017). · arXiv
- 10.1016/s0166--4115(08)62386--9 · doi:10.1016/s0166--4115(08)62386--9
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016). · arXiv
- Prashan Madumal, Tim Miller, Frank Vetere, and Liz Sonenberg. 2018. Towards a Grounded Dialog Model for Explainable Artificial Intelligence. In First international workshop on socio-cognitive systems at IJCAI 2018. https://arxiv.org/abs/1806.08055 · arXiv
- Tim Miller. 2018. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence (2018).
- Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. 2019. InterpretML: A Unified Framework for Machine Learning Interpretability. arXiv preprint arXiv:1909.09223 (2019). · arXiv
- Sarah Tan, Rich Caruana, Giles Hooker, Paul Koch, and Albert Gordo. 2018. Learning global additive explanations for neural nets using model distillation. arXiv preprint arXiv:1801.08640 (2018). · arXiv
- Richard Tomsett, Dave Braines, Dan Harborne, Alun Preece, and Supriyo Chakraborty. 2018. Interpretable to whom? A role-based model for analyzing interpretable machine learning systems. arXiv preprint arXiv:1806.07552 (2018). · arXiv
Cited by
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME
Salih, A. M., Raisi‐Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400304 · Cited by 897
“Data scientists who are working daily on ML and XAI tend to over-trust the explanations generated by XAI methods and do not accurately understand the visualized output of the XAI methods [26], that could result in a misuse of the interpretability tools.”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7, 30].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
“As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7, 30].”
Background
Metadata
- Authors
- Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J.
- Affiliations
- University of Michigan (US)
- Microsoft (United States) (US)
- Microsoft Research New York City (United States) (US)
- Microsoft Research, New York, NY, USA
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-14
- OpenAlex
- W3016099278
- Semantic Scholar
- fb7caddac20dca012f48c90b2e1e2383f7185051
- Cited by
- 564
- References
- 60
- Citations by year
- 2026: 77 · 2025: 83 · 2024: 96 · 2023: 88 · 2022: 104 · 2021: 88 · 2020: 26
- FWCI
- 29.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- model interpretability, data scientists, SHAP, generalized additive models, tool misuse, mental models, visualization comprehension, tool design, contextual inquiry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.