Manipulating and Measuring Model Interpretability
Open accessAbstract
With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model’s predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N = 3, 800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model’s predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model’s sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Trust is measured in different ways: some researchers measure subjective trust using questionnaires, while others measure objective trust-related behavior such as reliance.
“Some researchers measure subjective trust using questionnaires, while others measure objective trust related behavior such as reliance.”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Poursabzi-Sangdeh et al., 2021
Section 3.2 (page not available) contrasts participants’ stated confidence in a model’s predictions with their observed behavior in following those predictions. This supports the distinction between subjective reports and reliance behavior, but the paper does not identify the confidence ratings as trust questionnaires or establish the broader claim about how different researchers measure trust. (LLM, conf=0.82)
Experimental — accuracy not yet measuredWOA has been used in past by researchers in XAI as an alternative trust measurement, but never explicitly referred to as reliance and clearly differentiated from trust.
“While WOA has been used in the past by researchers in XAI as an alternative trust measurement (Logg et al., 2019; Mucha et al., 2021; Poursabzi-Sangdeh et al., 2021), it has never been explicitly referred to as reliance and thus clearly differentiated from trust to the best of our knowledge.”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Poursabzi-Sangdeh et al., 2021
In §5, Experiment 3: Weight of Advice (page unavailable), the paper uses weight of advice as an alternative measure of how closely participants follow model predictions. It describes the measure as advice-taking, not explicitly as a trust measurement. The sentence’s claim that prior researchers did not distinguish reliance from trust is framed as the citing authors’ own conclusion, rather than a finding attributed to this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPrior empirical studies have not found consistent evidence that AI explanations improve human-AI decision-making, and some suggest explanations can increase overreliance when the AI is wrong.
“However, across many domains, empirical studies of explanations have produced mixed results [4,11,49,53,58,68,82,87], with some suggesting that explanations increase decisionmakers' tendency to rely on AI predictions even when the AI system is wrong [4,68,82,87]-a phenomenon often referred to as overreliance [10,78].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [68]
The Overview of experiments and Experiment 3.2 Findings report that participants shown a clear model were less able than those shown a black-box model to detect and correct sizable erroneous predictions. This supports the concern about reliance when the model is wrong, but the study manipulated model transparency and number of features, not explanations as such; its third experiment did not reproduce the difference. The paper therefore supports part of the cited sentence, not its broader characterization of explanation studies. (LLM, conf=0.84)
Experimental — accuracy not yet measuredSeveral studies found that showing feature-based explanations increases people's tendency to over-rely on the model when it is wrong, compared to showing only the AI predictions.
“On the contrary, several studies found that showing feature-based explanations increases people's tendency to over-rely on the model when it is wrong, compared to showing only the AI predictions [4,68,82,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [68]
The Overview of experiments (§1.2; page not available) reports that participants shown a clear model were less able to correct sizable mistakes on unusual apartments than participants shown a black-box model. But this paper compares access to model internals, not a distinct feature-based explanation alongside an otherwise identical prediction-only display; §5.2 also reports that the effect did not appear in the third experiment. The claim therefore captures a finding but describes the intervention and consistency more broadly than the source supports. (LLM, conf=0.86)
Experimental — accuracy not yet measuredIncreased transparency can hamper users' ability to detect sizable model errors and correct for them, seemingly due to information overload.
“Work in the human–computer interaction community has identified that increased transparency can hamper users’ ability to detect sizable model errors and correct for them, “seemingly due to information overload””
From The false hope of current approaches to explainable artificial intelligence in health care (2021) · cites this paper as [15]
Evidence suggests models that use explainability techniques can hamper people's ability to detect when a model makes serious mistakes.
“and evidence suggests models that use explainability techniques can hamper people's ability to detect when a model makes serious mistakes”
From The false hope of current approaches to explainable artificial intelligence in health care (2021) · cites this paper as [15]
Many empirical studies on human-AI decision-making did not observe complementary performance.
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [68]
Prior research has shown that providing too much detail may reduce user performance due to information overload.
“and that providing too much detail may reduce user performance due to information overload [44].”
From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [44]
More 'clear' models can lead to information overload and hinder people's ability to detect AI errors.
“a study by Poursabzi-Sangdeh et al. [48] found that more 'clear' models can actually lead to information overload and hinder people's ability to detect AI errors.”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as Poursabzi-Sangdeh et al. [48]
In Poursabzi-Sangdeh et al. (2021), the utility of explanations in achieving a decision and outcome of interest was not established.
“However, they choose to use the task of real estate valuation, and how it relates to a decision and an outcome of interest and the utility of explanations in achieving that goal was not established.”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Poursabzi-Sangdeh et al. (2021)
Algorithmic transparency reduces people's ability to detect and correct model errors.
“Algorithmic transparency similarly reduces people's ability to detect and correct model errors”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as Poursabzi-Sangdeh et al., 2021
Task complexity plays a significant role in determining the degree of reliance on AI advice.
“The complexity of tasks plays a signifcant role in determining the degree of reliance on AI advice, consistent with the fndings of [9, 105].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [105]
Section 5.2 measures reliance on model predictions but reports no significant difference between the clear two-feature and black-box eight-feature conditions. Section 6.2 reports a difference associated with how a model was presented, not with task complexity. A search for task difficulty and complexity found no passage establishing that task complexity significantly affects reliance on AI advice; page numbers were unavailable in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredExisting XAI methods often do not match end-users' mental models, making explanations ineffective and sometimes harmful.
“Such explanations often do not match the end-users' mental models [56], making the explanations ineective [2,40,65] and at times even harmful [13,28,30,47,51].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [40]
Naive application of existing XAI methods may be ineffective for a broader audience of end-users, if not harmful.
“naive application of existing XAI methods may be ineective [2,40,65] to broader audience of end-users, if not harmful [28,47,49,51].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [40]
Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems.
“Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems [20, 34, 52, 55].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [55]
Previous works show that the availability of AI's explanations, regardless of their correctness, is likely to increase people's trust in AI.
“Previous works show that the availability of AI's explanations, regardless of their correctness, is likely to increase people's trust in AI [6,54,81,100].”
From Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making (2023) · cites this paper as [81]
Many studies have investigated the role of AI in assisting humans in making individual decisions.
“Many studies have investigated the role of AI in assisting humans in making individual decisions [9,16,36, 37, 50, 51, 55, 58, 69, 83, 84, 89], e.g., predicting whether a person will recidivate in the near future.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [69]
Recent research has shown that recommendation-driven explainable/interpretable AI has little effect on decision making, although this is not always the case.
“However, recent research has shown that the recommendation-driven explainable/interpretable AI has little effect on decision making [1, 6-8, 28, 32], although this is not always the case [22,23,41].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Simple Heuristics That Make Us Smart
Gigerenzer, G., & Todd, P. M. (1999) · Cited by 3,562 · see citing works
Reasoning the fast and frugal way: Models of bounded rationality.
Gigerenzer, G., & Goldstein, D. G. (1996) · Psychological Review · doi:10.1037/0033-295x.103.4.650 · Cited by 3,504 · see citing works
Self-citationAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709 · see citing works
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061 · see citing works
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,970 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Machine learning for medical diagnosis: history, state of the art and perspective
Kononenko, I. (2001) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(01)00077-x · Cited by 1,664 · see citing works
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1× · see citing works
“tudies show that data from high reputation Mechanical Turk workers are comparable to data from commercial panels and university pools when assessing factors such as attentiveness, honesty, and effort [11, 13, 16, 53, 59]. (a) Clear, two-feature condition (CLEAR-2). (b) Black-box, two-feature condition (BB-2). (c) Clea…”
BackgroundClinical versus mechanical prediction: a meta-analysis
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000) · Psychological Assessment · doi:10.1037/1040-3590.12.1.19 · Cited by 1,640 · see citing works
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · see citing works
The algorithmic imaginary: exploring the ordinary affects of Facebook algorithms
Bucher, T. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1154086 · Cited by 1,420 · see citing works
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · see citing works
Bennett, J. R., & Lanning, S. (2007) · Cited by 1,118 · see citing works
Management Misinformation Systems
Ackoff, R. L. (1967) · Management Science · doi:10.1287/mnsc.14.4.b147 · Cited by 1,087 · see citing works
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061 · see citing works
Statistical Classification Methods in Consumer Credit Scoring: A Review
Hand, D. J., & Henley, W. E. (1997) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.1997.00078.x · Cited by 1,042 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040 · see citing works
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · see citing works
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · see citing works
WHY SUMMARIES OF RESEARCH ON PSYCHOLOGICAL THEORIES ARE OFTEN UNINTERPRETABLE
Meehl, P. E. (1990) · Psychological Reports · doi:10.2466/pr0.1990.66.1.195 · Cited by 901 · see citing works
Models and Managers: The Concept of a Decision Calculus
Little, J. D. C. (2004) · Management Science · doi:10.1287/mnsc.1040.0267 · Cited by 808 · see citing works
The cost structure of sensemaking
Russell, D. M., Stefik, M., Pirolli, P. L. T., & Card, S. K. (1993) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/169059.169209 · Cited by 738 · see citing works
Contextual cueing of visual attention
Chun, M. M. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01476-5 · Cited by 732 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · see citing works
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · see citing works
Receiving other people’s advice: Influence and benefit
Yaniv, I. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2003.08.002 · Cited by 652 · see citing works
Effects of Quality and Quantity of Information on Decision Effectiveness
Keller, K. L., & Staelin, R. (1987) · Journal of Consumer Research · doi:10.1086/209106 · Cited by 634 · see citing works
Accurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564 · see citing works
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
Self-citation<i>Why and why not</i> explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · see citing works
Intelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547 · see citing works
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · see citing works
Perspectives on Information Overload
Jacoby, J. (1984) · Journal of Consumer Research · doi:10.1086/208981 · Cited by 501 · see citing works
Actionable Recourse in Linear Classification
Ustun, B., Spangher, A., & Liu, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287566 · Cited by 501 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 4× · see citing works
“[106] Michael Veale, Max Van Kleek, and Reuben Binns.”
BackgroundInfluentialDatabase Models and Managerial Intuition: 50% Model + 50% Manager
Blattberg, R. C., & Hoch, S. J. (1990) · Management Science · doi:10.1287/mnsc.36.8.887 · Cited by 473 · see citing works
Intelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471 · see citing works
Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · see citing works
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · see citing works
Explanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388 · see citing works
The Perceived Utility of Human and Automated Aids in a Visual Detection Task
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024494856 · Cited by 386 · see citing works
Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed
Rader, E., & Gray, R. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702174 · Cited by 386 · cited 1× · see citing works
“ent [33], judicial sentencing and bail [3, 14, 35, 37], and hiring [43]. In addition, these models influence decisions about what news people see on social media platforms (e.g., Twitter and Facebook, [2, 10, 48, 61]), provide recommendations for what movies to watch (e.g., Netflix, [7]) or what music to listen to on on…”
BackgroundAn Experimental Comparison of Different Approaches to Determining Weights in Additive Utility Models
Schoemaker, P. J. H., & Waid, C. C. (1982) · Management Science · doi:10.1287/mnsc.28.2.182 · Cited by 381 · see citing works
Explaining Recommendations: Design and Evaluation
Tintarev, N., & Masthoff, J. (2015) · Springer eBooks · doi:10.1007/978-1-4899-7637-6_10 · Cited by 344 · see citing works
The relative influence of advice from human experts and statistical methods on forecast adjustments
Önkal, D., Goodwin, P., Thomson, M. E., Gonul, M. S., & Pollock, A. C. (2009) · Journal of Behavioral Decision Making · doi:10.1002/bdm.637 · Cited by 331 · see citing works
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330 · see citing works
InterpretML: A Unified Framework for Machine Learning Interpretability
Nori, H., Jenkins, S., Koch, P., & Caruana, R. (2019) · arXiv · doi:10.48550/arxiv.1909.09223 · Cited by 322 · see citing works
Supersparse linear integer models for optimized medical scoring systems
Ustun, B., & Rudin, C. D. (2015) · Machine Learning · doi:10.1007/s10994-015-5528-6 · Cited by 316 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · see citing works
Explaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286 · see citing works
Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M., & Díaz, F. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3272027 · Cited by 280 · see citing works
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277 · see citing works
Effects of task difficulty on use of advice
Francesca Gino, B., & Moore, D. A. (2006) · Journal of Behavioral Decision Making · doi:10.1002/bdm.539 · Cited by 274 · see citing works
Faithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314229 · Cited by 261 · see citing works
psiTurk: An open-source framework for conducting replicable behavioral experiments online
Gureckis, T. M., Martin, J. B., McDonnell, J. V., Rich, A. S., Markant, D. B., Coenen, A., Halpern, D., Hamrick, J. B., & Chan, P. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0642-8 · Cited by 256 · see citing works
The challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253 · see citing works
Promberger, M., & Baron, J. (2006) · Journal of Behavioral Decision Making · doi:10.1002/bdm.542 · Cited by 242 · see citing works
Hohman, F., Head, A., Caruana, R., DeLine, R. A., & Drucker, S. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225 · see citing works
Miller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · see citing works
Efficient Search for Diverse Coherent Explanations
Russell, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287569 · Cited by 218 · see citing works
Toward establishing trust in adaptive agents
Glass, A., McGuinness, D. L., & Wolverton, M. J. (2008) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1378773.1378804 · Cited by 217 · see citing works
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184 · see citing works
Communicating Algorithmic Process in Online Behavioral Advertising
Eslami, M., Krishna Kumaran, S. R., Sandvig, C., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174006 · Cited by 171 · see citing works
(2015) · International Conference on Artificial Intelligence and Statistics · Cited by 162 · see citing works
Persuasiveness of expert systems
Dijkstra, J. J., Liebrand, W. B. G., & Timminga, E. (1998) · Behaviour & Information Technology · doi:10.1080/014492998119526 · Cited by 161 · see citing works
Goldstein, D. G., & Gigerenzer, G. (2009) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2009.05.010 · Cited by 161 · see citing works
Self-citationÅstebro, T. B., & Elhedhli, S. (2006) · Management Science · doi:10.1287/mnsc.1050.0468 · Cited by 142 · see citing works
Introduction to this Special Issue on Sensemaking
Pirolli, P. L. T., & Russell, D. M. (2011) · Human-Computer Interaction · doi:10.1080/07370024.2011.556557 · Cited by 139 · see citing works
Towards Algorithmic Experience
Alvarado, O., & Wærn, A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173860 · Cited by 139 · see citing works
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 127 · see citing works
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278725 · Cited by 127 · see citing works
User agreement with incorrect expert system advice
Dijkstra, J. J. (1999) · Behaviour and Information Technology · doi:10.1080/014492999118832 · Cited by 124 · see citing works
Interpretable and Explorable Approximations of Black Box ModelsPreprint
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2017) · arXiv (Cornell University) · Cited by 123 · see citing works
Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice
Rudin, C. D., & Ustun, B. (2018) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.2018.0957 · Cited by 114 · see citing works
Human Evaluation of Models Built for Interpretability
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S. J., & Doshi‐Velez, F. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5280 · Cited by 114 · see citing works
Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113 · see citing works
Kim, Y., Reinecke, K., & Hullman, J. R. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025592 · Cited by 105 · see citing works
A Semantic Approach for Estimating Consumer Content Preferences from Online Search Queries
Liu, J., & Toubia, O. (2018) · Marketing Science · doi:10.1287/mksc.2018.1112 · Cited by 102 · see citing works
Norms of Distributive Justice in Interest Arbitration
Bazerman, M. H. (1985) · Industrial and Labor Relations Review · doi:10.1177/001979398503800403 · Cited by 101 · see citing works
Liem, C. C. S., Langer, M., Demetriou, A. M., Hiemstra, A. M. F., Wicaksana, A. S., Born, M. P., & König, C. J. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-98131-4_9 · Cited by 101 · see citing works
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96 · see citing works
Simple Rules for Complex DecisionsPreprint
Jung, J., Concannon, C., Shroff, R., Goel, S., & Goldstein, D. G. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2919024 · Cited by 91 · see citing works
Self-citationAssessing long‐term fire risk at local scale by means of decision tree technique
Amatulli, G., Rodrigues, M. J., Trombetti, M., & Lovreglio, R. (2006) · Journal of Geophysical Research Atmospheres · doi:10.1029/2005jg000133 · Cited by 89 · see citing works
A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations
Krause, J., Dasgupta, A., Swartz, J. L., Aphinyanaphongs, Y., & Bertini, E. (2017) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585720 · Cited by 87 · see citing works
Design of an intelligible mobile context-aware application
Lim, B. Y., & Dey, A. (2011) · Mobile HCI · doi:10.1145/2037373.2037399 · Cited by 72 · see citing works
The Illusion of Control: Placebo Effects of Control Settings
Vaccaro, K., Huang, D., Eslami, M., Sandvig, C., Hamilton, K., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173590 · Cited by 65 · see citing works
Algorithm Appreciation: People Prefer Algorithmic To Human JudgmentPreprint
Logg, J. M., Minson, J., & Moore, D. A. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2941774 · Cited by 59 · see citing works
Visualizable and interpretable regression models with good prediction power
Kim, H., Loh, W., Shih, Y., & Chaudhuri, P. (2007) · IIE Transactions · doi:10.1080/07408170600897502 · Cited by 52 · see citing works
Effects of Sensemaking Translucence on Distributed Collaborative Analysis
Goyal, N., & Fussell, S. R. (2016) · ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW) · doi:10.1145/2818048.2820071 · Cited by 52 · see citing works
A Mathematical Model for Interpretable Clinical Decision Support with Applications in Gynecology
Belle, V. V., CALSTER, B. V., Timmerman, D., Bourne, T., Bottomley, C., Valentin, L., Neven, P., Huffel, S. V., Suykens, J. A. K., & Boyd, S. (2012) · PLoS ONE · doi:10.1371/journal.pone.0034312 · Cited by 42 · see citing works
Kleinberg, J. M., & Mullainathan, S. (2019) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3328526.3329621 · Cited by 36 · see citing works
Effects of Sensemaking Translucence on Distributed Collaborative AnalysisPreprint
Goyal, N., & Fussell, S. R. (2016) · Cited by 35 · see citing works
Human-in-the-Loop Interpretability Prior
Lage, I. (2024) · PubMed · doi:10.57702/j7gpfk6v · Cited by 31 · see citing works
The Earth is spherical (p < 0.05): alternative methods of statistical inference
Vicente, K. J., & Torenvliet, G. (2000) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110037065 · Cited by 28 · see citing works
Using Human and Machine Processing in Recommendation Systems
Colson, E. (2013) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v1i1.13100 · Cited by 10 · see citing works
Flagged first, then by citations
2 of the 116 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [44]44. Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M Drucker. 2019. Gamut: A design probe to understand how data scientists understand machine learning models. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-13.
- [47]47. Philip Johnson-Laird. 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge University Press.
- [48]48. Jongbin Jung, Connor Concannon, Ravi Shroff, Sharad Goel, and Daniel G Goldstein. 2020. Simple rules to guide expert classifications. Journal of the Royal Statistical Society: Series A (Statistics in Society) (2020).
- [82]82. Don A. Norman. 1987. Some Observations on Mental Models. In Human-Computer Interaction: A Multidisciplinary Approach, R. M. Baecker and W. A. S. Buxton (Eds.). Morgan Kaufmann Publishers Inc., 241-244.
Cited by
Sorted by citations
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · see citing works
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083 · see citing works
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008 · see citing works
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00165-5 · Cited by 882 · see citing works
“The next observation of note is that few of the available tools surveyed provide meaningful ways to assess, and respond to, the impact that the data-processing involved in their ML algorithm has on an individual, and even less on the impact on society as a whole (Poursabzi-Sangdeh et al. 2018).”
BackgroundMarkus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · see citing works
“Their models’ predictions resulted in lowest error (See Figure 6 in [55]).”
BackgroundMethodInfluentialInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · see citing works
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
“, user satisfaction, mental models, and trust [144, 155, 267].”
BackgroundNotions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · see citing works
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · see citing works
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · see citing works
Self-citationBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · see citing works
Transparency and trust in artificial intelligence systems
Schmidt, P., Biessmann, F., & Teubner, T. (2020) · Journal of Decision Systems · doi:10.1080/12460125.2020.1819094 · Cited by 440 · see citing works
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406 · see citing works
Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
Holstein, K., McLaren, B. M., & Aleven, V. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.62.3 · Cited by 429 · see citing works
AI in Health: State of the Art, Challenges, and Future Directions
Wang, F., & Preininger, A. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677908 · Cited by 398 · see citing works
Towards Automatic Concept-based Explanations
Ghorbani, A., Wexler, J., Zou, J., & Kim, B. (2019) · arXiv · doi:10.48550/arxiv.1902.03129 · Cited by 378 · see citing works
Explanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355 · see citing works
Challenges of Real-World Reinforcement LearningPreprint
Dulac-Arnold, G., Mankowitz, D. J., & Hester, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.12901 · Cited by 437 · see citing works
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318 · see citing works
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 326 · see citing works
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · see citing works
“This is already the case in regulated domains such as finance [78].”
BackgroundDisparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · see citing works
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278 · see citing works
The flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251 · see citing works
“Explainability also enables users to verify that the system does not rely on artifacts or noise in the training data, and can be used to assess whether the system is fair in its decisions, particularly when training data may include a biased or incomplete picture of the population [36].”
BackgroundInfluentialAbbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236 · see citing works
“Others emphasized the need for adjustable granularity of explanations i.e., providing both overview and drill-down levels of insight [51,125,126].”
Interpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9780367816377-16 · Cited by 271 · see citing works
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228 · see citing works
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · see citing works
Hohman, F., Head, A., Caruana, R., DeLine, R. A., & Drucker, S. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225 · see citing works
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219 · see citing works
The Disagreement Problem in Explainable Machine Learning: A Practitioner's PerspectivePreprint
Krishna, S., Han, T., Gu, A., Jabbari, S., Wu, Z. S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2963888/v1 · Cited by 279 · see citing works
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217 · see citing works
A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Bharati, S., Mondal, M. R. H., & Podder, P. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266418 · Cited by 235 · see citing works
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208 · see citing works
“recommendation, especially in cases where the AI advice is wrong [6, 57, 63].”
BackgroundA Survey of Explainable Artificial Intelligence for Smart Cities
Javed, A. R., Ahmed, W., Pandya, S., Maddikunta, P. K. R., Alazab, M., & Gadekallu, T. R. (2023) · Electronics · doi:10.3390/electronics12041020 · Cited by 208 · see citing works
Evaluating and Aggregating Feature-based Model Explanations
Bhatt, U., Weller, A., & Moura, J. M. F. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/417 · Cited by 208 · see citing works
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · see citing works
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 196 · see citing works
A literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180 · see citing works
Adversarial attacks and defenses in explainable artificial intelligence: A survey
Baniecki, H., & Biecek, P. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102303 · Cited by 204 · see citing works
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216 · see citing works
Should AI models be explainable to clinicians?
Abgrall, G., Holder, A. L., Chelly Dagdia, Z., Zeitouni, K., & Monnet, X. (2024) · Critical Care · doi:10.1186/s13054-024-05005-y · Cited by 175 · see citing works
Explainability scenarios: towards scenario-based XAI design
Wolf, C. T. (2019) · International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302317 · Cited by 153 · see citing works
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172 · see citing works
Mathew, D. E., Ebem, D. U., Ikegwu, A. C., Ukeoma, P. E., & Dibiaezue, N. F. (2025) · Neural Processing Letters · doi:10.1007/s11063-025-11732-2 · Cited by 168 · see citing works
Keane, M. T., Kenny, E. M., Delaney, E. D., & Smyth, B. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/609 · Cited by 169 · see citing works
Designing for human–AI complementarity in K‐12 education
Holstein, K., & Aleven, V. (2022) · AI Magazine · doi:10.1002/aaai.12058 · Cited by 156 · see citing works
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · see citing works
Materials Acceleration Platforms: On the way to autonomous experimentation
Flores‐Leonar, M. M., Mejía-Mendoza, L. M., Aguilar‐Granda, A., Sánchez-Lengeling, B., Tribukait, H., Amador‐Bedolla, C., & Aspuru‐Guzik, A. (2020) · Current Opinion in Green and Sustainable Chemistry · doi:10.1016/j.cogsc.2020.100370 · Cited by 150 · see citing works
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
Wischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146 · see citing works
“Poursabzi-Sangdeh et al. [108] found that providing users with more information (through increased interpretability and transparency) decreased the user’s capabilities to detect and correct mistakes by the system.”
BackgroundHuman-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · see citing works
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 145 · see citing works
“decision tasks with different data types including text data [4, 49, 54], tabular data [61, 73, 78], and image data [10, 45].”
BackgroundResultInfluentialUnderstanding artificial intelligence ethics and safetyPreprint
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 268 · see citing works
It's complicated: the relationship between user trust, model accuracy and explanations in AI
Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3495013 · Cited by 144 · see citing works
Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · see citing works
You, S., Yang, C., & Li, X. (2022) · Journal of Management Information Systems · doi:10.1080/07421222.2022.2063553 · Cited by 142 · see citing works
A multi-objective approach for profit-driven feature selection in credit scoring
Kozodoi, N., Lessmann, S., Papakonstantinou, K., Gatsoulis, Y., & Baesens, B. (2019) · Decision Support Systems · doi:10.1016/j.dss.2019.03.011 · Cited by 141 · see citing works
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143 · see citing works
Kliegr, T., Bahník, Š., & Fürnkranz, J. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103458 · Cited by 139 · see citing works
Contemporary Symbolic Regression Methods and their Relative Performance
Cava, W. L., Orzechowski, P., Burlacu, B., de França, F. O., Virgolin, M., Jin, Y., Kommenda, M., & Moore, J. H. (2021) · PubMed · doi:10.48550/arxiv.2107.14351 · Cited by 137 · see citing works
Four Principles of Explainable Artificial Intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Broniatowski, D. A., & Przybocki, M. A. (2020) · doi:10.6028/nist.ir.8312-draft · Cited by 136 · see citing works
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132 · see citing works
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · see citing works
Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research
Vaughan, J. W. (2018) · Journal of Machine Learning Research · Cited by 130 · see citing works
Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”
Shashikumar, S. P., Wardi, G., Malhotra, A., & Nemati, S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00504-6 · Cited by 130 · see citing works
How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129 · see citing works
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 127 · see citing works
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2021) · Philosophical studies series · doi:10.1007/978-3-030-81907-1_10 · Cited by 178 · see citing works
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126 · see citing works
An Evaluation of the Human-Interpretability of Explanation
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S., & Doshi-Velez, F. (2019) · arXiv · doi:10.48550/arxiv.1902.00006 · Cited by 125 · see citing works
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124 · see citing works
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120 · see citing works
Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462131 · Cited by 119 · see citing works
Human Evaluation of Models Built for Interpretability
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S. J., & Doshi‐Velez, F. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5280 · Cited by 114 · see citing works
Smith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112 · see citing works
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · see citing works
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111 · see citing works
Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging
Fogliato, R., Chappidi, S., Lungren, M. P., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K. M., & Nushi, B. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533193 · Cited by 115 · see citing works
Does Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16819 · Cited by 106 · see citing works
Silva, A., Schrum, M., Hedlund-Botti, E., Gopalan, N., & Gombolay, M. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2101698 · Cited by 104 · see citing works
Effect of AI Explanations on Human Perceptions of Patient-Facing AI-Powered Healthcare Systems
Zhang, Z., Genc, Y., Wang, D., Ahsen, M. E., & Fan, X. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01743-6 · Cited by 108 · see citing works
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103 · see citing works
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101 · see citing works
Building more accurate decision trees with the additive tree
Luna, J. M., Gennatas, E. D., Ungar, L., Eaton, E., Diffenderfer, E. S., Jensen, S. T., Simone, C. B., Friedman, J. H., Solberg, T. D., & Valdés, G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1816748116 · Cited by 101 · see citing works
Green learning: Introduction, examples and outlook
Kuo, C. ‐. J., & Madni, A. M. (2022) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2022.103685 · Cited by 104 · see citing works
Rechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97 · see citing works
Explainable Artificial Intelligence Approaches: A SurveyPreprint
Islam, S. R., Eberle, W., Ghafoor, S., & Ahmed, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09429 · Cited by 98 · see citing works
AI in medical diagnosis: AI prediction & human judgment
Göndöcs, D., & Dörfler, V. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102769 · Cited by 99 · see citing works
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
Topic modeling in software engineering research
Silva, C. C., Galster, M., & Gilson, F. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10026-0 · Cited by 90 · see citing works
Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645199 · Cited by 95 · see citing works
Counterfactual Explanations for Multivariate Time Series
Ates, E., Aksar, B., Leung, V. J., & Coskun, A. K. (2021) · 2021 International Conference on Applied Artificial Intelligence (ICAPAI) · doi:10.1109/icapai49758.2021.9462056 · Cited by 88 · see citing works
- Authors
- Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H.
- Affiliations
- Microsoft (United States) (US)
- Microsoft Research, United States
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-52
- OpenAlex
- W2788362053
- Semantic Scholar
- 2f4e42ab1368b83e38e3011fcec4edb57267b5d2
- Cited by
- 728
- References
- 116
- Citations by year
- 2026: 94 · 2025: 97 · 2024: 118 · 2023: 106 · 2022: 111 · 2021: 90 · 2020: 61 · 2019: 42 · 2018: 9
- FWCI
- 48.67
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, black-box models, model transparency, feature count, error detection, information overload, trust in machine learning, pre-registered experiment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
