Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Open accessThe DOI matches its Crossref record; 98 of the 101 references it cites were found in a registry or an index; 14 of the 28 sources its claims cite were checked for support.
Abstract
Many researchers motivate explainable AI with studies showing that human-AI team performance on decision-making tasks improves when the AI explains its recommendations. However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team. Can explanations help lead to complementary performance, where team accuracy is higher than either the human or the AI working solo? We conduct mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task (explaining itself in some conditions). While we observed complementary improvements from AI augmentation, they were not increased by explanations. Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness. Our result poses new challenges for human-centered AI: Can we develop explanatory approaches that encourage appropriate trust in AI, and therefore help generate (or improve) complementary performance?
Overview
CoChat Summary
TL;DR
This paper tests whether AI explanations produce complementary human-AI team performance when AI accuracy is comparable to human accuracy. Across three crowdsourced tasks, teams outperformed solo humans and solo AI, but explanations did not significantly improve team accuracy over simply showing AI confidence, and they increased acceptance of AI recommendations regardless of correctness.
Key findings
- Complementary performance occurred in every human-AI teaming condition. For Beer, Team (Conf) reached μ = 0.89 ± 0.05, surpassing AI alone (0.84) and unassisted humans (0.82 ± 0.09); Amzbook and LSAT showed relative gains of 2.2% and 20.1% over unassisted workers.
- Explanations did not significantly improve team accuracy over the confidence-only baseline. For Beer, Team (Conf) was 0.89 ± 0.05 vs. Team (Explain-Top-1, AI) 0.88 ± 0.06 (z = -1.18, p = .24); similar non-significant patterns appeared for Amzbook (p = .22) and LSAT (p = .64).
- Explain-Top-1 and Explain-Top-2 did not differ significantly: Beer z = 0.85, p = .40; Amzbook z = 0.81, p = .42; LSAT z = 0.42, p = .68.
- Explanations increased reliance on AI recommendations even when incorrect. Splitting by AI correctness, Explain-Top-1 improved accuracy when AI was correct but worsened it when AI was incorrect; Adaptive explanations reduced blind trust in low-confidence cases but did not significantly improve final team accuracy.
Methods
- Mixed-method user studies on three tasks: sentiment analysis of Beer and Amzbook reviews and LSAT logical-reasoning questions. Crowdworkers were recruited from Amazon Mechanical Turk; 566 (Beer), 552 (Amzbook), and 508 (LSAT) participated, with about 100 participants per condition and 1,626 users total across tasks.
- AI systems were RoBERTa-based classifiers. Sentiment explanations used LIME saliency highlights; LSAT used expert-generated narrative explanations. Confidence was calibrated with isotonic regression; Adaptive explanations switched between top-1 and top-2 based on median confidence thresholds of 89.2% (Beer) and 88.9% (Amzbook).
- Main analysis compared team decision accuracy across conditions using Student’s T-tests with Bonferroni correction. Qualitative analysis coded participants’ self-reported collaboration strategies, with average Cohen’s κ = 0.71 and average agreement 93% ± 6.5%.
Limitations
- The paper states that it evaluated only decision accuracy, not other possible benefits such as speed or multi-dimensional complementarity; it used crowdsourceable tasks and may not generalize to high-stakes expert domains; it explored only two explanation presentation formats; and it did not investigate user expertise.
- Evident from the text: LSAT used only 20 examples and exclusively expert explanations; the Adaptive threshold was set by median confidence rather than learned; AI recommendations were shown before participants solved the task, which the authors note may invoke anchoring; and subjective survey ratings were analyzed post-hoc without claims of statistical significance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 3 Not valid · 11 Not checked · 82 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“displaying uncertainty in predictions can help users can make more optimal decisions [19, 22, 25, 37, 60]”
What the sources say
- Not checked — no full text of the sourcecites [19] [19] Xiao Dong and Caroline C Hayes. 2012. Uncertainty visualizations: Helping decision makers become more aware of uncertainty and its implications. Journal of Cognitive Engineering and Decision Making 6, 1 (2012), 30-56. - source
- Not checked — no full text of the sourcecites [22] [22] Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, and Matthew Kay. 2018. Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18) . Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/3173574.3173718 - source
- Supportedcites [25] [25] Dimitra Gkatzia, Oliver Lemon, and Verena Rieser. 2016. Natural Language Generation enhances human decision-making with uncertain information. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . Association for Computational Linguistics, Berlin, Germany, 264-268. https://doi.org/10.18653/v1/P16-2043 - source
The Abstract (p. 1) reports that presenting uncertain weather-forecast information through natural language improved decision-making compared with a graphical presentation. The Results (p. 3) measure that improvement using players’ monetary gains. This supports the hedged claim that displaying uncertainty can help users make better decisions. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“most ML models can generate confidence scores that, in practice, correlate with the model's true likelihood to err [32]”
What the sources say
- Partly supportedcites [32] [32] Dan Hendrycks and Kevin Gimpel. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In 5th International Conference on Learning Representations, ICLR 2017 . OpenReview.net, Toulon, France, 1-12. https://openreview.net/forum?id=Hkg4TI9xl - sourceExperimental — accuracy not yet measured
The Introduction (pp. 1–2) and Table 1 (p. 4) show that softmax-derived scores can help distinguish correct from incorrect predictions in the neural classifiers tested. The paper does not establish this for most ML models, and it explicitly cautions that softmax prediction probabilities have poor direct correspondence to confidence when viewed in isolation. (LLM, conf=0.88)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, it is worth noting that this increased preference did not translate to an improvement in team performance, which is consistent with observations made by Buçinca et al. [12] that show that people may prefer one explanation but make better decisions with another.”
What the sources say
- Supportedcites Buçinca et al. [12] [12] Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI '20) . Association for Computing Machinery, New York, NY, USA, 454-464. https://doi.org/10.1145/3377325.3377498 - source
The paper reports that participants preferred deductive explanations in the actual decision-making task (ACM Reference format, p. 2), but were more accurate with inductive explanations when the AI gave an incorrect recommendation (Section 4.2, p. 7). That supports the attributed observation that people may prefer one explanation while making better decisions with another; the performance advantage was conditional, not an overall difference. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“which is consistent with observations made by Buçinca et al. [12] that show that people may prefer one explanation but make better decisions with another.”
What the sources say
- Supportedcites Buçinca et al. [12] [12] Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI '20) . Association for Computing Machinery, New York, NY, USA, 454-464. https://doi.org/10.1145/3377325.3377498 - source
The Conclusion (p. 10) says participants preferred deductive explanations but recognized AI errors better with inductive explanations. The Discussion (p. 9) adds that, when the AI gave an incorrect recommendation in the actual decision-making task, participants answered correctly more often with inductive explanations. This supports the hedged claim that preference and decision performance can diverge. (LLM, conf=0.92)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“having the ML produce the maximally convincing explanation - a common objective shared in the design of many AI explanation algorithms - might be a poor choice for complementary performance [12].”
What the sources say
- Partly supportedcites [12] [12] Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI '20) . Association for Computing Machinery, New York, NY, USA, 454-464. https://doi.org/10.1145/3377325.3377498 - sourceExperimental — accuracy not yet measured
The Conclusion (p. 10) reports that participants trusted and preferred deductive explanations more but recognized AI errors better with inductive explanations. This supports the warning that a more convincing explanation may not improve human–AI performance. The paper does not establish that producing maximally convincing explanations is a common objective of many explanation algorithms. (LLM, conf=0.84)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
44 more claims · 1 with their cited sources checked
Create a free account to see all 107 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 37
- It may state two separate claims as one. 33
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Our exploratory quantitative analysis shows that, consistent with prior work [4,82,87], featurebased explanations did not improve participants' decision outcomes compared to their performance without AI support, and led to overreliance on incorrect AI predictions.”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [4]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Explanations have been shown to potentially increase people's reliance on an AI, whether it is right or wrong, leading to worse team performance than a human or AI alone [5, 26, 63, 66].”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as [5]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“in contrast to prior work which has evaluated reliance in action space only [3, 28, 35].”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [3]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recently, trust calibration became an essential metric for evaluating collaborative Human-AI decision-making tools [7, 16, 96].”
From Explainable recommendation: when design meets trust calibration (2021) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“in AI-assisted decision-making settings, communicating (un)certainty has been shown to support trust calibration [118], increase vigilance [85], and improve task performance [7].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [7]
Experimental — accuracy not yet measured
60 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“𝑧 =0.427, 𝑝 =.64”
Reported p = .64 · recomputed p = .669
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upz = 0.427reported p = .64 · recomputed p = .669
“z =0.427, p = .64”
- Consistentz = 0.85reported p = .4 · recomputed p = .395
“z =0.85, p = .40”
- Consistentz = 0.81reported p = .42 · recomputed p = .418
“z =0.81, p = .42”
- Consistentz = 0.42reported p = .68 · recomputed p = .674
“z =0.42, p = .68”
- Consistentz = -1.18reported p = .24 · recomputed p = .238
“z =-1.18, p = .24”
6 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3 Study Procedure · p. 7
“508 crowd workers participated in our study, 35% of whom passed the screening and completed the main task, resulting in a total of 100 participants per condition.”
§ 4.1 Hypotheses, Conditions, and Interface · p. 5
“For both domains, we ran two baseline conditions: unassisted users ( Human ), as well as a simple AI assistance that shows the AI's recommendation and confidence but no explanation ( Team (Conf) ).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Updated Strategy | 'I decided myself after seeing that sometimes Marvin failed me. ' | 12 (2%) |”
Reported 2 · recomputed 2.933985
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Hypotheses, Conditions, and Interface · p. 5
“Figure 2 shows an example UI for sentiment classification for Team (Adaptive, AI).”
§ 4.2 AI Model, Study Samples and Explanations · p. 6
“Figure 2: A screenshot of the Team (Adaptive, Expert) condition for the Amzbook reviews dataset.”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 3×
“Explanations can be useful in many scenarios where a human and AI interact: transparently communicating model predictions [10, 21, 38, 40, 66], teaching humans tasks like translation [3, 26] or content moderation [36], augmenting human analysis procedure [36] or creativity [17], legal imperatives [57, 77], etc.”
BackgroundMethodSelf-citationRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 4×
“In short, the use of explanations increased the human’s blind trust in the AI system, rather than appropriate reliance [39], making them more likely to adopt its recommendation both when the AI was right and when it was wrong.”
BackgroundResultInfluentialLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“We chose text classification because it is a popular task in natural language processing (NLP) that has been used in several previous studies on human-AI teaming [17, 24, 35, 43, 53, 70], requires little domain expertise, and is thus amenable to crowdsourcing.”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · cited 1×
“AI explanations are useful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others.”
BackgroundAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 1×
“…tried three existing post-hoc explainers to explain the RoBERTa model: LIME trains a new intelligible model to provide a local approximation of a model’s behavior[56], SimpleGrad computes gradients with respect to input, and IntergratedGrad uses an approach that aggregates multiple gradients [59].”
MethodTheory of Games and Economic Behavior
N., E., von Neumann, J., & Morgenstern, O. (1945) · The Journal of Philosophy · doi:10.2307/2019327 · Cited by 4,006
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · cited 1×
“Despite offering accurate predictions, RoBERTa generated poorly calibrated confidence scores, a common issue with neural networks [22], which we mitigate with posthoc calibration (isotonic regression [4]) on the validation set.”
MethodModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“asks and advise human decision makers on higher stake decisions, such as medical diagnosis [11, 31], wildlife protection [19], improving housing for homeless youth [12, 29], and recidivism prediction [23, 25, 60]. While there are many motivations for creating these human-AI teams, a primary one is increasing the effec…”
BackgroundAttention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450 · cited 2×
“ghlights, where the interface highlights spans of text that are most salient to the model’s prediction directly in the input text, so the user need not go back and forth between input and explanation [18, 36, 42, 62]. Figure2shows one example beer review used in our study. We also experimented with a more challenging…”
BackgroundMethodInfluentialA Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1610.02136 · Cited by 1,968 · cited 1×
“Relying on commonly seen correlations between mistakes and low-confidence [26], we developed a new, dynamic strategy, adaptive explanations, that switches between Explain-Top-1 and Explain-Top-2 depending on the AI’s confidence.”
BackgroundExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967 · cited 4×
“Indeed, at first glance, it appears that researchers have already confirmed the utility of explanations on tasks ranging from medical diagnosis [10, 47] and data annotation [57] to deception detection [36].”
BackgroundInfluentialA Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743 · cited 2×
“Finally, we only explored a small fraction of possible ways to present explanations: highlighting key words [18, 36, 42, 62] and natural language arguments.”
BackgroundMethodIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 2×
“EARCH Increasingly, we see organizations incorporating AI systems into the work process to automate routine tasks and advise human decision makers on higher stake decisions, such as medical diagnosis [11, 31], wildlife protection [19], improving housing for homeless youth [12, 29], and recidivism prediction [23, 25, 6…”
BackgroundUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 1×
“AI explanations are useful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others.”
BackgroundPrinciples of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
He, R., & McAuley, J. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883037 · Cited by 1,459 · cited 1×
“Specifcally, we selected two sentiment analysis datasets to improve the generalization of our results: beer reviews [56] and book reviews [31].”
MethodDesigning Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · cited 4×
“’s suggestion to mitigate heuristic biases by explaining “multiple outcomes” [74].”
BackgroundMethodResultInfluentialZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · cited 3×
“Income category [83] Accuracy ↑ 65% 75% 73% ✗”
BackgroundMethodRationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 1×
“…covered existing NLP models for generating explanations along with predictions [43, 53]: an intelligible model ( Logistic Regression classifier), a encoder-decoder rationale generator (Lei et al. [40]), and a recent contextual embedding-based model ( RoBERTa with linear classification layer).”
BackgroundA dynamical systems perspective on agent-environment interaction
Beer, R. D. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00005-l · Cited by 755 · cited 1×
“hms for classification, whose practical benefits for team performance remain unclear. Specifically, we selected two sentiment analysis datasets where specialized knowledge is not needed: beer reviews [6] and book reviews [50]. Explanations in this task are represented by saliency scores, where each input feature (i.e.…”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 5×
“Their models’ predictions resulted in lowest error (See Figure 6 in [55]).”
BackgroundMethodInfluentialManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 1×
“n natural language (Section3.5). Admittedly, explanations can also be presented in other ways. For structured data, most studies present the salient features, sometimes paired with importance weights [8, 21, 47, 52, 63]. On vision tasks, some work takes the demonstrate-by-example approach, i.e., by explaining an examp…”
MethodExplainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · cited 1×
“s, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7,30]. , Vol. 1, No. 1, Article . Publication date: July 2020. 4 G. Bansal et al. Domain Task Performance Metric Human…”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“Indeed, at first glance, it appears that researchers have already confirmed the utility of explanations on tasks ranging from medical diagnosis [10, 47] and data annotation [57] to deception detection [36].”
BackgroundTruth-Default Theory (TDT): A Theory of Human Deception and Deception Detection
Levine, T. R. (2014) · Journal of Language and Social Psychology · doi:10.1177/0261927x14535916 · Cited by 585 · cited 1×
“” These user behaviors are similar to observations in psychology literature on Truth-Default Theory [41], which shows that people exhibit truth-default behavior: by default people are biased to”
ResultProceedings of the SIGCHI Conference on Human Factors in Computing Systems
(2009) · doi:10.1145/1518701 · Cited by 570
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 2×
“As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7, 30].”
BackgroundExplanation, imagination, and confidence in judgment.
Koehler, D. J. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.110.3.499 · Cited by 561 · cited 3×
“This is consistent with psychology literature [32], which has shown that human explanations cause listeners to agree even when the explanation is wrong.”
ResultInfluentialEnglich, B., Mussweiler, T., & Strack, F. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167205282152 · Cited by 554 · cited 2×
“This approach risks invoking the anchor effect, studied in Psychology [16] — people rely heavily on the first information that is presented by others when making decisions.”
BackgroundWhy and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 1×
“While we limit the our context to the AI’s confdence, in general, one could rely on more features of the human-AI team, such as the user, location, or time [34, 48].”
MethodClinical cognition and diagnostic error: applications of a dual process model of reasoning
Croskerry, P. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9182-2 · Cited by 494
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“” While this observation empirically shows that end-users develop mental model of trust in AI-assisted decision making [5], it more importantly shows how the AI’s confdence is a simple, yet salient feature via which users create a mental model of the AI’s global behavior [23].”
BackgroundSelf-citationThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 2×
“Examples include AI systems that predict likely hospital readmission to assist doctors with correlated care decisions [8, 13, 15, 78] and AIs that estimate recidivism to help judges decide whether to grant bail to defendants [2, 30].”
BackgroundThe Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 5×
“’s initial answer or only having the AI present an explanation if it disagreed with the human’s choice. We note that these approaches correspond to the Update and Feedback methods of Green & Chen [21], which were effective, albeit not in the complementary zone. However, by delaying display of the AI’s input until…”
BackgroundMethodResultInfluentialSimplicity and probability in causal explanation
LOMBROZO, T. (2007) · Cognitive Psychology · doi:10.1016/j.cogpsych.2006.09.006 · Cited by 384 · cited 1×
“d in the design of many AI explanation algorithms — might be a poor choice for complementary performance. Indeed, while it is known that human explainers and listeners often have different objectives [46], it seems quite problematic to develop AI systems that actively seek to minimize human oversight. A more ideal con…”
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 1×
“ge-scale study provides informative insights that will inform future work on these domains. Finally, we only explored a small fraction of possible ways to present explanations: highlighting key words [18, 36, 42, 62] and natural language arguments. It may be that alternative approaches provide more benefit to team per…”
BackgroundInfluentialExplaining Recommendations: Satisfaction vs. Promotion
Bilgic, M., & Mooney, R. J. (2005) · Intelligent User Interfaces · Cited by 334
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 1×
“ter. For papers with multiple domains or experiments, we took one sample with the most comparable human and AI performance. In this paper, we focus explicitly on AI-assisted decision making scenarios [3], where an AI assistant (e.g., a classification model) makes recommendations to a human (e.g., a judge), who is resp…”
BackgroundSelf-citationCreative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257 · cited 1×
“ful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others. An assumption underlying all these case is that a human will be in the lo…”
BackgroundEvaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Hase, P., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.491 · Cited by 254 · cited 1×
“We chose text classification because it is a popular task in natural language processing (NLP) that has been used in several previous studies on human-AI teaming [17, 24, 35, 43, 53, 70], requires little domain expertise, and is thus amenable to crowdsourcing.”
MethodThe challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253 · cited 3×
“y one is increasing the effectiveness of decision making, while reducing the cost of operations. But increased use of AI brings several well-known dangers as well, including bias and erratic behavior [64]. Many researchers have argued that if AI could explain its reasoning, these , Vol. 1, No. 1, Article . Publication…”
BackgroundInfluentialThe Isotonic Regression Problem and its Dual
Barlow, R. E., & Brunk, H. D. (1972) · Journal of the American Statistical Association · doi:10.1080/01621459.1972.10481216 · Cited by 245 · cited 1×
“Despite offering accurate predictions, RoBERTa generated poorly calibrated confidence scores, a common issue with neural networks [22], which we mitigate with posthoc calibration (isotonic regression [4]) on the validation set. , Vol. 1, No. 1, Article . Publication date: July 2020. Does the Whole Exceed its Parts? Th…”
MethodLearning Attitudes and Attributes from Multi-aspect Reviews
McAuley, J., Leskovec, J., & Jurafsky, D. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.110 · Cited by 245 · cited 1×
“Specifically, we selected two sentiment analysis datasets where specialized knowledge is not needed: beer reviews [6] and book reviews [50].”
MethodJhaver, S., Bruckman, A., & Gilbert, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359252 · Cited by 203 · cited 1×
“ations. 2 BACKGROUND AND RELATED WORK AI explanations are useful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others. An assumptio…”
BackgroundHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 1×
“mes paired with importance weights [8, 21, 47, 52, 63]. On vision tasks, some work takes the demonstrate-by-example approach, i.e., by explaining an example with its closest match in the training set [48, 67]. These two methods are typically less suitable for domains involving unstructured text, as the text features a…”
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184 · cited 1×
“, explanations can lead humans to either follow incorrect AI suggestions or ignore the correct ones [13, 69].”
BackgroundOverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Glassman, E. L., Scott, J., Singh, R., Guo, P. J., & Miller, R. C. (2015) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2699751 · Cited by 177
Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making
Fernandes, M., Walls, L., Munson, S., Hullman, J., & Kay, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173718 · Cited by 175 · cited 1×
“Second, displaying uncertainty in predictions can help users can make more optimal decisions [19, 22, 25, 37, 60].”
BackgroundComparing Automatic and Human Evaluation of Local Explanations for Text Classification
Nguyen, D. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1097 · Cited by 173 · cited 1×
“red no specific background knowledge. We chose text classification because it is a popular task in natural language processing (NLP) that has been used in several previous studies on human-AI teaming [17, 24, 35, 43, 53, 70], requires little domain expertise, and is thus amenable to crowdsourcing.Further, there are va…”
Method'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
(2018) · doi:10.31235/osf.io/9wqxr · Cited by 161
Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156 · cited 1×
“ely to benefit the overall process. As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a bett…”
BackgroundNarayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · cited 1×
“n natural language (Section3.5). Admittedly, explanations can also be presented in other ways. For structured data, most studies present the salient features, sometimes paired with importance weights [8, 21, 47, 52, 63]. On vision tasks, some work takes the demonstrate-by-example approach, i.e., by explaining an examp…”
MethodIncentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146 · cited 1×
“The assigned additional bonuses were intended to motivate workers to strive for performance in the complementary zone and improve over the AIonly performance [33].”
BackgroundIdentifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
Lakkaraju, H., Kamar, E., Caruana, R., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10821 · Cited by 145 · cited 2×
“s: (1) in high confidence regions (circles in Figure5), not only did workers have to agree more, but they also had to identify cases where the model failed with very high confidence (unknown unknowns [37]). Identifying unknown unknowns could have been a difficult and time-consuming task for workers, and they may have…”
BackgroundSelf-citationReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Yu, W., Jiang, Z., Dong, Y., & Feng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04326 · Cited by 127 · cited 2×
“We finetuned a RoBERTa model on ReClor data [69], which contains questions from standardized exams like the LSAT and GMAT.”
BackgroundMethod"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123 · cited 6×
“However, models outperformed every team (see Table 1 and Figure 6 in [35]).”
BackgroundMethodResultInfluentialBayati, M., Braverman, M., Gillam, M., Mack, K. M., Ruiz, G., Smith, M. S., & Horvitz, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109264 · Cited by 122 · cited 1×
“ns, but the people retains agency and have accountability on the final decisions. Examples include AI systems that predict likely hospital-readmission to assist doctors with correlated care decisions [5, 11, 65] and AIs that estimate recidivism to help judges decide whether to grant bail to defendants [1, 25]. In such…”
BackgroundMental Models of AI Agents in a Cooperative Game Setting
Gero, K. I., Ashktorab, Z., Dugan, C., Pan, Q., Johnson, J., Geyer, W., Ruiz, M., Miller, S., Millen, D. R., Campbell, M., et al. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376316 · Cited by 121 · cited 1×
“” While this observation empirically shows that end-users develop mental model of trust in AI-assisted decision making [5], it more importantly shows how the AI’s confdence is a simple, yet salient feature via which users create a mental model of the AI’s global behavior [23].”
BackgroundWilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118 · cited 2×
“ted. This suggests that the benefit of AI assistance may be much more complex than mere accuracy, and that it may be useful to directly optimize for complementary behavior as suggested by recent work [2,66] We hope this paper will serve as a “Call to action” for the HCI and AI communities, as there is much future work…”
BackgroundSelf-citationDecisions With Uncertainty: The Glass Half Full
Joslyn, S., & LeClerc, J. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413481473 · Cited by 109 · cited 1×
“Second, displaying uncertainty in predictions can help users can make more optimal decisions [19, 22, 25, 37, 60].”
BackgroundWhat can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92 · cited 6×
“[17], experts and novices played Quiz Bowl with recommendation from Elastic Search system.”
BackgroundMethodInfluentialInterpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Ghaeini, R., Fern, X., & Tadepalli, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1537 · Cited by 91 · cited 2×
“ghlights, where the interface highlights spans of text that are most salient to the model’s prediction directly in the input text, so the user need not go back and forth between input and explanation [18, 36, 42, 62]. Figure2shows one example beer review used in our study. We also experimented with a more challenging…”
BackgroundMethodInfluentialPredict Responsibly: Improving Fairness and Accuracy by Learning to Defer
Madras, D., Pitassi, T., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1711.06664 · Cited by 90 · cited 1×
“As recent work has suggested [4, 55, 59, 79], it may be useful to directly optimize for complementary behavior by accounting for the human behavior during training, who may have access to a diferent set of features [72].”
BackgroundUncertainty Forecasts Improve Decision Making Among Nonexperts
Nadav-Greenberg, L., & Joslyn, S. L. (2009) · Journal of Cognitive Engineering and Decision Making · doi:10.1518/155534309x474460 · Cited by 87 · cited 1×
“Second, displaying uncertainty in predictions can help users can make more optimal decisions [19, 22, 25, 37, 60].”
BackgroundSmith, C. E., Yu, B., Srivastava, A., Halfaker, A., Terveen, L., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376783 · Cited by 77 · cited 1×
“In fact, research shows that large collaborative communities such as Wikipedia require AI systems that balance multiple aspects such as reducing human effort, improving trust and positive engagement [58].”
BackgroundDo I trust my machine teammate?: an investigation from perception to decision
Yu, K., Berkovsky, S., Taib, R., Zhou, J., & Chen, F. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302277 · Cited by 71 · cited 1×
“Various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust in the AI [41, 81] or enables users to simulate the model predictions [16, 65], or assists developers to debug models [9, 38].”
BackgroundSelf-citationEffective Crowd Annotation for Relation Extraction
Liu, A., Soderland, S., Bragg, J., Lin, C. H., Ling, X., & Weld, D. S. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1104 · Cited by 68 · cited 1×
“; and they were shown dataset-specific examples. 2) To familiarize them with the task and verify their understanding, a screening phase required the participant to correctly label four of six reviews [44]. Only participants who passed the gating test were allowed to proceed to the main task. 3) The main task randomly…”
Self-citationQuantifying Interpretability and Trust in Machine Learning Systems
Schmidt, P. S., & Bießmann, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08558 · Cited by 68 · cited 1×
“Indeed, at first glance, it appears that researchers have already confirmed the utility of explanations on tasks ranging from medical diagnosis [10, 47] and data annotation [57] to deception detection [36].”
BackgroundA Slow Algorithm Improves Users' Assessments of the Algorithm's Accuracy
Park, J. S., Barber, R., Kirlik, A., & Karahalios, K. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359204 · Cited by 65 · cited 1×
“ndence and improve accuracy. For example, pairing the humans with slower AIs (that take more time to make recommendation) may provide the humans with a better chance to reflect on their own decisions [54]. Methods that embody management-science recommendations for avoiding group-think [49] might also be effective,e.g.…”
BackgroundConsistent Estimators for Learning to Defer to an Expert
Mozannar, H., & Sontag, D. (2020) · arXiv · doi:10.48550/arxiv.2006.01862 · Cited by 61 · cited 1×
“As recent work has suggested [4, 55, 59, 79], it may be useful to directly optimize for complementary behavior by accounting for the human behavior during training, who may have access to a diferent set of features [72].”
BackgroundCrowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study
van Berkel, N., Goncalves, J., Hettiachchi, D., Wijenayake, S., Kelly, R. M., & Kostakos, V. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359130 · Cited by 60 · cited 1×
“asks and advise human decision makers on higher stake decisions, such as medical diagnosis [11, 31], wildlife protection [19], improving housing for homeless youth [12, 29], and recidivism prediction [23, 25, 60]. While there are many motivations for creating these human-AI teams, a primary one is increasing the effec…”
BackgroundNatural Language Generation enhances human decision-making with uncertain information
Gkatzia, D., Lemon, O., & Rieser, V. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2043 · Cited by 59
Do explanations make VQA models more predictable to a human?
Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., & Parikh, D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1128 · Cited by 55 · cited 1×
“As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7, 30].”
BackgroundPatient risk stratification with time-varying parameters: a multitask learning approach
Wiens, J., Guttag, J. V., & Horvitz, E. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 54
Teaching Categories to Human Learners with Visual Explanations
Aodha, O. M., Su, S., Chen, Y., Perona, P., & Yue, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00402 · Cited by 52 · cited 3×
“AI explanations are useful in many scenarios: communicating model predictions [17, 30, 33, 56], teaching humans tasks like translation [48] or content moderation [28], augmenting human analysis procedure [28] or creativity [14], legal imperatives [51, 64], and others.”
BackgroundLearning to Prescribe Interventions for Tuberculosis Patients Using Digital Adherence Data
Killian, J. A., Wilder, B., Sharma, A., Choudhary, V., Dilkina, B., & Tambe, M. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330777 · Cited by 52
Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided ExplorationPreprint
Lakkaraju, H., Kamar, E. S., Caruana, R., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09064 · Cited by 41 · cited 2×
“s: (1) in high confidence regions (circles in Figure5), not only did workers have to agree more, but they also had to identify cases where the model failed with very high confidence (unknown unknowns [37]). Identifying unknown unknowns could have been a difficult and time-consuming task for workers, and they may have…”
BackgroundSelf-citationAllenNLP Interpret: A Framework for Explaining Predictions of NLP ModelsPreprint
Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., & Singh, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09251 · Cited by 40
Can AI become Reliable Source to Support Human Decision Making in a Court Scene?
Hayashi, Y., & Wakabayashi, K. (2017) · Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3022198.3026338 · Cited by 37 · cited 2×
“asks and advise human decision makers on higher stake decisions, such as medical diagnosis [11, 31], wildlife protection [19], improving housing for homeless youth [12, 29], and recidivism prediction [23, 25, 60]. While there are many motivations for creating these human-AI teams, a primary one is increasing the effec…”
BackgroundDong, X., & Hayes, C. C. (2012) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/1555343411432338 · Cited by 36
Finding Generalizable Evidence by Learning to Convince Q&A Models
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1244 · Cited by 27 · cited 1×
“For example, one might design the AI to summarize the best evidence for all possible options, without giving hard predictions, by training evidence agents [64].”
BackgroundHorvitz, E., & Paek, T. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-006-9026-1 · Cited by 26 · cited 1×
“For example, signifcantly improving decision accuracy by closing defciencies in automated reasoning with human efort, and vice versa [35, 70].”
BackgroundInvestigating Human + Machine Complementarity for Recidivism Predictions
Tan, S., Adebayo, J., Inkpen, K. M., & Kamar, E. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09123 · Cited by 25 · cited 1×
“For example, signifcantly improving decision accuracy by closing defciencies in automated reasoning with human efort, and vice versa [35, 70].”
BackgroundSelf-citationInterpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · cited 1×
“al explanations on team performance – that is, explaining each individual recommendation made by a model [56] rather than trying to provide a global understanding of the full model all at once (e.g., [38]). We are motivated by high-stakes decisions, such as medical diagnosis, but were daunted by the difficulty of runn…”
BackgroundSelf-citationVarshney, K. R., Khanduri, P., Sharma, P., Zhang, S., & Varshney, P. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09710 · Cited by 20 · cited 1×
“As recent work has suggested [4, 55, 59, 79], it may be useful to directly optimize for complementary behavior by accounting for the human behavior during training, who may have access to a diferent set of features [72].”
BackgroundA Human-Grounded Evaluation of SHAP for Alert Processing
Weerts, H., van Ipenburg, W., & Pechenizkiy, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03324 · Cited by 18 · cited 1×
“While there exist other explanatory approaches, such as feature-importance visualization [11, 27, 53, 61, 76] (more suitable for tabular data) or communicating influential training examples [3, 40, 80] (more suitable for images), these techniques are not ideal for text because they add an additional cognitive cost to…”
BackgroundOptimizing AI for Teamwork.Preprint
Bansal, G., Nushi, B., Kamar, E. S., Horvitz, E., & Weld, D. S. (2020) · arXiv (Cornell University) · Cited by 16
Self-citationOverCode: visualizing variation in student solutions to programming problems at scale
Glassman, E. L., Scott, J., Singh, R., Guo, P. J., & Miller, R. C. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2658779.2658809 · Cited by 15
Chan, H., Tran-Thanh, L., Wilder, B., Rice, E. R. W., Vayanos, P., & Tambe, M. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278757 · Cited by 14 · cited 1×
“e work process to automate routine tasks and advise human decision makers on higher stake decisions, such as medical diagnosis [11, 31], wildlife protection [19], improving housing for homeless youth [12, 29], and recidivism prediction [23, 25, 60]. While there are many motivations for creating these human-AI teams, a…”
BackgroundAvoiding "groupthink" A manager's challenge
MacLeod, L. (2011) · Nursing Management · doi:10.1097/01.numa.0000394953.94337.b4 · Cited by 7 · cited 1×
“e more time to make recommendation) may provide the humans with a better chance to reflect on their own decisions [54]. Methods that embody management-science recommendations for avoiding group-think [49] might also be effective,e.g., showing the AI’s prediction after the human’s initial answer or only having the AI p…”
BackgroundPatient risk stratification with time-varying parameters
WiensJenna, GuttagJohn, & HorvitzEric (2016) · Journal of Machine Learning Research · doi:10.5555/2946645.3007032 · Cited by 2
Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
Bansal, G., Nushi, B., Kamar, E., Horvitz, E., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2004.13102
Self-citation
Research works printed but not linked: 5
- 10.5555/647985.743843 · doi:10.5555/647985.743843
- 10.18653/v1 · doi:10.18653/v1
- 10.5555/3524938.3525594 · doi:10.5555/3524938.3525594
- 10.18653/v1 · doi:10.18653/v1
- LSAT Prep Books Team. 2017. LSAT prep book study guide: quick study & practice test questions for the Law School Admissions council’s (LSAC) Law school admission test. Mometrix Test Preparation, Beaumont, TX.
Cited by
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“Thus, whether information biases the generative process, as, for example, with the prompt in a language-to-image stable diffusion model, is sufficiently original for intellectual property and author attribution claims remains an open question.”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
“Collaboration between humans and AI can take many forms, but specific formulations of human–AI collaboration remain poorly defined and challenging to measure 11 .”
BackgroundWu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
“Human-AI interaction has been explored in domains such as classification [7, 62], drawing [26, 54], translation [30], creative writing [25, 29], and design ideation [38].”
BackgroundSelf-citationWang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
“Finally, Desideratum 3 concerns the ultimate goal of AI-assisted decision-making, that is, to maximize the joint human-AI team performance [5, 6, 11].”
Background
Metadata
- Authors
- Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- Microsoft Research, United States
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-16
- OpenAlex
- W3037634279
- Cited by
- 757
- References
- 101
- Citations by year
- 2026: 238 · 2025: 148 · 2024: 128 · 2023: 125 · 2022: 88 · 2021: 26
- FWCI
- 42.30
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Microsoft Research
- the University of Washington WRF/Cable Professorship
- Office of Naval Research (N00014-18-1-2193)
- NSF RAPID grant (2040196)
- the Allen Institute for Artificial Intelligence (AI2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- explainable AI, AI explanations, human-AI teaming, AI augmentation, trust in AI, human-AI decision-making, recommendation acceptance, human-centered AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.