Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
Open accessAbstract
We present a randomized controlled trial for a model-in-the-loop regression task, with the goal of measuring the extent to which (1) good explanations of model predictions increase human accuracy, and (2) faulty explanations decrease human trust in the model. We study explanations based on visual saliency in an image-based age prediction task for which humans and learned models are individually capable but not highly proficient and frequently disagree. Our experimental design separates model quality from explanation quality, and makes it possible to compare treatments involving a variety of explanations of varying levels of quality. We find that presenting model predictions improves human accuracy. However, visual explanations of various kinds fail to significantly alter human accuracy or trust in the model - regardless of whether explanations characterize an accurate model, an inaccurate one, or are generated randomly and independently of the input image. These findings suggest the need for greater evaluation of explanations in downstream decision making tasks, better design-based tools for presenting explanations to users, and better approaches for generating explanations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Post-hoc model explanations have a small or even detrimental effect on the performance of AI-assisted decision making.
“Despite the insights post-hoc model explanations can provide end-users, they have been found to have a small or even detrimental effect on the performance of AI-assisted decision making [17, 34].”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as [17]
Prior empirical work finds that explanations do not reduce overreliance compared to prediction-only.
“Surprisingly, overreliance does not reduce when the AI produces explanations for its predictions, compared to only providing predictions.”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [15]
Some studies found that explanations increase trust, some found no significant effect, while others found varying effects under different conditions.
“Some studies found that explanations increase trust [4,8], some found no significant effect [15,42,97], while others found varying effects under different conditions [49,51].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [15]
Some studies use reliance as a proxy measure of trust, while others subjectively measure an individual's trust in the system and innate disposition to trust.
“Some studies use reliance as a proxy measure of trust [4,15,55,97], while others subjectively measure an individual's trust in the system and innate disposition to trust [8-10, 42, 51, 77].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [15]
The citation lists haven’t been fetched yet
References 46 · Cited by 34
- Authors
- Chu, E., Roy, D., & Andreas, J.
- Affiliations
- MIT Media Lab (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2007.12248
- OpenAlex
- W3043865406
- Cited by
- 34
- References
- 46
- Citations by year
- 2026: 1 · 2024: 6 · 2023: 4 · 2022: 11 · 2021: 8 · 2020: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- explainable AI, visual explanations, model-in-the-loop, explanation quality, age prediction, image-based prediction, explanation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
