Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.

How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review

Open access
Bertrand, Astrid, Belloum, Rafik, Eagan, James R., Maxwell, Winston · 2022 · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

Abstract

The field of eXplainable Artificial Intelligence (XAI) aims to bring transparency to complex AI systems. Although it is usually considered an essentially technical field, effort has been made recently to better understand users' human explanation methods and cognitive constraints. Despite these advances, the community lacks a general vision of what and how cognitive biases affect explainability systems. To address this gap, we present a heuristic map which matches human cognitive biases with explainability techniques from the XAI literature, structured around XAI-aided decision-making. We identify four main ways cognitive biases affect or are affected by XAI systems: 1) cognitive biases affect how XAI methods are designed, 2) they can distort how XAI techniques are evaluated in user studies, 3) some cognitive biases can be successfully mitigated by XAI techniques, and, on the contrary, 4) some cognitive biases can be exacerbated by XAI techniques. We construct this heuristic map through the systematic review of 37 papers-drawn from a corpus of 285-that reveal cognitive biases in XAI systems, including the explainability method and the user and task types in which they arise. We use the findings from our review to structure directions for future XAI systems to better align with people's cognitive processes.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3514094.3534164

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 66 references

Research works printed but not linked: 19

  1. H. Lakkaraju , E. Kamar , R. Caruana , and J. Leskovec , " Interpretable & Explorable Approximations of Black Box Models," ArXiv170701154 Cs , Jul. 2017 . H. Lakkaraju, E. Kamar, R. Caruana, and J. Leskovec, "Interpretable & Explorable Approximations of Black Box Models," ArXiv170701154 Cs, Jul. 2017.
  2. 10.1145/3233231 · doi:10.1145/3233231 · Paper page
  3. D. Kahneman , Thinking , fast and slow . New York, NY, US : Farrar, Straus and Giroux , 2011 , p. 499. D. Kahneman, Thinking, fast and slow. New York, NY, US: Farrar, Straus and Giroux, 2011, p. 499.
  4. A. Springer and S. Whittaker , " Progressive disclosure: empirically motivated approaches to designing effective transparency," in Proceedings of the 24th International Conference on Intelligent User Interfaces, New York, NY , USA , Mar. 2019 , pp. 107 -- 120 . A. Springer and S. Whittaker, "Progressive disclosure: empirically motivated approaches to designing effective transparency," in Proceedings of the 24th International Conference on Intelligent User Interfaces, New York, NY, USA, Mar. 2019, pp. 107--120.
  5. D. J. Hilton , "Logic and causal attribution," in Contemporary science and natural explanation : Commonsense conceptions of causality , New York, NY, US : New York University Press , 1988 , pp. 33 -- 65 . D. J. Hilton, "Logic and causal attribution," in Contemporary science and natural explanation: Commonsense conceptions of causality, New York, NY, US: New York University Press, 1988, pp. 33--65.
  6. M. M. A. de Graaf and B. F. Malle , " How People Explain Action (and Autonomous Intelligent Systems Should Too)," presented at the 2017 AAAI Fall Symposium Series , Oct. 2017 . M. M. A. de Graaf and B. F. Malle, "How People Explain Action (and Autonomous Intelligent Systems Should Too)," presented at the 2017 AAAI Fall Symposium Series, Oct. 2017.
  7. U. Bhatt : Measuring, Communicating, and Using Uncertainty," in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY , USA , Jul. 2021 , pp. 401 -- 413 . U. Bhatt et al., "Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty," in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, Jul. 2021, pp. 401--413.
  8. T. Kulesza , S. Stumpf , M. Burnett , S. Yang , I. Kwan , and W. Wong , " Too much, too little, or just right? Ways explanations impact end users' mental models," in 2013 IEEE Symposium on Visual Languages and Human Centric Computing , Sep. 2013 , pp. 3 -- 10 . T. Kulesza, S. Stumpf, M. Burnett, S. Yang, I. Kwan, and W. Wong, "Too much, too little, or just right? Ways explanations impact end users' mental models," in 2013 IEEE Symposium on Visual Languages and Human Centric Computing, Sep. 2013, pp. 3--10.
  9. 10.1145/3449287 · doi:10.1145/3449287 · Paper page
  10. N. M. Barbosa and M. Chen , " Rehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning," in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, New York, NY , USA , May 2019 , pp. 1 -- 12 . N. M. Barbosa and M. Chen, "Rehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning," in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, May 2019, pp. 1--12.
  11. V. Danry , P. Pataranutaporn , Y. Mao , and P. Maes , " Wearable Reasoner: Towards Enhanced Human Rationality Through A Wearable Device With An Explainable AI Assistant," in Proceedings of the Augmented Humans International Conference, New York, NY , USA , Mar. 2020 , pp. 1 -- 12 . V. Danry, P. Pataranutaporn, Y. Mao, and P. Maes, "Wearable Reasoner: Towards Enhanced Human Rationality Through A Wearable Device With An Explainable AI Assistant," in Proceedings of the Augmented Humans International Conference, New York, NY, USA, Mar. 2020, pp. 1--12.
  12. L. Coba , L. Rook , M. Zanker , and P. Symeonidis , " Decision making strategies differ in the presence of collaborative explanations: two conjoint studies," in Proceedings of the 24th International Conference on Intelligent User Interfaces, New York, NY , USA , Mar. 2019 , pp. 291 -- 302 . L. Coba, L. Rook, M. Zanker, and P. Symeonidis, "Decision making strategies differ in the presence of collaborative explanations: two conjoint studies," in Proceedings of the 24th International Conference on Intelligent User Interfaces, New York, NY, USA, Mar. 2019, pp. 291--302.
  13. A. Simkute , E. Luger , M. Evans , and R. Jones , " Experts in the Shadow of Algorithmic Systems: Exploring Intelligibility in a Decision-Making Context," in Companion Publication of the 2020 ACM Designing Interactive Systems Conference, New York, NY , USA , Jul. 2020 . A. Simkute, E. Luger, M. Evans, and R. Jones, "Experts in the Shadow of Algorithmic Systems: Exploring Intelligibility in a Decision-Making Context," in Companion Publication of the 2020 ACM Designing Interactive Systems Conference, New York, NY, USA, Jul. 2020.
  14. G. A. Klein , Sources of Power : How People Make Decisions . Nature , 1988 . G. A. Klein, Sources of Power: How People Make Decisions. Nature, 1988.
  15. T. Kim and H. Song , " The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence's Suggestion," in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems , New York, NY, USA , Apr. 2020 , pp. 1 -- 8 . T. Kim and H. Song, "The Effect of Message Framing and Timing on the Acceptance of Artificial Intelligence's Suggestion," in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, Apr. 2020, pp. 1--8.
  16. H. Suresh , S. R. Gomez , K. K. Nam , and A. Satyanarayan , " Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York , NY, USA , May 2021 , pp. 1 -- 16 . H. Suresh, S. R. Gomez, K. K. Nam, and A. Satyanarayan, "Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, May 2021, pp. 1--16.
  17. G. Bansal AI Explanations on Complementary Team Performance ," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , New York, NY, USA , May 2021 , pp. 1 -- 16 . G. Bansal et al., "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, May 2021, pp. 1--16.
  18. M. McKenna and D. J. Coates , " Compatibilism," in The Stanford Encyclopedia of Philosophy , Fall 2021 ., E. N. Zalta, Ed . Metaphysics Research Lab, Stanford University , 2021. M. McKenna and D. J. Coates, "Compatibilism," in The Stanford Encyclopedia of Philosophy, Fall 2021., E. N. Zalta, Ed. Metaphysics Research Lab, Stanford University, 2021.
  19. J. Schaffer , J. O'Donovan , J. Michaelis , A. Raglin , and T. Höllerer , " I can do better than your AI: expertise and explanations," in Proceedings of the 24th International Conference on Intelligent User Interfaces, New York, NY , USA , Mar. 2019 , pp. 240 -- 251 . J. Schaffer, J. O'Donovan, J. Michaelis, A. Raglin, and T. Höllerer, "I can do better than your AI: expertise and explanations," in Proceedings of the 24th International Conference on Intelligent User Interfaces, New York, NY, USA, Mar. 2019, pp. 240--251.

Cited by

See all 134

Metadata

Authors
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W.
Affiliations
Télécom Paris (FR)
Venue
Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society
Publisher
ACM
Published
2022-07-26
Pages
78-91
ISSN
3065-8365 (ISSN-L)
OpenAlex
W4283032682
Cited by
129
References
66
Citations by year
2026: 26 · 2025: 46 · 2024: 36 · 2023: 20 · 2022: 1
FWCI
14.24
Type
conference paper
Language
en
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • HAL (submittedVersion, other-oa)
Funders
Agence Nationale de la Recherche (ANR-20-CHIA-0023-01)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
cognitive biases, explainable AI, bias mitigation, heuristic maps, user study, human-AI interaction, cognitive processes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review. https://app.cochat.ai/papers/10.1145/3514094.3534164
The paper
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022). How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3514094.3534164

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.