The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Abstract
EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems [20, 34, 52, 55].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“The opportunity to 'be really curious to just kind of play around' [A2-05] can help mitigate anchoring bias [52].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“However, recent research has shown that the recommendation-driven explainable/interpretable AI has little effect on decision making [1, 6-8, 28, 32], although this is not always the case [22,23,41].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [28]
Citations
Cites
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Craik, K. J. W. (1943) · Cited by 1,845
Baron, J. A. (2006) · Cambridge University Press eBooks · doi:10.1017/9781009263672 · Cited by 1,434
Merton, R. Κ., & Kendall, P. L. (1946) · American Journal of Sociology · doi:10.1086/219886 · Cited by 1,212
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Research works printed but not linked: 5
- Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). · arXiv
- Antino Kim , Mochen Yang , and Jingjng Zhang . 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users ’ Reliance on Algorithms. Late Errors on Users ’ Reliance on Algorithms (July 2020 ) (2020). Antino Kim, Mochen Yang, and Jingjng Zhang. 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms. Late Errors on Users’ Reliance on Algorithms (July 2020) (2020).
- Tim Miller , Piers Howe , and Liz Sonenberg . 2017 . Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). · arXiv
- Don A Moore and Paul J Healy . 2008. The trouble with overconfidence.Psychological review 115, 2 ( 2008 ), 502. Don A Moore and Paul J Healy. 2008. The trouble with overconfidence.Psychological review 115, 2 (2008), 502.
- Forough Poursabzi-Sangdeh , Daniel G Goldstein , Jake M Hofman , Jennifer Wortman Vaughan , and Hanna Wallach . 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) ( 2018 ). Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) (2018). · arXiv
Cited by
Sorted by citations
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
“The dimensions that are often considered include users’ trust [20], reliance, satisfaction [24], mental model formation [27], and system acceptance.”
BackgroundTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“[56] study the order of observing system weakness and strengths, which reveals that encountering weakness first results in a lower rate of us-”
BackgroundInfluentialDing, W., Abdel‐Basset, M., Hawash, H., & Ali, A. M. A. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.013 · Cited by 213
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“…machine learning has little effect on decision making [Gajos and Mamykina, 2022, Buçinca et al., 2021, Green and Chen, 2019a,b, Poursabzi-Sangdeh et al., 2021, Nourani et al., 2021], although this is not always the case [Leichtmann et al., 2022, Madumal et al., 2020, van der Waa et al., 2021].”
BackgroundA survey on explainable artificial intelligence for cybersecurity
Rjoub, G., Bentahar, J., Abdel Wahab, O., Mizouni, R., Song, A., Cohen, R., Otrok, H., & Mourad, A. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3282740 · Cited by 158
“Other XAI techniques employed only post-hoc explanation methods, such as in the case of Confident Itemsets Explanation ( CIE ) [62], which explains a model’s predictions using the language model trained on a large corpus of text, ANCHOR [63], LRP [64], and SHAP [65].”
Method
Metadata
- Authors
- Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V.
- Affiliations
- University of Florida (US)
- The University of Texas at Dallas (US)
- INDIE Lab Department of Computer & Information Science & Engineering, United States
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 340-350
- OpenAlex
- W3152997756
- Cited by
- 107
- References
- 44
- Citations by year
- 2026: 21 · 2025: 29 · 2024: 20 · 2023: 23 · 2022: 10 · 2021: 4
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- NSF (National Science Foundation) (1900767)
- Defense Advanced Research Projects Agency (N66001-17-2-4032)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- anchoring effect, explainable AI, cognitive biases, automation bias, order effects, first impressions, video activity recognition, cooking, human-AI decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
