The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Abstract
EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems [20, 34, 52, 55].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“The opportunity to 'be really curious to just kind of play around' [A2-05] can help mitigate anchoring bias [52].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“However, recent research has shown that the recommendation-driven explainable/interpretable AI has little effect on decision making [1, 6-8, 28, 32], although this is not always the case [22,23,41].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [28]
Citations
Cites
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Craik, K. J. W. (1943) · Cited by 1,845
Baron, J. A. (2006) · Cambridge University Press eBooks · doi:10.1017/9781009263672 · Cited by 1,434
Merton, R. Κ., & Kendall, P. L. (1946) · American Journal of Sociology · doi:10.1086/219886 · Cited by 1,212
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Research works printed but not linked: 5
- Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). · arXiv
- Antino Kim , Mochen Yang , and Jingjng Zhang . 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users ’ Reliance on Algorithms. Late Errors on Users ’ Reliance on Algorithms (July 2020 ) (2020). Antino Kim, Mochen Yang, and Jingjng Zhang. 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms. Late Errors on Users’ Reliance on Algorithms (July 2020) (2020).
- Tim Miller , Piers Howe , and Liz Sonenberg . 2017 . Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). · arXiv
- Don A Moore and Paul J Healy . 2008. The trouble with overconfidence.Psychological review 115, 2 ( 2008 ), 502. Don A Moore and Paul J Healy. 2008. The trouble with overconfidence.Psychological review 115, 2 (2008), 502.
- Forough Poursabzi-Sangdeh , Daniel G Goldstein , Jake M Hofman , Jennifer Wortman Vaughan , and Hanna Wallach . 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) ( 2018 ). Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) (2018). · arXiv
Cited by
Sorted by citations
Items 101–110 of 110
For Those About to Rely—A Taxonomy of Experimental Studies on AI Reliance
Schaschek, M., Spatscheck, N., & Winkelmann, A. (2026) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-08480-4_2
From Curiosity to Caution: How Expertise Shapes the Use of Interpretable Machine Learning
Naqvi, S., Balaji, G., Laju, A., & Kaur, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812376
“This shows that interpretability is bounded by human cognitive limits [92, 140, 144].”
Supporting Bayesian Reasoning to Mitigate Over-Reliance in AI-Assisted Decision Making
Mikhaylova, D., Turchi, T., Cevolani, G., & Malizia, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30849-8_6
Ask Before or After? Clarification Timing and User Experience in Generative AI
Jung, G., Ho, H. Y., Quartson, P., & Cheshire, C. (2026) · Creativity and Cognition · doi:10.1145/3803784.3816856
Navigating AI Errors: The Role of Mental Models in Human-AI Team Performance
Sanchez, D. N., Lancaster, C., Winokur, E. J., Treiman, L. S., Schutte, P., Bruner, T., & Ingram, J. E. (2026) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813261475166
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.010
Dhungel, A.-K., Aleknonytė‐Resch, M., Watermann, L. S., Hanke, K., Wolfram, E., & Heine, M. (2026) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg70169.2026.11695473
“While some studies suggest that explanations may foster overreliance on incorrect recommendations [24], others report improvements in decision accuracy, particularly in complex tasks such as medical diagnosis [26].”
User-Friendly AI-Based Decision Support Systems
Echeverry, N. (2026) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw71585.2026.11712078
“Nourani et al. found that generic representations of uncertainty can be counterproductive and possibly trigger misconceptions and cognitive biases in users as a coping mechanism [14].”
Armstrong, B. A., Cheuy, L., Kottler, N. E., & Larson, D. B. (2026) · American Journal of Roentgenology · doi:10.2214/ajr.26.35826
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“[79] demonstrated that when interacting with XAI, people were vulnerable to common cognitive biases such as anchoring bias after observing model behaviors early on.”
Background
Metadata
- Authors
- Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V.
- Affiliations
- University of Florida (US)
- The University of Texas at Dallas (US)
- INDIE Lab Department of Computer & Information Science & Engineering, United States
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 340-350
- OpenAlex
- W3152997756
- Cited by
- 107
- References
- 44
- Citations by year
- 2026: 21 · 2025: 29 · 2024: 20 · 2023: 23 · 2022: 10 · 2021: 4
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- NSF (National Science Foundation) (1900767)
- Defense Advanced Research Projects Agency (N66001-17-2-4032)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- anchoring effect, explainable AI, cognitive biases, automation bias, order effects, first impressions, video activity recognition, cooking, human-AI decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
