The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Abstract
EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems [20, 34, 52, 55].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“The opportunity to 'be really curious to just kind of play around' [A2-05] can help mitigate anchoring bias [52].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“However, recent research has shown that the recommendation-driven explainable/interpretable AI has little effect on decision making [1, 6-8, 28, 32], although this is not always the case [22,23,41].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [28]
Citations
Cites
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Craik, K. J. W. (1943) · Cited by 1,845
Baron, J. A. (2006) · Cambridge University Press eBooks · doi:10.1017/9781009263672 · Cited by 1,434
Merton, R. Κ., & Kendall, P. L. (1946) · American Journal of Sociology · doi:10.1086/219886 · Cited by 1,212
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Mental models and human reasoning
Johnson-Laird, P. N. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1012933107 · Cited by 652
The Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582
The validity of verbal protocols
Russo, J. E., Johnson, E. J., & STEPHENS, D. L. (1989) · Memory & Cognition · doi:10.3758/bf03202637 · Cited by 545
Merritt, S. M., & Ilgen, D. R. (2008) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x288574 · Cited by 527 · cited 1×
“Research on mental models in intelligent systems shows that as users work with an intelligent system, they develop more robust mental models, thus relying less on their dispositional trust and more-so on their experiential trust [22, 23].”
BackgroundAccountability and the Perseverance of First Impressions
Tetlock, P. E. (1983) · Social Psychology Quarterly · doi:10.2307/3033716 · Cited by 513 · cited 1×
“First impressions have strong influences on human’s minds towards information [43], and as shown by our results, they can be strong against automated systems as well.”
BackgroundClinical applications of machine learning algorithms: beyond the black box
Watson, D. S., Krutzinna, J., Bruce, I. N., Griffiths, C. E., McInnes, I. B., Barnes, M. R., & Floridi, L. (2019) · BMJ · doi:10.1136/bmj.l886 · Cited by 450
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Affective Processes in Human–Automation Interactions
Merritt, S. M. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811411912 · Cited by 260
The challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253 · cited 1×
“Theoretically, explanations should help users build a better mental model of an intelligent system [48].”
BackgroundExplainable Artificial Intelligence (XAI)
Mohaghegh, S. D. (2024) · CRC Press eBooks · doi:10.1201/9781003369356-8 · Cited by 230
The relation between problem categorization and problem solving among experts and novices
Hardiman, P. T., Dufresne, R. J., & Mestre, J. P. (1989) · Memory & Cognition · doi:10.3758/bf03197085 · Cited by 227
Activity Recognition for the Smart Hospital
Sánchez-Cortés, D., Tentori, M. E., & Favela, J. (2008) · IEEE Intelligent Systems · doi:10.1109/mis.2008.18 · Cited by 220
Coherent Multi-sentence Video Description with Variable Level of Detail
Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Pinkal, M., & Schiele, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11752-2_15 · Cited by 209
Suspicious human activity recognition: a review
Tripathi, R. K., Jalal, A. S., & Agrawal, S. C. (2017) · Artificial Intelligence Review · doi:10.1007/s10462-017-9545-7 · Cited by 181
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163
Khan, Z. A., & Sohn, W. (2011) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2011.6131162 · Cited by 136
Yeung, S., Rinaldo, F., Jopling, J., Liu, B., Mehra, R., Downing, N. L., Guo, M., Bianconi, G. M., Alahi, A., Lee, J., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0087-z · Cited by 125
Understanding Decision Making in Critical Care
Lighthall, G. K., & Guillamet, C. V. (2015) · Clinical Medicine & Research · doi:10.3121/cmr.2015.1289 · Cited by 121
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119
Nourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
Self-citationClinical Applications of Machine Learning Algorithms: Beyond the Black BoxPreprint
Watson, D., Krutzinna, J. I., Bruce, I. N., Griffiths, C. E. M., McInnes, I. B., Barnes, M. R., & Floridi, L. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3352454 · Cited by 93
The Anchoring Effect in Decision-Making with Visual Analytics
Cho, I., Wesslen, R., Karduni, A., Santhanam, S., Shaikh, S., & Dou, W. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585665 · Cited by 88
Honeycutt, D. R., Nourani, M., & Ragan, E. D. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7464 · Cited by 86
Self-citationA Survey of Evaluation Methods and Measures for Interpretable Machine LearningPreprint
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv (Cornell University) · Cited by 69
Self-citationRahman, T., Kothalkar, P. V., & Gogate, V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44851-9_40 · Cited by 66
Self-citationToward Intelligent Decision Support Systems: An Artificially Intelligent Statistician
Remus, W., & Kottemann, J. E. (1986) · MIS Quarterly · doi:10.2307/249197 · Cited by 66
Kostopoulou, O., Sirota, M., Round, T., Samaranayaka, S., & Delaney, B. C. (2016) · Medical Decision Making · doi:10.1177/0272989x16644563 · Cited by 53
Investigating the Importance of First Impressions and Explainable AI with Interactive Video Analysis
Nourani, M., Honeycutt, D. R., Block, J. E., Roy, C., Rahman, T., Ragan, E. D., & Gogate, V. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382967 · Cited by 36
Self-citationA Formative Study of Interactive Bias Metrics in Visual Analytics Using Anchoring Bias
Wall, E., Blaha, L. M., Paul, C. L., & Endert, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29384-0_34 · Cited by 23
Robust Little Flame Detection on Real-Time Video Surveillance System
Lai, T. Y., Kuo, J.-Y., Fanjiang, Y., Ma, S., & Liao, Y. H. (2012) · International Conference on Innovations in Bio-inspired Computing and Applications · doi:10.1109/ibica.2012.41 · Cited by 15
Explainable Activity Recognition in Videos.
Roy, C., Shanbhag, M., Nourani, M., Rahman, T., Kabir, S., Gogate, V., Ruozzi, N., & Ragan, E. D. (2019) · IUI Workshops · Cited by 10
Self-citation4. Confidence considered: assessing the quality of decisions and performance
Dunning, D. A. (2012) · Dialnet (Universidad de la Rioja) · Cited by 7
Research works printed but not linked: 5
- Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). · arXiv
- Antino Kim , Mochen Yang , and Jingjng Zhang . 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users ’ Reliance on Algorithms. Late Errors on Users ’ Reliance on Algorithms (July 2020 ) (2020). Antino Kim, Mochen Yang, and Jingjng Zhang. 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms. Late Errors on Users’ Reliance on Algorithms (July 2020) (2020).
- Tim Miller , Piers Howe , and Liz Sonenberg . 2017 . Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). · arXiv
- Don A Moore and Paul J Healy . 2008. The trouble with overconfidence.Psychological review 115, 2 ( 2008 ), 502. Don A Moore and Paul J Healy. 2008. The trouble with overconfidence.Psychological review 115, 2 (2008), 502.
- Forough Poursabzi-Sangdeh , Daniel G Goldstein , Jake M Hofman , Jennifer Wortman Vaughan , and Hanna Wallach . 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) ( 2018 ). Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) (2018). · arXiv
Cited by
Sorted by citations
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
“The dimensions that are often considered include users’ trust [20], reliance, satisfaction [24], mental model formation [27], and system acceptance.”
BackgroundTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“[56] study the order of observing system weakness and strengths, which reveals that encountering weakness first results in a lower rate of us-”
BackgroundInfluentialDing, W., Abdel‐Basset, M., Hawash, H., & Ali, A. M. A. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.013 · Cited by 213
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“…machine learning has little effect on decision making [Gajos and Mamykina, 2022, Buçinca et al., 2021, Green and Chen, 2019a,b, Poursabzi-Sangdeh et al., 2021, Nourani et al., 2021], although this is not always the case [Leichtmann et al., 2022, Madumal et al., 2020, van der Waa et al., 2021].”
BackgroundA survey on explainable artificial intelligence for cybersecurity
Rjoub, G., Bentahar, J., Abdel Wahab, O., Mizouni, R., Song, A., Cohen, R., Otrok, H., & Mourad, A. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3282740 · Cited by 158
“Other XAI techniques employed only post-hoc explanation methods, such as in the case of Confident Itemsets Explanation ( CIE ) [62], which explains a model’s predictions using the language model trained on a large corpus of text, ANCHOR [63], LRP [64], and SHAP [65].”
Method
Metadata
- Authors
- Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V.
- Affiliations
- University of Florida (US)
- The University of Texas at Dallas (US)
- INDIE Lab Department of Computer & Information Science & Engineering, United States
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 340-350
- OpenAlex
- W3152997756
- Cited by
- 107
- References
- 44
- Citations by year
- 2026: 21 · 2025: 29 · 2024: 20 · 2023: 23 · 2022: 10 · 2021: 4
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- NSF (National Science Foundation) (1900767)
- Defense Advanced Research Projects Agency (N66001-17-2-4032)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- anchoring effect, explainable AI, cognitive biases, automation bias, order effects, first impressions, video activity recognition, cooking, human-AI decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
