The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Abstract
EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems [20, 34, 52, 55].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“The opportunity to 'be really curious to just kind of play around' [A2-05] can help mitigate anchoring bias [52].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [52]
In the citing paper
“However, recent research has shown that the recommendation-driven explainable/interpretable AI has little effect on decision making [1, 6-8, 28, 32], although this is not always the case [22,23,41].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [28]
Citations
Cites
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Craik, K. J. W. (1943) · Cited by 1,845
Baron, J. A. (2006) · Cambridge University Press eBooks · doi:10.1017/9781009263672 · Cited by 1,434
Merton, R. Κ., & Kendall, P. L. (1946) · American Journal of Sociology · doi:10.1086/219886 · Cited by 1,212
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Research works printed but not linked: 5
- Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). · arXiv
- Antino Kim , Mochen Yang , and Jingjng Zhang . 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users ’ Reliance on Algorithms. Late Errors on Users ’ Reliance on Algorithms (July 2020 ) (2020). Antino Kim, Mochen Yang, and Jingjng Zhang. 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms. Late Errors on Users’ Reliance on Algorithms (July 2020) (2020).
- Tim Miller , Piers Howe , and Liz Sonenberg . 2017 . Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017). · arXiv
- Don A Moore and Paul J Healy . 2008. The trouble with overconfidence.Psychological review 115, 2 ( 2008 ), 502. Don A Moore and Paul J Healy. 2008. The trouble with overconfidence.Psychological review 115, 2 (2008), 502.
- Forough Poursabzi-Sangdeh , Daniel G Goldstein , Jake M Hofman , Jennifer Wortman Vaughan , and Hanna Wallach . 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) ( 2018 ). Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) (2018). · arXiv
Cited by
Sorted by citations
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
“The dimensions that are often considered include users’ trust [20], reliance, satisfaction [24], mental model formation [27], and system acceptance.”
BackgroundTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“[56] study the order of observing system weakness and strengths, which reveals that encountering weakness first results in a lower rate of us-”
BackgroundInfluentialDing, W., Abdel‐Basset, M., Hawash, H., & Ali, A. M. A. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.013 · Cited by 213
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“…machine learning has little effect on decision making [Gajos and Mamykina, 2022, Buçinca et al., 2021, Green and Chen, 2019a,b, Poursabzi-Sangdeh et al., 2021, Nourani et al., 2021], although this is not always the case [Leichtmann et al., 2022, Madumal et al., 2020, van der Waa et al., 2021].”
BackgroundA survey on explainable artificial intelligence for cybersecurity
Rjoub, G., Bentahar, J., Abdel Wahab, O., Mizouni, R., Song, A., Cohen, R., Otrok, H., & Mourad, A. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3282740 · Cited by 158
“Other XAI techniques employed only post-hoc explanation methods, such as in the case of Confident Itemsets Explanation ( CIE ) [62], which explains a model’s predictions using the language model trained on a large corpus of text, ANCHOR [63], LRP [64], and SHAP [65].”
MethodHow Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129
“[11], [12], [17], [32], [34], [59], [68] Related to information overload Choice overload Measure the user's cognitive load using the NASA Task Load Index (NASA-TLX) [23], [52], Eye-tracking measurements (for choice overload) [56] Do not use too many explainability types [40].”
BackgroundMethodInfluentialGómez, C., Cho, S. M., Ke, S., Huang, C., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 207
“Following this initial judgment, the AI’s predicted outcome is presented and may accompanied by support information.”
InfluentialPropagation of societal gender inequality by internet search algorithms
Vlasceanu, M., & Amodio, D. M. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2204529119 · Cited by 109
“Whereas past research has focused on interventions aimed at debiasing an algorithm’s training set (44, 45) and elucidating the computations of deep neural network models (46), our findings suggest that to increase transparency and fairness in AI it will also be critical to uncover how human decision makers interact wi…”
BackgroundInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
“The opportunity to “be really curious to just kind of play around” [A2-05] can help mitigate anchoring bias [52].”
BackgroundBao, Y., Gong, W., & Yang, K. (2023) · Systems · doi:10.3390/systems11090442 · Cited by 106
“Explanations have also been identified to be able to address users’ bias during human–AI synergy [42].”
Fok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81
The effects of explanations on automation bias
Vered, M., Livni, T., Howe, P. D. L., Miller, T., & Sonenberg, L. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103952 · Cited by 83
Bertrand, A., Viard, T., Belloum, R., Eagan, J. R., & Maxwell, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581314 · Cited by 71
“Number of correct guesses of the AI’s prediction by the user [23, 85, 111].”
BackgroundFok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54
“…✗ ✗ Wang and Yin [71] Recidivism prediction Feature importance ✗ ✗ Wang and Yin [71] Forest cover prediction Feature importance ✗ ✗ Nourani et al. [57] Kitchen policy verification Feature importance ✗ ✗ Poursabzi-Sangdeh et al. [59] Apartment price prediction Feature importance ✗ ✗ Jesus et al.…”
BackgroundMitigating knowledge imbalance in AI-advised decision-making through collaborative user involvement
Gómez, C., Unberath, M., & Huang, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102977 · Cited by 57
Al-Ansari, N., Al‐Thani, D., & Al-Mansoori, R. S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/4628855 · Cited by 58
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 87
“During the study, we randomly shuffled the task order for each participant to prevent ordering effects [84].”
Bach, A. K. P., Nørgaard, T. M., Brok, J. C., & van Berkel, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581513 · Cited by 47
Effects of AI and Logic-Style Explanations on Users’ Decisions Under Different Levels of Uncertainty
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588320 · Cited by 44
“Most notably, Nourani et al. [56] studied how ordering biases can affect users’ mental models and reliance/confidence formation in intelligent systems and the role the explanations play with such biases.”
BackgroundInfluentialA Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“Prior work has also examined how human trust and reliance on AI systems is affected by different design choices through empirical studies [16, 52, 67, 76].”
BackgroundMethodSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“Such factors can be broadly classified into three primary categories: human-related factors [35, 95, 96], attributes of the AI systems [94, 98], and characteristics of the decision-making tasks [16, 56, 126].”
BackgroundTo Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
Bo, J. Y., Wan, S., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714097 · Cited by 36
“On the individual level, people may also be averse or appreciative towards algorithms, perhaps through pre-conceptions or judgments formed after seeing the AI’s performance [21, 37, 53, 57, 62].”
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12708 · Cited by 34
“3.3); • Anchoring Bias [25].”
InfluentialAgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 33
Spatola, N. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100099 · Cited by 55
Miedema, E., Waschull, S., & Emmanouilidis, C. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104409 · Cited by 41
Effects of LLM-based Search on Decision Making: Speed, Accuracy, and Overreliance
Spatharioti, S. E., Rothschild, D., Goldstein, D. G., & Hofman, J. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714082 · Cited by 36
“Participants in both conditions saw this item in the last position (task 5), making it the task where overreliance was more likely to occcur [34].”
Linder, R., Mohseni, S., Yang, F., Pentyala, S., Ragan, E. D., & Hu, X. (2021) · Applied AI Letters · doi:10.1002/ail2.49 · Cited by 30
“For instance, in research by Nourani et al, 37 a user study with interactive video review for activity recognition provided cautions about cognitive biases and overconfidence due to the addition of explanations.”
MethodSelf-citationNourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E. D., & Gogate, V. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3531066 · Cited by 30
“For instance, as we observed from the results of this paper [45] and our past work [44], first impressions of the XAI system affects mental model formations, confidence, reliance, and trust calibration.”
BackgroundResultInfluentialSelf-citationEverett, S., Bunning, B., Jain, P., López, I., Agarwal, A., Desai, M., Gallo, R. J., Goh, E., Kadiyala, V., Kanjee, Z., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02545-1 · Cited by 66
Poornikoo, M., Gyldensten, W., Vesin, B., & Øvergård, K. I. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2399439 · Cited by 30
“The anchoring effect, which describes how individuals make estimates or decisions starting from an initial reference point (Nourani et al., 2021), can significantly impact trust dynamics.”
The effects of generative AI’s human-like competencies on clinical decision-making
Spatscheck, N., Schaschek, M., & Winkelmann, A. (2024) · Journal of Decision System · doi:10.1080/12460125.2024.2430731 · Cited by 28
Capturing Humans’ Mental Models of AI: An Item Response Theory Approach
Kelly, M., Kumar, A., Smyth, P., & Mark Steyvers, B. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594111 · Cited by 42
“stochastic [6] and can be biased by first impressions of the agent [51].”
BackgroundResultCoExploreDS: Framing and Advancing Collaborative Design Space Exploration Between Human and AI
Chen, P., Yao, J., Cheng, Z., Cai, Y., Li, J., You, W., & Sun, L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713869 · Cited by 22
“From the AI’s perspective, satisfactory results during the early stages of collaboration can help designers form a positive first impression, leading to greater reliance on the AI [58].”
InfluentialSchrills, T., & Franke, T. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588594 · Cited by 26
“In XAI research explanatory approaches partly refer to the confidence of the model [ 10 , 17 , 86 ] for a certain result or even to meta-information about the model [ 75 ].”
MethodExplorable Explainable AI: Improving AI Understanding for Community Health Workers in India
Solano-Kamaiko, I. R., Mishra, D., Dell, N. L., & Vashistha, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642733 · Cited by 28
“We found it difcult to shift AWWs’ inferences away from these anchoring biases [ 90].”
BackgroundHow Do HCI Researchers Study Cognitive Biases? A Scoping Review
Boonprakong, N., Tag, B., Gonçalves, J., & Dingler, T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713450 · Cited by 37
“These papers discuss the problems of explainable AI [37, 71, 73, 143, 151, 200], AI-assisted decision making [8, 25, 27, 35, 52, 128, 129, 155, 178], and trust-reliance in AI [81, 82, 103, 185].”
InfluentialA Scoping Review of Mental Model Research in HCI from 2010 to 2021
Hu, X., & Twidale, M. B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48038-6_7 · Cited by 19
Silent Vulnerable Dependency Alert Prediction with Vulnerability Key Aspect Explanation
Sun, J., Xing, Z., Lu, Q., Xu, X., Zhu, L., Hoang, T., & Zhao, D. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00089 · Cited by 29
“Studies on XAI [22], [63] show irrelevant explanations to AI decisions are even worse than no explanations.”
Eckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 38
“Most of these articles exclusively use a survey to measure reliance [e.g. 33, 92, 98], while others combine surveys with metrics like gaze duration or perception-based measures [22, 74].”
Kaur, H., Conrad, M. R., Rule, D., Lampe, C. A. C., & Gilbert, É. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637354 · Cited by 17
“Some recent work relies on cognitive science theories to assert that cognitive heuristics can affect the effectiveness of explainability tools in various domain-specific settings [3, 11, 13, 32, 58, 90, 103, 103].”
BackgroundA Diachronic Perspective on User Trust in AI under Uncertainty
Dhuliawala, S., Zouhar, V., El‐Assady, M., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.339 · Cited by 29
Framework for human–XAI symbiosis: extended self from the dual-process theory perspective
Litvinova, Y., Mikalef, P., & Luo, X. R. (2024) · Journal of Business Analytics · doi:10.1080/2573234x.2024.2396366 · Cited by 16
“Similarly, the order in which humans encounter XAI errors impacts how they evaluate XAI accuracy (Nourani et al., 2021).”
InfluentialExplainable activity recognition in videos: Lessons learned
Roy, C., Nourani, M., Honeycutt, D. R., Block, J. E., Rahman, T., Ragan, E. D., Ruozzi, N., & Gogate, V. (2021) · Applied AI Letters · doi:10.1002/ail2.59 · Cited by 13
Self-citationSalimzadeh, S., & Gadiraju, U. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659567 · Cited by 14
An Empirical Evaluation of Predicted Outcomes as Explanations in Human-AI Decision-Making
Jakubik, J., Schoeffer, J., Hoge, V., Vössing, M., & Kühl, N. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23618-1_24 · Cited by 15
Ma, S., Sun, M., & Ma, X. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3571813 · Cited by 12
“…making model behavior more intelligible through explanations (a.k.a. explainable AI [40, 54, 103, 188]) and informing users of the inner working status of the AI system (e.g., capabilities, confidence, uncertainties, trustworthiness, etc. [9, 10, 48, 128]) in easily understandable ways [48, 150].”
BackgroundLee, J., Jung, K., Newman, E., Chow, E., & Chen, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713244 · Cited by 15
The Amplifying Effect of Explainability in AI-assisted Decision-making in Groups
Duarte, R. D. B., Abreu, M. C., Campos, J., & Paiva, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713534 · Cited by 13
“One possible explanation for this effect is the amplification of cognitive biases such as the anchoring bias [7, 43, 61] and automation bias [35] in group dynamics.”
The Influence of Curiosity Traits and On-Demand Explanations in AI-Assisted Decision-Making
Cau, F. M., & Spano, L. D. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712165 · Cited by 13
de Jong, S., Bos, M. W., van Berkel, N., & Lamers, M. H. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2535732 · Cited by 19
Workshop on Understanding and Mitigating Cognitive Biases in Human-AI Collaboration
Boonprakong, N., He, G., Gadiraju, U., van Berkel, N., Wang, D., Chen, S., Liu, J., Tag, B., Gonçalves, J., & Dingler, T. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3611284 · Cited by 11
“Moreover, studies have shown that AI explanations can exacerbate our cognitive biases and compromise AI-assisted decision-making, such as trust in AI [33], reliance on AI [5, 10, 21, 39], and interpretation of AI results [27, 46].”
BackgroundMajority voting of doctors improves appropriateness of AI reliance in pathology
Gu, H., Yang, C., Magaki, S., Zarrin‐Khameh, N., Lakis, N. S., Cobos, I., Khanlou, N., Zhang, X. R., Assi, J. S., Byers, J. T., et al. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103315 · Cited by 7
Bubbleu: Exploring Augmented Reality Game Design with Uncertain AI-based Interaction
Kim, M., Lee, K., Balan, R. K., & Lee, Y. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581270 · Cited by 15
“Studies have shown that the timing of encountering AI model errors [44, 68], level of expertise [53, 67], and fidelity of the information used to explain the AI model results [45, 93] affects the user’s trust and understanding towards the system.”
BackgroundDETOXER: A Visual Debugging Tool With Multiscope Explanations for Temporal Multilabel Classification
Nourani, M., Roy, C., Honeycutt, D. R., Ragan, E. D., & Gogate, V. (2022) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2022.3201465 · Cited by 7
“When working with multilabel classification, instances that are detected as matches for any given class are often a smaller subset than nonmatches and easier to examine, while users may be biased toward inspecting more false positives than negatives.(4) We provide additional global explanations to help contextualize i…”
Self-citationSchäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757451 · Cited by 10
“However, multiple factors have been found to influence users’ mental models of an AI system and thus team performance, such as trust [191], confidence [191], observing system weaknesses [128], error boundaries [8], and perceived accuracy [92].”
Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 9
Predicting the need for XAI from high-granularity interaction data
de Santana, V. F., Fucs, A., Segura, V., de Moraes, D. B., & Cerqueira, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103029 · Cited by 6
Wizard of Errors: Introducing and Evaluating Machine Learning Errors in Wizard of Oz Studies
Jansen, A., & Colombo, S. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519684 · Cited by 9
“Previous studies have shown that ML errors can greatly impact the UX [1, 10, 11], the mental models users form to interact with the system, and their trust in the ML model [17].”
Raees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
“These studies use (experiment) conditions to test the effect of feature-based [60, 153, 156, 164], example-based [17, 23, 108, 146], model-based [8, 33, 55, 114], or general explanations methods [102, 117, 127].”
InfluentialExploring the impact of explainable AI and cognitive capabilities on users’ decisions
Cau, F. M., & Spano, L. D. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09438-0 · Cited by 16
“To account for ordering effects (Nourani et al. 2021), we prepared 400 random permutations for the practice and main session instances, ensuring each participant sees differently ordered loan requests.”
Thomas, I., Oh, S. Y., & Szafir, D. A. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645207 · Cited by 5
“However, the ways systems query for analyst input may negatively impact decision making by perpetuating preexisting bias [41, 44] or by leading to improper mental models [23, 54].”
BackgroundMethodResilience Through Appropriation: Pilots’ View on Complex Decision Support
Zhang, Z. T., Storath, C., Liu, Y., & Butz, A. M. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584056 · Cited by 11
“Reflecting the HB tradition of decision-making research, there has recently been a quickly growing number of controlled studies on intelligent DSTs, often through the lens of human cognitive biases [6, 11, 46].”
BackgroundTrust Junk and Evil Knobs: Calibrating Trust in AI Visualization
Wall, E., Matzen, L. E., El‐Assady, M., Masters, P., Hosseinpour, H., Endert, A., Borgo, R., Chau, P., Perer, A., Schupp, H. T., et al. (2024) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis60374.2024.00012 · Cited by 9
“Good initial performance from an AI system can lead to over-reliance on later outputs [64] whereas early errors have an out-sized negative impact on trust [64,65].”
BackgroundThe Role of Initial Acceptance Attitudes Toward AI Decisions in Algorithmic Recourse
Tominaga, T., Yamashita, N., & Kurashima, T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713573 · Cited by 4
“Conversely, when users initially view AI as highly accurate, they are prone to accept its suggestions [61, 62, 91].”
When AI reviews science: Can we trust the referee?
Wang, J., Liu, Y., Xu, H., Hu, K., Di, S., Ni, W., Yue, L., Zhang, M.-L., Ren, K., & Chen, L. (2026) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2026.100030 · Cited by 7
“By using persuasive rhetoric in these sections, they may anchor the AI’s initial assessment on a favorable premise before methods and evidence are scrutinized 119 , biasing the downstream evaluation.”
Evaluating What Others Say: The Effect of Accuracy Assessment in Shaping Mental Models of AI Systems
Do, H. J., Brachman, M., Dugan, C., Pan, Q., Rai, P., Johnson, J. M., & Thawani, R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686912 · Cited by 7
“Nourani et al. suggested that designers of intelligent systems provide guidance for users, as un-directed use of a system can lead to erroneous mental models [56].”
Detecting and explaining anomalies in industrial Internet of things systems using an autoencoder
Levshun, D., Levshun, D. S., & Kotenko, I. V. (2025) · Ontology of Designing · doi:10.18287/2223-9537-2025-15-1-96-113 · Cited by 4
“Методы апостериорного объяснения используются для объяснения производительности модели после еѐ обучения [27, 28] .”
Brereton, M. F., Ambe, A. H., Lovell, D. R., Sitbon, L., Capel, T., Soro, A., Xu, Y., Moreira, C., Favre, B., & Bradley, A. P. (2023) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/3638380.3638435 · Cited by 4
Mental Models of Adversarial Machine LearningPreprint
Bieringer, L., Grosse, K., Michael Backes, A., Biggio, B., & Krombholz, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03726 · Cited by 5
“Mental models have been found to serve as scaffolds not only for a given ML application [84], but also for its embedding in organizational practices [91].”
Folgado, D., Famiglini, L., Campagner, A., Dores, H., Barandas, M., Gamboa, H. F. S., & Cabitza, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95838-0_14 · Cited by 4
Human-Centered AI Design Principles for Generative AI
Weisz, J. D., He, J., & Muller, M. J. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_74 · Cited by 4
Sode, K. (2026) · AI & Society · doi:10.1007/s00146-026-02885-2 · Cited by 4
“…pathways remains limited; the claim here is therefore advanced as a theory-guided expectation consistent with broader findings on explanation-induced overconfidence (Rozenblit and Keil 2002; Nourani et al. 2021) and with the institutional “Impossible Task” of oversight under opacity.”
Biases and Ethical Considerations in AI-Driven Knowledge Management
Gafni, R., Kantsepolsky, B., & Sherman, S. (2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80592-391-620261013 · Cited by 3
Erlei, A., Cau, F. M., Georgiev, R., Kumar, S. C., Bizer, K., & Gadiraju, U. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791420 · Cited by 7
“To control for potential ordering biases [107], we (i) applied block randomisation across the presentation of the three tasks, and (ii) generated 400 random permutations of the ten instances to ensure each participant encountered uniquely ordered task instances.”
Human-AI Teaming: Following the IMOI Framework
Kleanthous, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60611-3_27 · Cited by 4
Antar, A. D., Molaei, S., Chen, Y.-Y., Lee, M. L., & Banović, N. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676323 · Cited by 9
Building Human-AI Reliance Through Cognitive Engagement and Exploratory AI Assistance
Raees, M., Khan, V.-J., & Papangelis, K. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3734171 · Cited by 5
“For instance, this requires supporting semantic alignment with domain users’ business processes and mental models [19].”
InfluentialRelEYEance: Gaze-based Assessment of Users' AI-reliance at Run-time
Wu, Z., Wang, Y., Langer, M., & Feit, A. M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3725841 · Cited by 6
“…stem from various factors, including biases toward AI such as expecting near-perfect performance [21, 25, 27], a lack of experience with an AI system thus having little evidence about the actual performance of those systems [16, 29], or unfit explanations for system behavior and outputs [2, 20].”
The Enterprising and Elusive Prospects of Human-AI Collaboration
Gadiraju, U., & Balayn, A. (2025) · Springer eBooks · doi:10.1007/978-3-032-01940-0_7 · Cited by 5
Satapathy, J., & Panda, J. K. (2025) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v6.i1.2025.4409 · Cited by 5
AI in the city: impact assessment of artificial intelligence uses in earth observation
Šćepanović, S., Bogucka, E. P., & Quercia, D. (2025) · Urban Informatics · doi:10.1007/s44212-025-00074-y · Cited by 3
“This level of agreement, which is lower than the alignment between the majority label and the LLM, suggests a potential over-reliance on the LLM labels, likely influenced by the well-documented anchoring effect (Furnham & Boo, 2011; Nourani et al., 2021).”
Human-Centered Explainable AI (XAI)
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 448
Manger, C., Vogl, A., Rosbach, E., Ziegler, C., & Riener, A. (2023) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3581961.3609897 · Cited by 2
“Research from other disciplines for example found that providing explanations about the automation’s capabilities can influence the formation of automation bias, depending on whether users were presented with the system’s strengths or weaknesses [15].”
Satapathy, J., & Panda, J. K. (2024) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v5.i3.2024.4409 · Cited by 1
Conceptualizing Multi-party AI Reliance for Design Research
Eckhardt, S., Dolata, M., Bauer-Hänsel, I., & Schwabe, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61175-9_4 · Cited by 1
Improving Tactical Decision-Making Through Multiobjective Contrastive Explanations
Blom, M. L., Singh, R., Miller, T., Sonenberg, L., Trentelman, K., Saulwick, A., & Wark, S. (2025) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3591163 · Cited by 1
“Empirical studies show that ex-plainability tools can positively impact decision-making [20], [22], [23], [33], however, in many cases, it has minimal effect [34], [35], [36], [37], [38].”
Ahmed, M. H., Alkhazaleh, S., & Aldallal, A. (2025) · Studies in systems, decision and control · doi:10.1007/978-3-031-87550-2_56 · Cited by 1
Manger, C., Vogl, A., & Riener, A. (2025) · Applied Sciences · doi:10.3390/app152011030 · Cited by 1
Sonenberg, L. (2023) · Logic Journal of IGPL · doi:10.1093/jigpal/jzad006
Der Mensch im Mittelpunkt: Einblick in die Gestaltung Menschenzentrierter Künstlicher Intelligenz
Weitz, K. (2024) · Springer eBooks · doi:10.1007/978-3-658-43816-6_14
Thomaschewski, L., Tausch, A., Berretta, S., Vogel, O., & Kluge, A. (2025) · Zeitschrift für wirtschaftlichen Fabrikbetrieb · doi:10.1515/zwf-2025-1113
Behzad, T., Gurney, N., Wang, N., & Pynadath, D. V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13187-4_8
Cognitive Process Tracing in Algorithm Augmented Decision Making
Wohlschlegel, J., & Jussupow, E. (2025) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-00815-2_9
Cognitive models facilitate real-time inference of latent motives
Fitch, A. K., & Kvam, P. D. (2026) · Scientific Reports · doi:10.1038/s41598-026-37587-8
“Furthermore, if cognitive models can help an AI system exhibit more human-like behavior, a person might be able to build a better mental model of the system which could improve trust 70,71 .”
The HEART Interface: Visualizing Risk Score Uncertainty in the Cardiothoracic ICU
Nourani, M., Nguyen, L., Barry, C. L., Jin, Q., & Tory, M. K. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789109
Self-citationExplainability for machine learning models : from data adaptability to user perception
Delaunay, J. (2023) · theses.fr (ABES) · doi:10.70675/933366c4z576ez464cz9fcfz24627f86dd2f
Bertrand, A. (2024) · theses.fr (ABES) · doi:10.70675/216d3647z78bcz4c68z9ba9z2d0a029cf1b4 · Cited by 1
From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Use
Chen, N., Lu, J., Wang, Z., Qiu, L. K., Chen, S., & Yang, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790661 · Cited by 3
“…users to develop understanding through hands-on practice [17, 42, 43], helps build robust mental models essential for effective software control [9, 67], reduces the risk of overreliance on AI systems [13, 82, 100], and develops domain knowledge that enables users to better evaluate AI responses…”
From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants
Rismani, S., Blodgett, S. L., Liao, Q. V., Olteanu, A., & Moon, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791670 · Cited by 1
“Prior work on predictive algorithms has shown that persuasive explanations can lead users to form mental models that promote over-reliance or under-trust [4–6 , 56].”
Metadata
- Authors
- Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V.
- Affiliations
- University of Florida (US)
- The University of Texas at Dallas (US)
- INDIE Lab Department of Computer & Information Science & Engineering, United States
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 340-350
- OpenAlex
- W3152997756
- Cited by
- 107
- References
- 44
- Citations by year
- 2026: 21 · 2025: 29 · 2024: 20 · 2023: 23 · 2022: 10 · 2021: 4
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- NSF (National Science Foundation) (1900767)
- Defense Advanced Research Projects Agency (N66001-17-2-4032)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- anchoring effect, explainable AI, cognitive biases, automation bias, order effects, first impressions, video activity recognition, cooking, human-AI decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
