Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
This paper addresses an under-explored problem of AI-assisted decision-making: when objective performance information of the machine learning model underlying a decision aid is absent or scarce, how do people decide their reliance on the model? Through three randomized experiments, we explore the heuristics people may use to adjust their reliance on machine learning models when performance feedback is limited. We find that the level of agreement between people and a model on decision-making tasks that people have high confidence in significantly affects reliance on the model if people receive no information about the model’s performance, but this impact will change after aggregate-level model performance information becomes available. Furthermore, the influence of high confidence human-model agreement on people’s reliance on a model is moderated by people’s confidence in cases where they disagree with the model. We discuss potential risks of these heuristics, and provide design implications on promoting appropriate reliance on AI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, confirmation bias and related confirmatory heuristics also contribute to inappropriate reliance on technology.”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lu and Yin 2021
In the citing paper
“As a result, users may over-rely on models that reinforce their own biases or under-rely on models that might challenge or correct those biases (Lu and Yin 2021).”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lu and Yin 2021
In the citing paper
“Participants' comments also reflected the strength of their intuition about the outcome (or confidence [59]), which impacted how they engaged with the AI system.”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [59]
In the citing paper
“In this set-up, agreement between a participant's answer and that of the AI system is a commonly studied behavioral measure of reliance and trust [19, 22, 55, 61, 67, 75, 114, 118].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [67]
In the citing paper
“Nevertheless, the field of humancomputer interaction (HCI) has so far considered metacognition mainly in the context of computer science education [107,142].”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [107]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“Theoretical models and frameworks have been proposed to explain human’s reliance on the automation in various situations [12, 19, 20, 31, 57].”
BackgroundHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“on the automation [13, 43], including both over-reliance on the automation when it performs poorly (i.”
BackgroundWhat is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“For example, tutorials can be provided to end-users of ML models to increase their general knowledge of ML, including that ML models are often learned from data, and depending on the quality of the data, ML models may exhibit dierent performance on dierent data [34].”
BackgroundThe “false consensus effect”: An egocentric bias in social perception and attribution processes
Ross, L., Greene, D., & House, P. (1977) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(77)90049-x · Cited by 3,139 · cited 2×
“ation reveals some consistent behavior pattern [33, 39, 46, 50, 63]— for example, the phenomenon of “naive realism” [61] suggests that people often consider their own judgment to be objective reections of reality and tend to discount advice that is more dierent”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“On the conceptual level, our ndings echos that of other work in human-AI collaboration [1, 3, 4, 26] and reiterates that the design and development of ML-based decision aids should not be isolated from the people who will use them.”
BackgroundLeshno, M., Lin, V. Y., Pinkus, A., & Schocken, S. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80131-5 · Cited by 2,121
Research works printed but not linked: 5
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77–91.
- David Dunning. 2014. We are all confident idiots. Pacific Standard 7(2014), 46–54.
- SM Fleming and ND Daw. 2016. Self-evaluation of decision performance: A general Bayesian framework for metacognitive computation. Psychol Rev 124(2016), 1–59.
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th australasian conference on information systems, Vol. 53. Citeseer, 6–8.
- Gary Marks and Norman Miller. 1987. Ten years of research on the false-consensus effect: An empirical and theoretical review.Psychological bulletin 102, 1 (1987), 72.
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217
“Remarkably, individuals can calibrate their dependence on AI even in the absence of explicit accuracy feedback (Lu & Yin, 2021; Wang et al., 2022).”
Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
Romeo, G., & Conti, D. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02422-7 · Cited by 209
“As a result, users may over-rely on models that reinforce their own biases or under-rely on models that might challenge or correct those biases (Lu and Yin 2021).”
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“User makes prediction before model [16, 45, 74, 83, 105, 110, 112], vary system response times [16, 82], outcome feedback to user [10, 11, 21, 45, 108, 111], training phase [21, 57, 61, 83, 112], source of recommendation or local explanation (human or AI) [29, 59, 73], varied model quality [34, 56,…”
BackgroundAI makes you smarter but none the wiser: The disconnect between performance and metacognition
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108779 · Cited by 188
Designing for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
“They may also resort to a positive confirmation heuristic if the AI’s suggestions align with their own speculation [38], which is likely unfounded.”
BackgroundMa, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642671 · Cited by 54
“It plays a pivotal role in decision-making and receptiveness to advice [4], even affecting our willingness to heed AI recommendations [10, 45, 80].”
Self-citationKim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143
“Evidence shows that people’s confdence in themselves signifcantly afects whether they will take AI’s advice [19, 62, 96].”
BackgroundSelf-citationEnhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
“A common way to quantify people’s trust and reliance on AI recommendations is to measure how frequently people’s final decisions agree with the AI model’s decision recommendations [6, 7, 11, 15, 35, 53, 59, 80, 83, 98, 109–112].”
BackgroundSelf-citationInkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132
“Lu & Yin ind that human reliance on a model is signiicantly inluenced by their agreement with an AI during task instances when they have high conidence in their own decision [39].”
BackgroundBreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102285 · Cited by 125
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
“Existing work has extensively explored how confidence [19, 130], risk perception [38, 40], performance feedback [71, 92], and explanations [32, 94, 119] can affect human-AI decision making.”
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
“Topics like performance feedback [2, 55], risk perception [27, 34], uncertainty [77] and confidence [10, 79, 87] of machine learning models, impact of explanations [3, 46] have been extensively studied in human-AI decision making.”
BackgroundZhang, S., Yu, J., Xu, X., Yin, C., Lu, Y., Yao, B., Tory, M., Padilla, L. M., Caterino, J., Zhang, P., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642343 · Cited by 99
“Previous research suggests that in sepsis diagnosis, the most important challenge is not to use XAI methods to explain the model outcome, but to use methods that are consistent with the doctor’s cognition to establish trust between the doctor and the model [69, 76].”
MethodRechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97
“Indeed, prior research has shown that trust in peer assessment is lowered when prior expectations are not met [30], and within the context of trust in ML and its predictions, in the absence of information about an ML model’s performance, people’s reliance on the model is dependent on how often the model’s predictions…”
BackgroundSelf-citationChiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 56
“As the primary metric of reliance, for each individual or group, we measured their overall reliance on the AI model using the fraction of formal tasks in which their fnal prediction was the same as the AI model’s prediction, following similar methods used in previous studies on human reliance on AI [84, 132].”
BackgroundMethodSelf-citationWill You Accept the AI Recommendation? Predicting Human Behavior in AI-Assisted Decision Making
Wang, X., Lu, Z., & Yin, M. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3512240 · Cited by 49
“…human-computer interaction community which empirically identifies a wide range of factors that can influence people’s trust in AI, such as the AI model’s accuracy [20, 35, 45], the model’s confidence in the decision-making task [35, 47], and the level of agreement between the human and the AI [26].”
BackgroundSelf-citationLu, Z., Li, P., Wang, W., & Yin, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555562 · Cited by 54
“In recent years, a rich set of empirical research has been conducted in understanding whether and how the presence of recommendations from an AI model can influence people’s decision-making ability [9, 14, 27, 29, 41, 53, 56, 57].”
BackgroundSelf-citationCao, S., Liu, A., & Huang, C.-M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637318 · Cited by 41
“Switch captures the number of times or frequency at which an individual switched their response such that their final response matched the AI suggestion, given that the user’s initial response did not match the AI suggestion [13, 57, 60, 105].”
BackgroundMethodInfluentialYou Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85
“Prior work [5, 28, 52] studied AI systems that have a single fixed level of expertise or performance relative to the human.”
BackgroundUncalibrated Models Can Improve Human-AI Collaboration
Vodrahalli, K., Gerstenberg, T., & Zou, J. (2022) · neural information processing systems · doi:10.52202/068431-0289 · Cited by 84
Understanding the Evolvement of Trust Over Time within Human-AI Teams
Duan, W., Zhou, S., Scalia, M. J., Yin, X., Weng, N., Zhang, R., Freeman, G., McNeese, N., Gorman, J., & Tolston, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687060 · Cited by 45
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making
Lu, Z., Wang, D., & Yin, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653708 · Cited by 39
“…performance-related information is absent, it is found that people may utilize other heuristics or cues to determine how to rely on the AI model, including how frequently the AI recommendations align with their own judgments [58] and their mental models of the AI model’s error boundaries [3, 4].”
Self-citationTowards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75
“, how much people’s decisions rely on or are influenced by the AI’s), such as acceptance of model suggestions [13, 16, 22, 26, 35, 80, 81, 90, 94, 96, 101, 143, 152, 153, 155], likelihood to switch [58, 96, 101, 109, 152, 155], weight of model advice [95, 111], choice to use the”
BackgroundInfluentialSpatola, N. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100099 · Cited by 77
Understanding User Reliance on AI in Assisted Decision-Making
Cao, S., & Huang, C.-M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555572 · Cited by 71
“…that a person’s initial response was different from the AI, the frequency at which the person revised their initial response for their final response to match that of the AI) [44, 47, 85], and the number of trials at which a person authorized the AI agent to make predictions on behalf of them [15].”
BackgroundMethodPerformance and Metacognition Disconnect when Reasoning in Human-AI Interaction
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2024) · arXiv · doi:10.48550/arxiv.2409.16708 · Cited by 71
He, G., Buijsman, S., & Gadiraju, U. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610067 · Cited by 39
“[41] found similar patterns for both objective reliance and subjective trust when feedback on model performance is limited.”
BackgroundCapable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making
Tolmeijer, S., Christen, M., Kandul, S., Kneer, M., & Bernstein, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517732 · Cited by 70
“Like trust, reliance in our setting is also likely to be influenced by heuristics, especially since there is less information available [86], and limited time to process information [131].”
BackgroundImpact of Model Interpretability and Outcome Feedback on Trust in AI
Ahn, D., Almaatouq, A., Gulabani, M., & Hosanagar, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642780 · Cited by 69
“Global interpretability clarifies which variables are important to the model’s decision-making in aggregate, while local interpretability clarifies which variables are important for a specific decision [73].”
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“First, we measured the agreement be-tween a participant’s answer and that of Theta; this is a commonly used behavioral measure of reliance [14, 19, 69, 76, 84, 89, 128, 132].”
Visser, R., Peters, T. M., Scharlau, I., & Hammer, B. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101357 · Cited by 58
“…al., 2020), reliance (Buçinca et al., 2021; Bussone et al., 2015; Chen et al., 2023; He et al., 2023; Klingbeil et al., 2024; Körber et al., 2018; Lu & Yin, 2021; Miller et al., 2016; Morrison et al., 2024; Naiseh et al., 2023; Schemmer et al., 2023; Schmidt & Biessmann, 2019; Vasconcelos et…”
Exploring the effects of human-centered AI explanations on trust and reliance
Scharowski, N., Perrig, S. A. C., Svab, M., Opwis, K., & Brühlmann, F. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1151150 · Cited by 57
Hupont, I., Tolan, S., Güneş, H., & Gómez, E. (2022) · Scientific Reports · doi:10.1038/s41598-022-14981-6 · Cited by 56
Ma, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2403.09552 · Cited by 56
“It plays a pivotal role in decision-making and receptiveness to advice [4], even affecting our willingness to heed AI recommendations [10, 45, 80].”
Self-citationAI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventions
Schoeffer, J., Jakubik, J., Vössing, M., Kühl, N., & Satzger, G. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15873 · Cited by 56
“…to this as the team performance of human and AI being greater than the performance of the individual agents alone (de Visser & Parasuraman, 2011; Hemmer et al., 2022). accuracy (Lai et al., 2023; Kim et al., 2023) or the human reliance on AI recommendations (Bu¸cinca et al., 2021; Lu & Yin, 2021).”
BackgroundTrust Development and Repair in AI-Assisted Decision-Making during Complementary Expertise
Pareek, S., Velloso, E., & Gonçalves, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658924 · Cited by 56
You’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53
“In addition, people’s first impression and mental model of an ML model [2, 26], the model’s confidence and interpretability [22, 29, 32], and the consistency between the model and humans in both their decisions and rationales [15, 33] are all shown to impact people’s reliance on the model.”
BackgroundSelf-citationAI-Assisted Decision-making: a Cognitive Modeling Approach to Infer Latent Reliance Strategies
Tejeda, H., Kumar, A., Smyth, P., & Steyvers, M. (2022) · Computational Brain & Behavior · doi:10.1007/s42113-022-00157-y · Cited by 53
“However, feedback is not always possible in real-world scenarios (Lu and Yin, 2021).”
BackgroundModeling Human Trust and Reliance in AI-Assisted Decision Making: A Markovian Approach
Li, Z., Lu, Z., & Yin, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25748 · Cited by 58
“2019a,b), and the level of humanmodel agreement (Lu and Yin 2021).”
BackgroundSelf-citationOffline Reinforcement Learning for Adaptive Support in AI-Assisted Decision-Making
Buçinca, Z., Swaroop, S., Paluch, A. E., Murphy, S. A., & Gajos, K. Z. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3815418 · Cited by 54
“However, mounting evidence shows that decision-makers struggle to incorporate AI recommendations into their decisions, often either over-relying or under-relying on AI, even when explanations are provided [5, 8, 13, 23, 43, 76, 80, 88, 91, 98, 98, 117].”
Göndöcs, D., Horváth, S., & Dörfler, V. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103622 · Cited by 52
Mitigating knowledge imbalance in AI-advised decision-making through collaborative user involvement
Gomez, C., Unberath, M., & Huang, C.-M. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102977 · Cited by 50
Erlei, A., Das, R., Meub, L., Anand, A., & Gadiraju, U. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517734 · Cited by 33
“Consequently, there has been a lot of behavioral research focusing on how humans make use of or rely algorithmic and AI-systems [42, 46, 114], or cooperate with them [41, 66, 117, 149, 168].”
BackgroundRegulating Government AI and the Challenge of Sociotechnical Design
Engstrom, D. F., & Haim, A. (2023) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-120522-091626 · Cited by 48
Duan, W., Zhou, S., Scalia, M. J., Freeman, G., Gorman, J. C., Tolston, M. T., McNeese, N. J., & Funke, G. J. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108560 · Cited by 48
Ahn, D., Almaatouq, A., Gulabani, M., & Hosanagar, K. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3964332 · Cited by 47
Measuring and Mitigating (Over)reliance to Build Human-Compatible AI
Ibrahim, L., Collins, K. M., Kim, S. S. Y., Reuel, A., Lamparth, M., Feng, K., Ahmad, L., Soni, P., Kattan, A. E., Stein, M., et al. (2026) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v9i2.43225 · Cited by 45
To Trust or Distrust Trust Measures: Validating Questionnaires for Trust in AIPreprint
Scharowski, N., Perrig, S. A. C., Aeschbach, L. F., von Felten, N., Opwis, K., Wintersberger, P., & Brühlmann, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.00582 · Cited by 45
“Meanwhile, reliance, as a behavior (Lee & See, 2004), can be assessed using more objective observational methods such as analyzing changes in an individual’s behavior after being presented with an AI recommendation (e.g., switch ratios (Lu & Yin, 2021; Yin et al., 2019)).”
MethodSupporting High-Uncertainty Decisions through AI and Logic-Style Explanations
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584080 · Cited by 30
“For the evaluation metrics, we decided to use the reliance [18, 30, 43] subjective measure, which is frequently used in the literature to collect information on users’ trust in the ML model.”
BackgroundMethodEckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 45
“However, none of the articles attends to the question of whether the provided information is understandable to the user and if they interpret it correctly, leaving it open Finally, roughly one third of the articles do not employ any transparency mechanisms (n=25) [e.g. 21, 50, 81].”
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“Prior work has also examined how human trust and reliance on AI systems is affected by different design choices through empirical studies [16, 52, 67, 76].”
BackgroundEffect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment
Pareek, S., van Berkel, N., Velloso, E., & Goncalves, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686922 · Cited by 42
Chiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42
“the ML model’s accuracy [48, 66, 68], confidence [48, 70], and the level of agreement between the ML model and one’s own judgment or reasoning [42, 71], are shown to affect howmuch a person will be willing to rely on an ML model.”
Self-citationSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“To this end, researchers and practitioners have explored the use of explanation methods [66, 94, 126], interventions such as tutorials [25, 84] and cognitive forcing functions [17] to foster appropriate reliance on AI systems with varying degrees of success.”
BackgroundRobot-Assisted Decision-Making: Unveiling the Role of Uncertainty Visualisation and Embodiment
Schömbs, S., Pareek, S., Gonçalves, J., & Johal, W. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642911 · Cited by 41
“Future applications have to carefully consider the type of visualisation and the expected confdence range to best support human decision-making in robot-assisted decision tasks.”
BackgroundWang, X., & Yin, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581366 · Cited by 38
“Although widely used in the literature [7, 15, 23, 52, 55, 57, 68, 95], we acknowledge that this metric may reect the natural agreement between people’s independent decisions and the AI recommendations to some extent.”
BackgroundSelf-citationExploreGen: Large Language Models for Envisioning the Uses and Risks of AI Technologies
Herdel, V., Šćepanović, S., Bogucka, E. P., & Quercia, D. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31660 · Cited by 41
From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis
Li, Z., Zhu, H., Lu, Z., Xiao, Z., & Yin, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713133 · Cited by 38
Self-citationHow Accurate Does It Feel? – Human Perception of Different Types of Classification Mistakes
Papenmeier, A., Kern, D., Hienert, D., Kammerer, Y., & Seifert, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501915 · Cited by 23
“Accordingly, research in HCML focuses on users’ evaluations of machine learning systems, including aspects such as users’ reliance [31] on the system, or trust [40, 68] in the system, the perceived explainability [51], interpretability [44], or fairness [20, 32] of the system, and the experienced or perceived accuracy…”
BackgroundTo Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
Bo, J. Y., Wan, S., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714097 · Cited by 36
Effects of LLM-based Search on Decision Making: Speed, Accuracy, and Overreliance
Spatharioti, S. E., Rothschild, D., Goldstein, D. G., & Hofman, J. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714082 · Cited by 36
“The problem of overreliance on AI has been a concern both before [31, 34, 34, 39, 43, 50] and after the current interest in LLM-generated errors [21, 38, 49].”
To Trust or Distrust AI: A Questionnaire Validation Study
Scharowski, N., Perrig, S. A. C., von Felten, N., Aeschbach, L. F., Opwis, K., Wintersberger, P., & Brühlmann, F. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732025 · Cited by 37
“Meanwhile, reliance, as a behavior [48], can be assessed using more objective observational methods such as analyzing changes in an individual’s behavior (e.g., switch ratios [55, 100]).”
Decoding AI’s Nudge: A Unified Framework to Predict Human Behavior in AI-Assisted Decision Making
Li, Z., Lu, Z., & Yin, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28872 · Cited by 34
Self-citationShould I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Kühl, N., Benz, C., & Satzger, G. (2022) · arXiv · doi:10.48550/arxiv.2204.06916 · Cited by 33
“Other researchers go one step further and define inappropriate reliance as under-or over-reliance [18, 32, 37].”
BackgroundComparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task
Xu, C., Lien, K.-C., & Höllerer, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581282 · Cited by 31
“Studying user’s trust and appropriate or inappropriate reliance on AI [7, 30, 35, 41, 59, 63] is another important direction.”
BackgroundEngaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-MakingPreprint
Chen, Z., Luo, Y., & Sra, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.16627 · Cited by 30
Human–AI collaborative multi-modal multi-rater learning for endometriosis diagnosis
Wang, H., Butler, D., Zhang, Y., Avery, J., Knox, S., Ma, C., Hull, M. L., & Carneiro, G. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad997e · Cited by 28
“Human-AI Collaboration (HAIC) integrates the unique strengths of human experts and AI systems, resulting in improved model capabilities and performance when compared to standalone AI systems [14, 15].”
BackgroundKellogg, K. C., Lifshitz‐Assaf, H., Randazzo, S., Mollick, E., Dell’Acqua, F., McFowland, E., Candelon, F., & Lakhani, K. R. (2025) · Information and Organization · doi:10.1016/j.infoandorg.2025.100559 · Cited by 27
Explainability for Transparent Conversational Information-Seeking
Łajewska, W., Spina, D., Trippas, J. R., & Balog, K. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657768 · Cited by 26
“Such limitations can be revealed using natural language utterances [42], using analogy [20], incorporating user interface elements [26, 35], or by providing a granular scale of the system’s confidence [48].”
BackgroundMethodQian, P., & Xie, X.-Z. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2526580 · Cited by 26
“This may be because “Occasionally or Basically don’t use” users, lacking sufficient experience, have unclear perceptions of AI capabilities and thus tend to rely on AI outputs to avoid uncertainty risks (Lu & Yin, 2021).”
Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge
Kawakami, A., Guerdan, L., Cheng, Y., Glazko, K. S., Lee, M. L., Carter, S., Aréchiga, N., Zhu, H., & Holstein, K. (2023) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3582269.3615595 · Cited by 25
“For example, participants aligned less with the model proxy following training even in the Practice condition, when participants did not receive explicit feedback on the accuracy of the AFST with respect to its targeted proxy (see [41]).”
MethodMa, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.21183 · Cited by 25
To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems
He, G., Bharos, A., & Gadiraju, U. (2024) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3648188.3675130 · Cited by 24
“Such a reliance pattern has been defined as appropriate reliance [26, 33, 43, 51].”
Algorithmic loafing and mitigation strategies in Human-AI teams
Inuwa-Dutse, I., Toniolo, A., Weller, A., & Bhatt, U. (2023) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2023.100024 · Cited by 24
“…detrimental, such as biased decisions, to team performance, but from explainable artificial intelligence (XAI) literature there are other factors (Lu & Yin, 2021; Vodrahalli, Gerstenberg, & Zou, 2021) that could, on the other hand, support team performance and potentially mitigate the effects of…”
BackgroundMeasuring the effect of mental workload and explanations on appropriate AI reliance using EEG
Zhang, Z. T., Argın, S. K., Bilen, M. B., Urgun, D., Deniz, S. M., Liu, Y., & Hassib, M. E. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2431055 · Cited by 24
“This is more in line with the notion of difficulty studied by Lu and Yin [22] or Papenmeier et al. [23], where difficulty is not caused by a lack of domain knowledge or the required effort, but rather by the presence of more than a single plausible answer.”
He, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
“Existing human-AI collaboration literature has provided potential solutions for such problems, ranging from performance feedback interventions [34] to agreement-in-confidence heuristic [65, 79].”
Bridging Prediction and Intervention Problems in Social SystemsPreprint
Liu, L. T., Raji, I. D., Zhou, A., Guerdan, L., Hullman, J. R., Malinsky, D., Wilder, B., Zhang, S., Adam, H., Coston, A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.05216 · Cited by 23
Reilly, D., Taylor, M. J., Fergus, P., Chalmers, C., & Thompson, S. L. (2022) · IEEE Access · doi:10.1109/access.2022.3187287 · Cited by 20
“Many researchers have noticed that heuristic procedures (e.g. sampling many special cases by trial and error) often lead to greater insights and stimulus by engaging thought processes [12].”
BackgroundStrategic Adversarial Attacks in AI-assisted Decision Making to Reduce Human Trust and Reliance
Lu, Z., Li, Z., Chiang, C.-W., & Yin, M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/337 · Cited by 19
“The long line of empirical research in the human-computer interaction community on understanding humans’ interactions with AI in AI-assisted decision making [ Lu and Yin, 2021; Chong et al. , 2022; Nourani et al. , 2020 ] , however, suggests that likely, not all attacks are created equal.”
BackgroundMethodInfluentialSelf-citationSecurity and Privacy Challenges of AIGC in Metaverse: A Comprehensive Survey
Zhang, S., Li, H., Sun, K., Chen, H., Wang, Y., & Li, S. (2025) · ACM Computing Surveys · doi:10.1145/3729419 · Cited by 19
He, G., Hemmer, P., Vössing, M., Schemmer, M., & Gadiraju, U. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.10909 · Cited by 20
“Existing empirical studies [9, 57, 70] and theoretical frameworks related to user trust [61] and reliance behavior [93] highlight that users of an AI system need to identify when an AI system is accurate to rely on and when it is inaccurate and should be overridden.”
Building Appropriate Mental Models: What Users Know and Want to Know about an Agentic AI Chatbot
Brachman, M., Kunde, S., Miller, S., Fucs, A., Dempsey, S., Jabbour, J., & Geyer, W. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712071 · Cited by 18
Learning to Complement and to Defer to Multiple Users
Zhang, Z., Ai, W., Wells, K., Rosewarne, D., Do, T.-T., & Carneiro, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72992-8_9 · Cited by 20
Expl(Ai)Ned: The Impact of Explainable Artificial Intelligence on Cognitive ProcessesPreprint
Bauer, K., von Zahn, M., & Hinz, O. F. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3872711 · Cited by 18
Monitoring Human Dependence On AI Systems With Reliance DrillsPreprint
Hunter, R., Moulange, R., Bernardi, J., & Stein, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.14055 · Cited by 18
“On examination the lungs are clear.”
Beauty and the Bias: Exploring the Impact of Attractiveness on Multimodal Large Language Models
Gulati, A., Moreno D’Incà, Sebe, N., Lepri, B., & Oliver, N. (2025) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v8i2.36619 · Cited by 18
“Since prompt order impacts the behavior of LLMs [Lu and Yin(2021)] and MLLMs [Shi et al.(2024), Chen et al.(2024), Liu et al.(2025)], each image was evaluated under all possible orderings of the options and the average across all these orderings was used to evaluate the model, as detailed in…”
To Solicit or Not to Solicit? Impact of AI Assistance Delivery Mechanisms on Decision-Making
Nair, P., Gressel, G., Anand, M. N., & Achuthan, K. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2536617 · Cited by 17
“In the absence of such information, individuals utilized other heuristics including the frequency of alignment between the AI assistance and their own judgments, and their mental models of the AI model ’ s error boundaries (Bansal, Nushi, Kamar, Lasecki, et al., 2019; Lu & Yin, 2021).”
Workshop on Trust and Reliance in AI-Human Teams (TRAIT)
Bansal, G., Smith, A., Buçinca, Z., Wu, T., Holstein, K., Hullman, J. R., & Stumpf, S. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3503704 · Cited by 17
“renewed interest in human-AI trust within the AI and HCI community [12, 16, 18, 21, 23] motivated by directions and approaches”
Underspecified Human Decision Experiments Considered Harmful
Hullman, J. R., Kale, A. M., & Hartline, J. D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714063 · Cited by 16
On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making
Schoeffer, J., Jakubik, J., Voessing, M., Kuehl, N., & Satzger, G. (2023) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia230074 · Cited by 15
Designing Behavior-Aware AI to Improve the Human-AI Team Performance in AI-Assisted Decision Making
Mahmood, S. H. A., Lu, Z., & Yin, M. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/344 · Cited by 15
“Indeed, empirical studies have shown that humans’ adoption of AI recommendations is often influenced by their cognitive biases [Lu and Yin, 2021; Rastogi et al. , 2022; Bertrand et al. , 2022].”
BackgroundSelf-citationPassi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 15
StorySage: Conversational Autobiography Writing Powered by a Multi-Agent Framework
Talaei, S., Li, M., Grover, K., Hippler, J. K., Yang, D., & Saberi, A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747681 · Cited by 15
Radensky, M., Séguin, J. A., Lim, J. S., Olson, K., & Geiger, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594043 · Cited by 14
“Agreement rate determines how often the human’s final decision is in agreement with the AI, while the switch rate determines how often the human agrees with the AI after having made a different initial decision [33, 40].”
BackgroundCollaborative Artificial Intelligence Needs Stronger Assurances Driven by Risks
Adigun, J. G., Camilli, M., Felderer, M., Giusti, A., Matt, D. T., Perini, A., Russo, B., & Susi, A. (2022) · Computer · doi:10.1109/mc.2021.3131990 · Cited by 14
“Indeed, there exist increasing concerns on how can people determine how much they could rely on ML models and their decision-making in safety critical domains [11].”
BackgroundDesigning for Continuous Interaction with Artificial Intelligence Systems
Wintersberger, P., van Berkel, N., Fereydooni, N., Tag, B., Glassman, E. L., Buschek, D., BLANDFORD, A. E., & Michahelles, F. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3516409 · Cited by 13
“For example, experiments frequently assess compliance and reliance rates to assess if users comply with AI or automated systems [13, 18, 20].”
Background
Metadata
- Authors
- Lu, Z., & Yin, M.
- Affiliations
- Purdue University West Lafayette (US)
- ORCID
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-16
- OpenAlex
- W3163667721
- Cited by
- 131
- References
- 60
- Citations by year
- 2026: 16 · 2025: 30 · 2024: 35 · 2023: 24 · 2022: 22 · 2021: 4
- FWCI
- 36.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- machine learning models, performance feedback, trust in AI, human-AI agreement, heuristics, AI-assisted decision-making, decision aids
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.