Designing for Appropriate Reliance: The Roles of AI Uncertainty Presentation, Initial User Decision, and User Demographics in AI-Assisted Decision-Making
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Appropriate reliance is critical to achieving synergistic human-AI collaboration. For instance, when users over-rely on AI assistance, their human-AI team performance is bounded by the model's capability. This work studies how the presentation of model uncertainty may steer users' decision-making toward fostering appropriate reliance. Our results demonstrate that showing the calibrated model uncertainty alone is inadequate. Rather, calibrating model uncertainty and presenting it in a frequency format allow users to adjust their reliance accordingly and help reduce the effect of confirmation bias on their decisions. Furthermore, the critical nature of our skin cancer screening task skews participants' judgment, causing their reliance to vary depending on their initial decision. Additionally, step-wise multiple regression analyses revealed how user demographics such as age and familiarity with probability and statistics influence human-AI collaborative decision-making. We discuss the potential for model uncertainty presentation, initial user decision, and user demographics to be incorporated in designing personalized AI aids for appropriate reliance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 7×
“However, presenting explanations of the AI decisions did not appear to reduce inappropriate reliance in its users [2, 49, 54, 70].”
BackgroundResultInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 2×
“Other metrics of reliance, such as weight of advice [56, 70], how fast an AI suggestion is accepted by the user [22], user’s self-reported level of reliance on AI [14, 15], and the relative length of user’s gaze duration on the AI suggestion [14], have also been used to estimate user reliance on AI…”
BackgroundMethodManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
The Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · cited 1×
“In interaction paradigms where the user provides an initial decision before receiving the AI suggestion, the “switch” metric is commonly used.”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 4×
“…model uncertainty presentation, there are other types of interventions that may also help users more appropriately adjust their reliance, including local [2, 78, 107] and global [80, 103] explanations, cognitive forcing functions [8], model performance-related information [75, 105], and so on.”
BackgroundMethodInfluentialAttachment and trust in artificial intelligence
Gillath, O., Ai, T., Branicky, M. S., Keshmiri, S., Davison, R. B., & Spaulding, R. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106607 · Cited by 448 · cited 2×
“This finding is inconsistent with prior work that found being younger is positively correlated with trust in AI agents [31], which is highly correlated with reliance and adoption of AI advice [8, 95].”
MethodResultThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 5×
“Thus, while ethical and legal concerns remain, AI agents are called to be used to assist humans in critical decisions [12, 49].”
BackgroundInfluentialProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Intuitive evaluation of likelihood judgment producers: evidence for a confidence heuristic
Price, P. C., & Stone, E. R. (2004) · Journal of Behavioral Decision Making · doi:10.1002/bdm.460 · Cited by 272 · cited 1×
“Previous work found that people tend to equate confidence with competence [88] and believe that more confident advisors possess more knowledge [71], suggesting that participants would prefer more confident advice.”
BackgroundChong, L., Zhang, G., Goucher-Lambert, K., Kotovsky, K., & Cagan, J. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107018 · Cited by 262 · cited 1×
“This finding is consistent with the results of prior work [16], which found that users with higher initial confidence are less likely to adopt AI suggestions.”
ResultAffective Processes in Human–Automation Interactions
Merritt, S. M. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811411912 · Cited by 260 · cited 2×
“Switch captures the number of times or frequency at which an individual switched their response such that their final response matched the AI suggestion, given that the user’s initial response did not match the AI suggestion [13, 57, 60, 105].”
BackgroundMethodAdaptive trust calibration for human-AI collaboration
Okamura, K., & Yamada, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229132 · Cited by 239 · cited 1×
“…over-reliance and under-reliance, i.e., requesting automation when the reliability of the AI is lower than that of the user (over-reliance) [65], agreeing with incorrect AI suggestions (over-reliance) [8, 10, 87, 104], disagreeing with correct AI suggestions (under-reliance) [10, 87, 104].”
BackgroundHow to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216 · cited 1×
“Cognitive forcing function, such as delaying the presentation of the AI suggestion to the user or slowing down the users’ decision-making process, was found to significantly reduce user over-reliance on incorrect AI recommendations [8, 74].”
BackgroundWang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 1×
“2.1 Appropriate Reliance in Human-AI Teaming AI agents’ performance on select critical tasks from numerous domains, e.g., recidivism prediction [43] and cancer diagnosis [10, 98, 99], are surpassing that of human experts.”
BackgroundInfluentialHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 1×
“…over-reliance and under-reliance, i.e., requesting automation when the reliability of the AI is lower than that of the user (over-reliance) [65], agreeing with incorrect AI suggestions (over-reliance) [8, 10, 87, 104], disagreeing with correct AI suggestions (under-reliance) [10, 87, 104].”
BackgroundPooch, E. H. P., Ballester, P., & Barros, R. C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-62469-9_7 · Cited by 134
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
Mobile applications in dermatology
Brewer, A. C., Endly, D. C., Henley, J., Amir, M., Sampson, B. P., Moreau, J. F., & Dellavalle, R. P. (2013) · JAMA Dermatology · doi:10.1001/jamadermatol.2013.5517 · Cited by 121
Investigating intelligibility for uncertain context-aware applications
Lim, B. Y., & Dey, A. K. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030168 · Cited by 101
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92
Understanding User Reliance on AI in Assisted Decision-Making
Cao, S., & Huang, C.-M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555572 · Cited by 71 · cited 3×
“Furthermore, people were more confident in their response when they perceived the AI suggestion to be in agreement with them [14].”
BackgroundMethodSelf-citationDo Humans Trust Advice More if it Comes from AI?: An Analysis of Human-AI Interactions
Vodrahalli, K., Daneshjou, R., Gerstenberg, T., & Zou, J. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534150 · Cited by 63
Do People Inherently Dislike Uncertain Advice?
Gaertig, C., & Simmons, J. P. (2018) · Psychological Science · doi:10.1177/0956797617739369 · Cited by 52 · cited 1×
“Prior work supports this hypothesis as studies found that users desire the AI uncertainty information for AI suggestions [24, 26], but often find the presented information difficult to understand [10, 66].”
ResultThe Role of Environmental Predictability and Costs in Relying on Automation
Sutherland, S. C., Harteveld, C., & Young, M. E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702609 · Cited by 12
Cited by
Raees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
“…102, 151, 164, 167, 168, 170], Trust in Automation [22, 40, 60– 63, 112, 127], Affinity for Technology [60–63, 112, 127] Others Understanding [9, 24, 32, 33, 43, 60, 76, 88, 94, 96, 112, 156, 164, 167], Confidence Rating [15, 17, 23, 24, 27, 29, 32, 40, 55, 76, 93, 94, 96–98, 112, 114, 117,…”
InfluentialPe, S., Bergomi, L., Nicora, G., Simonelli, C. A., Kour, P., Diaz, E., Alendal, G., Hernáiz Ferrer, A. I., Corso, V., Bortolotto, C., et al. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8070216
“Examples in healthcare include the works of Cao et al. [35], Bashkirova et al. [36], and Buçinca et al. [37].”
Metadata
- Authors
- Cao, S., Liu, A., & Huang, C.-M.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-04-17
- Volume
- 8
- Issue
- CSCW1
- Pages
- 1-32
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4396230888
- Cited by
- 41
- References
- 36
- FWCI
- 19.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.