Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
This paper addresses an under-explored problem of AI-assisted decision-making: when objective performance information of the machine learning model underlying a decision aid is absent or scarce, how do people decide their reliance on the model? Through three randomized experiments, we explore the heuristics people may use to adjust their reliance on machine learning models when performance feedback is limited. We find that the level of agreement between people and a model on decision-making tasks that people have high confidence in significantly affects reliance on the model if people receive no information about the model’s performance, but this impact will change after aggregate-level model performance information becomes available. Furthermore, the influence of high confidence human-model agreement on people’s reliance on a model is moderated by people’s confidence in cases where they disagree with the model. We discuss potential risks of these heuristics, and provide design implications on promoting appropriate reliance on AI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, confirmation bias and related confirmatory heuristics also contribute to inappropriate reliance on technology.”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lu and Yin 2021
In the citing paper
“As a result, users may over-rely on models that reinforce their own biases or under-rely on models that might challenge or correct those biases (Lu and Yin 2021).”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lu and Yin 2021
In the citing paper
“Participants' comments also reflected the strength of their intuition about the outcome (or confidence [59]), which impacted how they engaged with the AI system.”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [59]
In the citing paper
“In this set-up, agreement between a participant's answer and that of the AI system is a commonly studied behavioral measure of reliance and trust [19, 22, 55, 61, 67, 75, 114, 118].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [67]
In the citing paper
“Nevertheless, the field of humancomputer interaction (HCI) has so far considered metacognition mainly in the context of computer science education [107,142].”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [107]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“Theoretical models and frameworks have been proposed to explain human’s reliance on the automation in various situations [12, 19, 20, 31, 57].”
BackgroundHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“on the automation [13, 43], including both over-reliance on the automation when it performs poorly (i.”
BackgroundWhat is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“For example, tutorials can be provided to end-users of ML models to increase their general knowledge of ML, including that ML models are often learned from data, and depending on the quality of the data, ML models may exhibit dierent performance on dierent data [34].”
BackgroundThe “false consensus effect”: An egocentric bias in social perception and attribution processes
Ross, L., Greene, D., & House, P. (1977) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(77)90049-x · Cited by 3,139
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“On the conceptual level, our ndings echos that of other work in human-AI collaboration [1, 3, 4, 26] and reiterates that the design and development of ML-based decision aids should not be isolated from the people who will use them.”
BackgroundLeshno, M., Lin, V. Y., Pinkus, A., & Schocken, S. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80131-5 · Cited by 2,121
Deep content-based music recommendation
van den Oord, A., Dieleman, S., & Schrauwen, B. (2013) · Ghent University Academic Bibliography (Ghent University) · Cited by 1,425
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“Theoretical models and frameworks have been proposed to explain human’s reliance on the automation in various situations [12, 19, 20, 31, 57].”
BackgroundZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Advances in Experimental Social Psychology, Vol. 7.
Fink, E. L., & Berkowitz, L. (1976) · Contemporary Sociology · doi:10.2307/2062988 · Cited by 819
Evaluation of clustering algorithms for financial risk analysis using MCDM methods
Kou, G., Peng, Y., & Wang, G. (2014) · Information Sciences · doi:10.1016/j.ins.2014.02.137 · Cited by 764
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“In the context of AI-assisted decision-making, the ultimate goal is to optimize the human-AI joint team decision-making performance [5, 26], and achieving this goal requires humans and AI to complement each other.”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“On the other hand, while a number of studies have shown the promise that explanations of an ML model enhance people’s understanding of the model and therefore lead to more appropriate reliance on the model [30, 48, 62], the impact of model intelligibility and transparency on reliance also seems to vary across decision…”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Receiving other people’s advice: Influence and benefit
Yaniv, I. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2003.08.002 · Cited by 652
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527
Self-citationGender Differences in Mate Selection: Evidence From a Speed Dating Experiment
Fisman, R., Iyengar, S. S., Kamenica, E., & Simonson, I. (2006) · The Quarterly Journal of Economics · doi:10.1162/qjec.2006.121.2.673 · Cited by 522
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 2×
“Indeed, previous research has shown that people adjust their reliance on an ML model based on its accuracy [30, 64], and correctness feedback on the level of individual decision-making task further helps people build “mental models” of the ML model by understanding when the model is likely to err [3, 4].”
BackgroundBaeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432
Subjective Construal, Social Inference, and Human Misunderstanding
Griffin, D. W., & Ross, L. (1991) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60333-0 · Cited by 404 · cited 1×
“that people exhibit a degree of “naive realism” [23, 61], that is, they tend to believe they perceive the world objectively and people who”
BackgroundHumans: Still Vital After All These Years of Automation
Parasuraman, R., & Wickens, C. D. (2008) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x312198 · Cited by 401 · cited 1×
“The rapid development of automation over the past decades has inspired active research in understanding reliance on automation [10, 44, 49].”
BackgroundThe Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 1×
“More recently, with the widespread usage of machine learning (ML) based technologies in aiding human decision-making, a new line of research examining human’s reliance on ML emerges [22, 55, 64].”
Background(2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 390
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 4×
“Indeed, previous research has shown that people adjust their reliance on an ML model based on its accuracy [30, 64], and correctness feedback on the level of individual decision-making task further helps people build “mental models” of the ML model by understanding when the model is likely to err [3, 4].”
BackgroundMethodInfluentialImproving Content-based and Hybrid Music Recommendation using Deep Learning
Wang, X., & Wang, Y. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654940 · Cited by 367
Racial faces in the wild: Reducing racial bias by information maximization adaptation network
Wang, M., Deng, W., Hu, J., Tao, X., & Huang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00078 · Cited by 353 · cited 1×
“The usage of such heuristics may raise additional risks if the ML model’s performance has systematic discrepancy on dierent data, which in fact is not rare in real-world applications of ML models [6, 42, 59].”
BackgroundThe detrimental effects of power on confidence, advice taking, and accuracy
See, K. E., Morrison, E. W., Rothman, N. B., & Soll, J. B. (2011) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2011.07.006 · Cited by 338
Automation and Human Performance: Theory and Applications
(2018) · Routledge eBooks · doi:10.1201/9781315137957 · Cited by 307
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 1×
“On the other hand, while a number of studies have shown the promise that explanations of an ML model enhance people’s understanding of the model and therefore lead to more appropriate reliance on the model [30, 48, 62], the impact of model intelligibility and transparency on reliance also seems to vary across decision…”
BackgroundUpdates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 2×
“Indeed, previous research has shown that people adjust their reliance on an ML model based on its accuracy [30, 64], and correctness feedback on the level of individual decision-making task further helps people build “mental models” of the ML model by understanding when the model is likely to err [3, 4].”
BackgroundOn the Independence of Compliance and Reliance: Are Automation False Alarms Worse Than Misses?
Dixon, S. R., Wickens, C. D., & McCarley, J. S. (2007) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872007x215656 · Cited by 265
Affective Processes in Human–Automation Interactions
Merritt, S. M. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811411912 · Cited by 260
Galindo, J., & Tamayo, P. (2000) · Computational Economics · doi:10.1023/a:1008699112516 · Cited by 254
Factors affecting the acceptance of expert advice
van Swol, L. M., & Sniezek, J. A. (2005) · British Journal of Social Psychology · doi:10.1348/014466604x17092 · Cited by 224
The effects of example-based explanations in a machine learning interface
Cai, C. J., Jongejan, J., & Holbrook, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302289 · Cited by 220 · cited 1×
“On the other hand, while a number of studies have shown the promise that explanations of an ML model enhance people’s understanding of the model and therefore lead to more appropriate reliance on the model [30, 48, 62], the impact of model intelligibility and transparency on reliance also seems to vary across decision…”
BackgroundHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 1×
“On the other hand, while a number of studies have shown the promise that explanations of an ML model enhance people’s understanding of the model and therefore lead to more appropriate reliance on the model [30, 48, 62], the impact of model intelligibility and transparency on reliance also seems to vary across decision…”
BackgroundRisk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learning
Hussein, S., Cao, K., Song, Q., & Bagci, U. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-59050-9_20 · Cited by 161
Chancey, E. T., Bliss, J. P., Yamani, Y., & Handley, H. A. H. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816682648 · Cited by 144
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129 · cited 1×
“This includes raising people’s awareness of biases in their own decision-making [2, 24], designing methods to allow people assess their own independent decision-making performance (e.”
MethodTowards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126 · cited 1×
“The usage of such heuristics may raise additional risks if the ML model’s performance has systematic discrepancy on dierent data, which in fact is not rare in real-world applications of ML models [6, 42, 59].”
BackgroundGao, J., & Lee, J. D. (2006) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2005.855783 · Cited by 124 · cited 1×
“Theoretical models and frameworks have been proposed to explain human’s reliance on the automation in various situations [12, 19, 20, 31, 57].”
BackgroundExpert-augmented machine learning
Gennatas, E. D., Friedman, J. H., Ungar, L. H., Pirracchio, R., Eaton, E., Reichmann, L. G., Interian, Y., Luna, J. M., Simone, C. B., Auerbach, A., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1906831117 · Cited by 120
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119
Two to Tango: Effects of Collaboration and Disagreement on Dyadic Judgment
Minson, J. A., Liberman, V., & Ross, L. (2011) · Personality and Social Psychology Bulletin · doi:10.1177/0146167211410436 · Cited by 117
Associative judgment and vector space semantics.
Bhatia, S. (2017) · Psychological Review · doi:10.1037/rev0000047 · Cited by 107
Nourani, M., Kabir, S., Mohseni, S., & Ragan, E. D. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5284 · Cited by 100
Understanding reliance on automation: effects of error type, error distribution, age and experience
Sanchez, J., Rogers, W. A., Fisk, A. D., & Rovira, E. (2014) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2011.611269 · Cited by 92
A framework for explaining reliance on decision aids
van Dongen, K., & van Maanen, P.-P. (2012) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2012.10.018 · Cited by 91
The perceptual and social components of metacognition.
Pescetelli, N., Rees, G. E., & Bahrami, B. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000180 · Cited by 72
Naïve realism and capturing the “wisdom of dyads”
Liberman, V., Minson, J. A., Bryan, C. J., & Ross, L. D. (2011) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2011.10.016 · Cited by 59
Gao, J., Lee, J. D., & Zhang, Y. (2006) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2006.01.013 · Cited by 53
Bayesian-Assisted Inference from Visualized Data
Kim, Y., Kayongo, P., Grunde-McLaughlin, M., & Hullman, J. R. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028984 · Cited by 52
Dissonance Between Human and Machine Understanding
Zhang, Z., Singh, J., Gadiraju, U., & Anand, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359158 · Cited by 50
Misuse and Disuse of Automated AIDS
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (1999) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129904300345 · Cited by 43
Hit or Miss: What Leads Experts to Take Advice for Long-Term Judgments?
Ecken, P., & Pibernik, R. (2016) · Management Science · doi:10.1287/mnsc.2015.2219 · Cited by 37
Predictive Multiplicity in Classification
Marx, C. T., Calmon, F. P., & Ustun, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06677 · Cited by 28
The role of decision confidence in advice-taking and trust formationPreprint
Pescetelli, N., & Yeung, N. (2019) · doi:10.31234/osf.io/sjgf7 · Cited by 8
Reliability and age-related effects on trust and reliance of a decision support aid
Sánchez, J., Fisk, A. D., & Rogers, W. A. (2004) · PsycEXTRA Dataset · doi:10.1037/e577272012-066 · Cited by 7
Bayesian-Assisted Inference from Visualized DataPreprint
Kim, Y., Kayongo, P., Grunde-McLaughlin, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.00142 · Cited by 2
Research works printed but not linked: 5
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77–91.
- David Dunning. 2014. We are all confident idiots. Pacific Standard 7(2014), 46–54.
- SM Fleming and ND Daw. 2016. Self-evaluation of decision performance: A general Bayesian framework for metacognitive computation. Psychol Rev 124(2016), 1–59.
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th australasian conference on information systems, Vol. 53. Citeseer, 6–8.
- Gary Marks and Norman Miller. 1987. Ten years of research on the false-consensus effect: An empirical and theoretical review.Psychological bulletin 102, 1 (1987), 72.
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217
Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
Romeo, G., & Conti, D. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02422-7 · Cited by 209
“As a result, users may over-rely on models that reinforce their own biases or under-rely on models that might challenge or correct those biases (Lu and Yin 2021).”
Metadata
- Authors
- Lu, Z., & Yin, M.
- Affiliations
- Purdue University West Lafayette (US)
- ORCID
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-16
- OpenAlex
- W3163667721
- Cited by
- 131
- References
- 60
- Citations by year
- 2026: 16 · 2025: 30 · 2024: 35 · 2023: 24 · 2022: 22 · 2021: 4
- FWCI
- 36.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- machine learning models, performance feedback, trust in AI, human-AI agreement, heuristics, AI-assisted decision-making, decision aids
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.