You’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Open accessAbstract
Decision-making aids powered by machine learning models become increasingly prevalent on the web today. However, when applied to a new distribution of data that is different from the training data (i.e., when covariate shift occurs), machine learning models often suffer from performance degradation and may provide misleading recommendations to human decision-makers. In this paper, we conduct a randomized experiment to investigate how people rely on machine learning models to make decisions under covariate shift. Surprisingly, we find that people rely on machine learning models more when making decisions on out-of-distribution data than in-distribution data. Moreover, while increasing people’s awareness of the machine learning model’s possible performance disparity on different data helps decrease people’s over-reliance on the model under covariate shift, enabling people to visualize the data distributions and the model’s performance does not seem to help. We conclude by discussing the implication of our results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The diverging performance of in-distribution and out-of-distribution highlights the importance of considering distribution shift in empirical studies of human-AI collaboration to inform generalizability.
“The diverging performance of in-distribution and out-of-distribution highlights the importance of considering the efect of distribution shift when conducting empirical studies of human-AI collaboration to inform the generalizablity of results, echoing recent fndings in other studies [23,56].”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [23]
Chiang and Yin found that humans rely on AI more out-of-distribution, where AI performance is worse.
“Chiang and Yin [23] examined human reliance on the model in human-AI decision making and found that surprisingly humans rely on AI more out-of-distribution, where the AI performance is worse.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [23]
The effect of distribution shift has been rarely examined in empirical studies of human-AI collaboration, with a few recent exceptions.
“Despite substantial interest in distribution shift in the AI community, the efect has been rarely examined in empirical studies of human-AI collaboration, with a few recent exceptions [23,56].”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1× · see citing works
“On the high level, this is related to increase the general public’s AI literacy [9, 14], especially on enabling people to recognize that ML models learn from data, and the data used to train the models largely influences the results of the model.”
BackgroundAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · see citing works
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · see citing works
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061 · see citing works
Improving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678 · see citing works
Dataset Shift in Machine Learning
(2008) · doi:10.7551/mitpress/9780262170055.001.0001 · Cited by 1,553 · see citing works
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029 · see citing works
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989 · see citing works
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Dataset Shift in Machine Learning
Adams, N. M. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00624_10.x · Cited by 716 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1× · see citing works
“For example, people are shown to increase their reliance on models with higher levels of accuracy [31].”
BackgroundSelf-citationBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · see citing works
Making sense of recommendations
Yeomans, M., Shah, A., Mullainathan, S., & Kleinberg, J. (2019) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2118 · Cited by 459 · see citing works
Communicating uncertainty about facts, numbers and science
van der Bles, A. M., van der Linden, S., Freeman, A. L. J., Mitchell, J., Galvao, A. B., Zaval, L., & Spiegelhalter, D. J. (2019) · Royal Society Open Science · doi:10.1098/rsos.181870 · Cited by 456 · see citing works
Racial faces in the wild: Reducing racial bias by information maximization adaptation network
Wang, M., Deng, W., Hu, J., Tao, X., & Huang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00078 · Cited by 353 · see citing works
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · see citing works
Self-citationData visualization literacy: Definitions, conceptual frameworks, exercises, and assessments
Börner, K., Bueckle, A. D., & Ginda, M. (2019) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1807180116 · Cited by 298 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 1× · see citing works
“To encourage people to trust/rely on ML models appropriately, previous research has explored the usage of interactive visualization to explain the model predictions [5, 11].”
MethodMachine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation
Sugiyama, M., & Kawanabe, M. (2012) · Cited by 260 · see citing works
Hohman, F., Head, A., Caruana, R., DeLine, R. A., & Drucker, S. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225 · see citing works
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · see citing works
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · see citing works
Self-citationAmes, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression Project
Cock, D. R. D. (2011) · Journal of Statistics Education · doi:10.1080/10691898.2011.11889627 · Cited by 124 · see citing works
Forster, Y., Hergeth, S., Naujoks, F., Krems, J. F., & Keinath, A. (2019) · Information · doi:10.3390/info10040143 · Cited by 85 · see citing works
Data Visualization Literacy: Investigating Data Interpretation Along the Novice–Expert Continuum
Maltese, A. V., Harsh, J. A., & Svetina, D. (2015) · The journal of college science teaching · doi:10.2505/4/jcst15_045_01_84 · Cited by 82 · see citing works
Second Chance for a First Impression? Trust Development in Intelligent System Interaction
Tolmeijer, S., Gadiraju, U., Ghantasala, R., Gupta, A., & Bernstein, A. (2021) · Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3450613.3456817 · Cited by 67 · see citing works
Trust Repair Strategies with Self-Driving Vehicles: An Exploratory Study
Kohn, S. C., Quinn, D., Pak, R., de Visser, E. J., & Shaw, T. H. (2018) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931218621254 · Cited by 64 · see citing works
Dissonance Between Human and Machine Understanding
Zhang, Z., Singh, J., Gadiraju, U., & Anand, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359158 · Cited by 59 · see citing works
Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining
Erlei, A., Nekdem, F., Meub, L., Anand, A., & Gadiraju, U. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7462 · Cited by 50 · see citing works
Data Visualization Literacy: Investigating Data Interpretation along the Novice-Expert Continuum
Maltese, A. V., Harsh, J. A., & Svetina, D. (2015) · The journal of college science teaching · Cited by 49 · see citing works
Predicting the Working Time of Microtasks Based on Workers' Perception of Prediction Errors
Saito, S., Chiang, C.-W., Savage, S., Nakano, T., Kobayashi, T., & Bigham, J. P. (2019) · Human Computation · doi:10.15346/hc.v6i1.10 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu- racy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77-91.
- Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M Drucker. 2019. Gamut: A design probe to understand how data scientists un- derstand machine learning models. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-13.
- Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wort- man Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
Cited by
Sorted by citations
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208 · see citing works
“Chiang and Yin [13] evaluate the impact of tutorials on AR and measure AR through team performance.”
BackgroundHuman-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · see citing works
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132 · see citing works
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107 · see citing works
“Based on the effectiveness of tutorials as interventions in previous HCI literature [8, 9, 45], we designed a tutorial as a means to shed light on the fallibility of AI advice.”
BackgroundMethodInfluentialRechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97 · see citing works
Self-citationCapable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making
Tolmeijer, S., Christen, M., Kandul, S., Kneer, M., & Bernstein, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517732 · Cited by 70 · see citing works
Understanding User Reliance on AI in Assisted Decision-Making
Cao, S., & Huang, C.-M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555572 · Cited by 71 · see citing works
How Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration
Cao, S., Gomez, C., & Huang, C.-M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610068 · Cited by 48 · see citing works
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42 · see citing works
Chiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42 · see citing works
Self-citationLarge Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong
Si, C., Goyal, N., Wu, T., Zhao, C., Feng, S., Daumé Iii, H., & Boyd-Graber, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.81 · Cited by 26 · see citing works
He, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23 · see citing works
Gemalmaz, M. A., & Yin, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534201 · Cited by 10 · see citing works
Self-citationZhang, Z. T., Buchner, F., Liu, Y., & Butz, A. (2024) · Proceedings of Mensch und Computer 2024 · doi:10.1145/3670653.3670660 · Cited by 6 · see citing works
Stop Automating Peer Review Without Rigorous Evaluation
Baumann, J., Pei, J., Koyejo, S., & Hovy, D. (2026) · arXiv · doi:10.48550/arxiv.2605.03202
“Overreliance is a well-documented risk in human-AI collaboration (Buc¸inca et al., 2021; Chiang & Yin, 2021).”
- Authors
- Chiang, C.-W., & Yin, M.
- Affiliations
- Purdue University West Lafayette (US)
- ORCID
- Yin, Ming
- Venue
- 13th ACM Web Science Conference 2021
- Publisher
- ACM
- Published
- 2021-06-21
- Pages
- 120-129
- OpenAlex
- W3174333417
- Cited by
- 53
- References
- 33
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 9 · 2023: 12 · 2022: 10 · 2021: 1
- FWCI
- 3.70
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Web Science Conference (WebSci) (publishedVersion)
- Funders
- National Science Foundation (IIS-1850335)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- covariate shift, out-of-distribution data, decision-making, data distribution visualization, performance gap, human-AI interaction, recommender systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
