The effects of mixing machine learning and human judgment
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Collaboration between humans and machines does not necessarily lead to better outcomes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Likewise, these results mirror other recent research revealing human–AI collaborative decision-making does not necessarily lead to more accurate results than either humans or AI alone (72–76).”
From Deepfake detection by human crowds, machines, and machine-informed crowds (2022) · cites this paper as [72]
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
A literature review of the anchoring effect
Furnham, A., & Boo, H. C. (2011) · The Journal of Socio-Economics · doi:10.1016/j.socec.2010.10.008 · Cited by 1,179
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Northcraft, G. B., & Neale, M. A. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90046-x · Cited by 1,073
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Englich, B., Mussweiler, T., & Strack, F. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167205282152 · Cited by 554
An Anchoring and Adjustment Model of Purchase Quantity Decisions
Wansink, B., Kent, R. J., & Hoch, S. J. (1998) · Journal of Marketing Research · doi:10.1177/002224379803500108 · Cited by 439 · cited 1×
“Scholars have observed similar values of the anchoring index in contexts such as probability estimates, 19 purchasing decisions, 20 and sales forecasting.”
BackgroundDesigning incentives for inexpert human raters
Shaw, A. D., Horton, J. J., & Chen, D. L. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958865 · Cited by 236
Research works printed but not linked: 2
- Angwin J., 2016, ProPublica (May 23
- Johnson R.C., 2018, Overcoming AI bias with AI fairness. Commun. ACM (Dec. 6
Cited by
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“Recent studies, however, show that although human+AI teams typically outperform people working alone, their performance is usually inferior to the AI’s [2, 7, 9, 26, 30, 41, 59].”
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“These papers reported the results of 106 unique experiments, and many of the experiments had multiple conditions, so we collected a total of 370 unique effect sizes measuring the impact of human–AI collaboration coordination between humans and AI systems can hinder the collaborative process 4–9 .”
Self-citationTrunk, A., Birkel, H., & Hartmann, E. (2020) · Business Research · doi:10.1007/s40685-020-00133-x · Cited by 252
“Opportunities and hazards, however, are neither agreed nor analyzed in more detail, making research necessary (Lawrence 1991; Silva and Kenney 2018; Vaccaro and Waldo 2019).”
BackgroundDeepfake detection by human crowds, machines, and machine-informed crowds
Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2110013119 · Cited by 248
Keding, C., & Meissner, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120970 · Cited by 168
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144
“Even though the IOED itself received little attention in the context of intelligent systems and XAI, there is prior research on cognitive biases of explanations from intelligent systems investigating automation [9, 31, 40, 49, 52], anchoring [19, 27, 57], framing [26], and confirmation biases [23, 55].”
BackgroundDo People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
“Contrary to the initial expectations [35, 36], people supported by AI-powered decision support systems often make less accurate decisions on average than AI-powered systems on their own [4, 8, 9, 11, 29, 34, 40, 62].”
ResultWhither AutoML? Understanding the Role of Automation in Machine Learning Workflows
Xin, D., Wu, E. Y., Lee, D. J.-L., Salehi, N., & Parameswaran, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445306 · Cited by 110
Hornung, O., & Smolnik, S. (2021) · Electronic Markets · doi:10.1007/s12525-021-00493-0 · Cited by 113
“…does not guarantee better outcomes, and when a PVA errs or shows untoward bias, people often insufficiently intervene to address the problem (Vaccaro & Waldo, 2019) due to the employees’ reduced ability to critically reflect on their work once the PVA has taken over the entire…”
BackgroundFairness Perceptions of Artificial Intelligence: A Review and Path Forward
Narayanan, D., Nagpal, M., McGuire, J., Schweitzer, S., & Cremer, D. D. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2210890 · Cited by 104
Effects of personality traits on user trust in human–machine collaborations
Zhou, J. J., Luo, S., & Chen, F. (2020) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-020-00329-9 · Cited by 58
Studying human-to-computer bias transference
Johansen, J., Pedersen, T., & Johansen, C. (2021) · AI & Society · doi:10.1007/s00146-021-01328-4 · Cited by 54
“2019; Vaccaro and Waldo 2019).”
Safaei, M., & Longo, J. (2023) · Digital Government Research and Practice · doi:10.1145/3604570 · Cited by 62
“Note, however, that Vaccaro and Waldo [148] have shown that a blend of human and AI judgment does not necessarily generate better results.”
BackgroundHuman, Do You Think This Painting is the Work of a Real Artist?
Park, J., Kang, H., & Kim, H. Y. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2232978 · Cited by 39
“Vaccaro and Waldo (2019) investigated how the risk prediction information of the COMPAS algorithm affects human evaluation, finding that the evaluation information of the algorithm did not significantly affect the human evaluation of recidivism but served as an anchor to induce cognitive bias.”
A Decision Theoretic Framework for Measuring AI Reliance
Guo, Z., Wu, Y., Hartline, J. D., & Hullman, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658901 · Cited by 27
“However, many studies have shown that human-AI teams under-perform the AI alone in tasks where the AI’s accuracy is higher than humans [3, 4, 6, 14, 18, 20, 21, 28].”
Fogliato, R., Chouldechova, A., & Lipton, Z. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479572 · Cited by 23
Do I Look Like a Criminal? Examining how Race Presentation Impacts Human Judgement of Recidivism
Mallari, K., Inkpen, K., Johns, P., Tan, S., Ramesh, D., & Kamar, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376257 · Cited by 24
“Algorithmic aids have also been shown to bias workers’ predictions of recidivism [19, 45].”
BackgroundLiang, Z., Yang, K., Sha, L., Gašević, D., Yan, L., & Chen, G. (2026) · British Journal of Educational Technology · doi:10.1111/bjet.70055 · Cited by 26
Framework for human–XAI symbiosis: extended self from the dual-process theory perspective
Litvinova, Y., Mikalef, P., & Luo, X. R. (2024) · Journal of Business Analytics · doi:10.1080/2573234x.2024.2396366 · Cited by 20
“…for these outcomes, researchers suggest that humans are trapped by anchoring bias or effort heuristic (Arnott & Gao, 2022: Table 3, p. 84) associated with such biases as confirmation trap or ease of recall and illusory correlation (Arnott & Gao, 2022; Rosenbacke, 2024; Vaccaro & Waldo, 2019).”
On prediction-modelers and decision-makers: why fairness requires more than a fair prediction model
Scantamburlo, T., Baumann, J., & Heitz, C. (2024) · AI & Society · doi:10.1007/s00146-024-01886-3 · Cited by 28
“g by comparing human estimates to algorithmic outcomes [31, 40, 58, 93].”
Advancing AI negotiations: A large-scale autonomous negotiation competition
Vaccaro, M. A., Caosun, M., Ju, H., Aral, S., & Curhan, J. R. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2521774123 · Cited by 16
Self-citationBrodeur, A., Valenta, D., Marcoci, A., Aparicio, J. P., Mikola, D., Barbarioli, B., Alexander, R., Deer, L., Stafford, T., Vilhuber, L., et al. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2524747123 · Cited by 16
Just Resource Allocation? How Algorithmic Predictions and Human Notions of Justice Interact
Kube, A., Das, S., Fowler, P. J., & Vorobeychik, Y. (2022) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3490486.3538305 · Cited by 17
“In contrast, the presence of predictions by themselves has been known to trigger a cognitive bias (anchoring) that worsens accuracy [39, 41].”
BackgroundZhu, Y., Hua, G., Liu, X., Wang, C., & Tang, M. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1539054 · Cited by 14
“This partnership moves beyond mere mechanical execution, combining subjective human insight with efficient machine computation (Vaccaro and Waldo, 2019).”
Ku, C.-Y. (2020) · Információs Társadalom · doi:10.22503/inftars.xix.2019.4.5 · Cited by 7
Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 15
Passing the Buck to AI: How Individuals’ Decision-Making Patterns Affect Reliance on AI
Mei, K. X., Pang, R. Y., Lyford, A., Wang, L. L., & Reinecke, K. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3786326 · Cited by 8
“As evidenced by prior work, making decisions with the help of AI could lead to more accurate decisions when the AI has high advice accuracy and quality [66, 70] (though there are some exceptions where this was found not to be the case [35, 67]).”
One Decade of Algorithmic Management Research
Lippert, I., Alizadeh, A., Tarafdar, M., Möhlmann, M., Benlian, A., Parent‐Rocheleau, X., & Stein, M. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-026-00995-1 · Cited by 8
“Such hybrid pipelines can increase efficiency but also risk anchoring human judgment to algorithmic rankings (Vaccaro and Waldo 2019).”
Comparing Human and Machine Deepfake Detection with Affective and Holistic Processing.Preprint
Groh, M., Epstein, Z., Firestone, C., & Picard, R. W. (2021) · arXiv (Cornell University) · Cited by 5
Human-AI Teaming: Following the IMOI Framework
Kleanthous, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60611-3_27 · Cited by 6
ANDREESCU, F. (2022) · Informatica Economica · doi:10.24818/issn14531305/26.1.2022.05 · Cited by 3
“The study [23] analyzes human position games and artificial intelligence algorithms in decision making.”
Rozony, F. Z., & Vudugula, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5255980 · Cited by 3
Beyond black boxes and biases: advancing artificial intelligence in sentencing
Alimardani, A., & Istiqomah, M. (2025) · Current Issues in Criminal Justice · doi:10.1080/10345329.2025.2527994 · Cited by 5
“This phenomenon, known as the anchoring effect, may cause judges to place undue emphasis on the AI’s output and potentially disregard other relevant facts or evidence during their evaluation (Donohue, 2019; Vaccaro & Waldo, 2019; also see Mussweiler et al., 2004).”
Chen, Z. Y. (2026) · Management Decision · doi:10.1108/md-07-2025-1890 · Cited by 3
Design Principles Supporting Data-driven Decisions PlatformsPreprint
Elragal, A. A., & Elgendy, N. (2024) · Research Square · doi:10.21203/rs.3.rs-4240652/v1 · Cited by 3
ReviseMate: Exploring Contextual Support for Digesting STEM Paper Reviews
Xu, Y., Zhang, S., Fan, Y., Shi, S., Peng, Z., & Li, Q. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757502 · Cited by 1
“Although AI-assisted systems are believed to enhance human analysis efficiency and quality significantly, some studies indicate that mixed-initiative human-AI workflows can often be inferior to standalone AI [6, 27, 45, 84].”
INCREASING TIME REGISTRATION IN AN AGILE ENVIRONMENT THROUGH NUDGING IN HUMAN-COMPUTER INTERACTION
Schrijen, M., & van der Pas, M. (2024) · IADIS International Conference Information Systems · doi:10.33965/is2024_202401l014 · Cited by 1
On Prediction-Modelers and Decision-Makers: Why Fairness Requires More Than a Fair Prediction ModelPreprint
Scantamburlo, T., Baumann, J., & Heitz, C. (2023) · Zürcher Hochschule für Angewandte Wissenschaften digital collection (Zurich University of Applied Sciences) · doi:10.48550/arxiv.2310.05598 · Cited by 2
“g by comparing human estimates to algorithmic outcomes [31, 40, 58, 93].”
Guerreiro, S. L. P. D. (2022) · Operations Research Forum · doi:10.1007/s43069-022-00142-y
“However, the question of to what extent could a machine be able to support cooperative decisions in an operational business process environment with incomplete information is still an open research issue [10, 11].”
BackgroundInnovation in Tourism Systems, Intelligent Gamification, and User Interaction
(2022) · Advances in educational technologies and instructional design book series · doi:10.4018/978-1-7998-8721-8.ch005
A Dynamic Generator for Machine Learning Training for Traffic Management Systems
ANDREESCU, F. (2022) · Informatica Economica · doi:10.24818/issn14531305/26.4.2022.05
SUSTAINABILITY TRACING FOR COMPANIES: GOVERNANCE, REGULATION, ECONOMICAL AND ETHICAL CONSIDERATIONS
Pisoni, G., Molnár, B., & Moloney, S. K. A. M. (2024) · IADIS International Conference Information Systems · doi:10.33965/is2024_202401c002
Error in the Loop: How Human Mistakes Can Improve Algorithmic Learning
Copus, R. W., Spackman, C., & Laqueur, H. S. (2025) · Journal of law & empirical analysis. · doi:10.1177/2755323x251357643 · Cited by 2
“As algorithmic risk assessments proliferate, so does demand to identify the proper role of human judgment in these systems (Dawes et al., 1989; Vaccare & Waldo, 2019; De-Arteaga et al., 2020).”
Ethical considerations for big data analytics
Balfagih, Z., & Elkafrawy, P. (2026) · Elsevier eBooks · doi:10.1016/b978-0-44-326735-2.00020-1 · Cited by 1
Gelashvili-Luik, T., Vihma, P., Pappel, I., & Ferreira, F. A. F. (2026) · Journal of Modelling in Management · doi:10.1108/jm2-09-2025-0483 · Cited by 1
Generative artificial intelligence in policy analysis: a study of UK think tanks
Pautz, H., Zwet, A. V. D., Munoz-Ramirez, P., & Leatham, S. (2026) · AI & Society · doi:10.1007/s00146-026-03084-9
“However, Vaccaro and Waldo (2019) have shown that a blend of human and AI judgement does not necessarily generate better results than separately made judgements.”
The invisible layer: ethics, accountability, and algorithmic workforce governance
Castillo, L. D. (2026) · AI and Ethics · doi:10.1007/s43681-026-01130-2 · Cited by 3
“The AI ethics literature has generated frameworks including the OECD AI Principles [ 27 ], the IEEE Ethically Aligned Design standards [28], and numerous academic proposals [29, 30], consistently emphasizing transparency, accountability, and human oversight.”
Tuo, B., Wang, B. T., Miller, T., & Demartini, G. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3848504
“A recent systematic review and meta-analysis by Vaccaro et al. (2024) [57] shows that human–AI teams typically perform at an intermediate level between their individual members, rarely surpassing the stronger contributor across a wide range of tasks and domains [58].”
Metadata
- Authors
- Vaccaro, M., & Waldo, J.
- Affiliations
- Harvard University (US)
- Venue
- Communications of the ACM
- Published
- 2019-10-24
- Volume
- 62
- Issue
- 11
- Pages
- 104-110
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2980535893
- Cited by
- 49
- References
- 17
- Citations by year
- 2026: 11 · 2025: 6 · 2024: 9 · 2023: 4 · 2022: 6 · 2021: 9 · 2020: 4
- FWCI
- 5.97
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- human-machine collaboration, machine learning, human judgment, human-AI interaction, decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.