The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
The effects of mixing machine learning and human judgment
Abstract
Collaboration between humans and machines does not necessarily lead to better outcomes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Likewise, these results mirror other recent research revealing human–AI collaborative decision-making does not necessarily lead to more accurate results than either humans or AI alone (72–76).”
From Deepfake detection by human crowds, machines, and machine-informed crowds (2022) · cites this paper as [72]
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
A literature review of the anchoring effect
Furnham, A., & Boo, H. C. (2011) · The Journal of Socio-Economics · doi:10.1016/j.socec.2010.10.008 · Cited by 1,179
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Northcraft, G. B., & Neale, M. A. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90046-x · Cited by 1,073
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Englich, B., Mussweiler, T., & Strack, F. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167205282152 · Cited by 554
An Anchoring and Adjustment Model of Purchase Quantity Decisions
Wansink, B., Kent, R. J., & Hoch, S. J. (1998) · Journal of Marketing Research · doi:10.1177/002224379803500108 · Cited by 439 · cited 1×
“Scholars have observed similar values of the anchoring index in contexts such as probability estimates, 19 purchasing decisions, 20 and sales forecasting.”
BackgroundDesigning incentives for inexpert human raters
Shaw, A. D., Horton, J. J., & Chen, D. L. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958865 · Cited by 236
Research works printed but not linked: 2
- Angwin J., 2016, ProPublica (May 23
- Johnson R.C., 2018, Overcoming AI bias with AI fairness. Commun. ACM (Dec. 6
Cited by
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“Recent studies, however, show that although human+AI teams typically outperform people working alone, their performance is usually inferior to the AI’s [2, 7, 9, 26, 30, 41, 59].”
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“These papers reported the results of 106 unique experiments, and many of the experiments had multiple conditions, so we collected a total of 370 unique effect sizes measuring the impact of human–AI collaboration coordination between humans and AI systems can hinder the collaborative process 4–9 .”
Self-citationTrunk, A., Birkel, H., & Hartmann, E. (2020) · Business Research · doi:10.1007/s40685-020-00133-x · Cited by 252
“Opportunities and hazards, however, are neither agreed nor analyzed in more detail, making research necessary (Lawrence 1991; Silva and Kenney 2018; Vaccaro and Waldo 2019).”
BackgroundDeepfake detection by human crowds, machines, and machine-informed crowds
Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2110013119 · Cited by 248
Keding, C., & Meissner, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120970 · Cited by 168
Metadata
- Authors
- Vaccaro, M., & Waldo, J.
- Affiliations
- Harvard University (US)
- Venue
- Communications of the ACM
- Published
- 2019-10-24
- Volume
- 62
- Issue
- 11
- Pages
- 104-110
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2980535893
- Cited by
- 49
- References
- 17
- Citations by year
- 2026: 11 · 2025: 6 · 2024: 9 · 2023: 4 · 2022: 6 · 2021: 9 · 2020: 4
- FWCI
- 5.97
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- human-machine collaboration, machine learning, human judgment, human-AI interaction, decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The effects of mixing machine learning and human judgment. https://app.cochat.ai/papers/10.1145/3359338
- The paper
- Vaccaro, M., & Waldo, J. (2019). The effects of mixing machine learning and human judgment. Communications of the ACM. https://doi.org/10.1145/3359338
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.