Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness
Open accessAbstract
Much of machine learning research focuses on predictive accuracy: given a task, create a machine learning model (or algorithm) that maximizes accuracy. In many settings, however, the final prediction or decision of a system is under the control of a human, who uses an algorithm’s output along with their own personal expertise in order to produce a combined prediction. One ultimate goal of such collaborative systems is complementarity: that is, to produce lower loss (equivalently, greater payoff or utility) than either the human or algorithm alone. However, experimental results have shown that even in carefully-designed systems, complementary performance can be elusive. Our work provides three key contributions. First, we provide a theoretical framework for modeling simple human-algorithm systems and demonstrate that multiple prior analyses can be expressed within it. Next, we use this model to prove conditions where complementarity is impossible, and give constructive examples of where complementarity is achievable. Finally, we discuss the implications of our findings, especially with respect to the fairness of a classifier. In sum, these results deepen our understanding of key factors influencing the combined performance of human-algorithm systems, giving insight into how algorithmic tools can best be designed for collaborative environments.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human-AI synergy requires that humans be better at some parts of a task, AI be better at other parts, and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask.
“Additionally, as discussed in ref. 33, human–AI synergy requires that humans be better at some parts of a task, AI be better at other parts of the task and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask.”
From When combinations of humans and AI are useful: A systematic review and meta-analysis (2024) · cites this paper as [33]
Learning to defer involves understanding when humans can effectively complement AI models.
“It also involves understanding the conditions under which humans can effectively complement AI models [24].”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [24]
Cites
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,224 · see citing works
Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,450 · see citing works
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,187 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“For example, some prior work has demonstrated that, for many algorithms, reducing loss becomes harder as the level of loss decreases, which could make it more difficult to achieve highly variable loss rates [16, 17].”
BackgroundDietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 1× · see citing works
“[10] suggests that allowing algorithmic predictions to be modified may make humans more likely to use them.”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 3× · see citing works
“[2]) is achieved whenever the combined humanalgorithm system has strictly lower expected loss than either the human or the algorithm alone (Figure 1c).”
BackgroundInfluentialManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1× · see citing works
“[38] analyze how explainability and accuracy, respectively, influence how humans use algorithmic predictions.”
BackgroundSayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2018.11.016 · Cited by 418 · see citing works
Investigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. M. (2018) · Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 230 · see citing works
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · see citing works
Self-citationIs the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
Bansal, G., Nushi, B., Kamar, E. S., Horvitz, E., & Weld, D. S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17359 · Cited by 129 · see citing works
Okolo, C. T., Kamath, S., Dell, N. L., & Vashistha, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445420 · Cited by 96 · see citing works
Geirhos, R., Meding, K., & Wichmann, F. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16736 · Cited by 63 · see citing works
Do Humans Trust Advice More if it Comes from AI?: An Analysis of Human-AI Interactions
Vodrahalli, K., Daneshjou, R., Gerstenberg, T., & Zou, J. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534150 · Cited by 63 · see citing works
Towards Unbiased and Accurate Deferral to Multiple Experts
Keswani, V., Lease, M., & Kenthapadi, K. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462516 · Cited by 45 · see citing works
Self-citationAn Empirical Analysis of Backward Compatibility in Machine Learning Systems
Srivastava, M., Nushi, B., Kamar, E. S., Shah, S., & Horvitz, E. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403379 · Cited by 35 · see citing works
Algorithmic Social Engineering
Cowgill, B., & Stevenson, M. T. (2020) · AEA Papers and Proceedings · doi:10.1257/pandp.20201037 · Cited by 30 · see citing works
Diversity in Classifier Ensembles: Fertile Concept or Dead End?
Didaci, L., Fumera, G., & Roli, F. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38067-9_4 · Cited by 29 · see citing works
Practical Compositional Fairness: Understanding Fairness in Multi-Component Recommender Systems
Wang, X., Thain, N., Sinha, A., Prost, F., Ed, H., Chen, J., & Beutel, A. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441732 · Cited by 18 · see citing works
Classification Under Human Assistance
De, A., Okati, N., Zarezade, A., & Gomez-Rodriguez, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16738 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570 · see citing works
“33, human–AI synergy requires that humans be better at some parts of a task, AI be better at other parts of the task and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask.”
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36 · see citing works
Self-citationToward a science of human–AI teaming for decision making: a complementarity framework
Gonzalez, C., Donahue, K., Goldstein, D. G., Heidari, H., Jalali, M. S., Schelble, B., Singh, A., & Woolley, A. W. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag030 · Cited by 33 · see citing works
- Authors
- Donahue, K., Chouldechova, A., & Kenthapadi, K.
- Affiliations
- Cornell University (US)
- Amazon (United States) (US)
- Fiddler AI, USA
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 1639-1656
- OpenAlex
- W4283167699
- Cited by
- 33
- References
- 23
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 6 · 2023: 7 · 2020: 1
- FWCI
- 6.46
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (National Science Foundation) (DGE-1650441)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- human-algorithm collaboration, human-in-the-loop systems, complementarity, unfairness, human expertise, theoretical framework, impossibility results, algorithm design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
