Sample Efficient Learning of Predictors that Complement Humans
Open accessAbstract
One of the goals of learning algorithms is to complement and reduce the burden on human decision makers. The expert deferral setting wherein an algorithm can either predict on its own or defer the decision to a downstream expert helps accomplish this goal. A fundamental aspect of this setting is the need to learn complementary predictors that improve on the human's weaknesses rather than learning predictors optimized for average error. In this work, we provide the first theoretical analysis of the benefit of learning complementary predictors in expert deferral. To enable efficiently learning such predictors, we consider a family of consistent surrogate loss functions for expert deferral and analyze their theoretical properties. Finally, we design active learning schemes that require minimal amount of data of human expert predictions in order to learn accurate deferral systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Learning to Defer algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.
“Learning to Defer (L2D) algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [4]
The paper chose to create a total of 50 experts because this far surpasses the numbers previously considered in learning-to-defer research.
“We chose to create a total of 50 experts as this far surpasses the numbers previously considered in L2D research4–13.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [4]
Various authors use CIFAR-10, simulating experts with high accuracy on five classes but random accuracy on the remaining five.
“Various authors4,8,11 use CIFAR-1016, simulating experts with high accuracy on five classes but random accuracy on the remaining five.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [4]
The state-of-the-art approach to managing assignments in human-AI collaboration is learning to defer.
“The state-of-the-art approach to managing assignments in HAIC is learning to defer (L2D)4–13, in which algorithms choose whether to assign a task to a human expert or an ML model, leveraging their complementary strengths.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [4]
For α=0, the probability of error is the same for all instances within a class, which is equivalent to class-dependent label noise.
“For α=0, the probability of error is the same for all instances within a class, which is equivalent to class-dependent label noise, as used in previous L2D testing4,8,11.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [4]
The Problem Setting (p. 3) allows expert predictions to depend on both the instance and its label, but does not define an α parameter or class-dependent label noise. The Experimental Illustration (p. 13) describes a different synthetic error pattern. Searches for α=0 and class-dependent label noise found no passage establishing the claimed equivalence. (LLM, conf=0.76)
Experimental — accuracy not yet measured
A common approach in learning-to-defer research involves using label noise to produce arbitrarily accurate predictions.
“A common approach involves using label noise to produce arbitrarily accurate predictions.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [4]
The Experimental Illustration (p. 13) uses random labels in one region of a synthetic dataset, but does not describe using label noise to produce arbitrarily accurate predictions. The Discussion (p. 14) instead identifies surrogate losses and active learning as approaches to learning-to-defer. Searches for label noise and arbitrarily accurate predictions found no passage making the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 33
- [GEY17] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems , pages 4878-4887, 2017.
- [Gil20] Tarleton Gillespie. Content moderation, ai, and the question of scale. Big Data & Society , 7(2):2053951720943234, 2020.
- [GKS21] Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via onesided prediction. In International Conference on Artificial Intelligence and Statistics , pages 2179-2187. PMLR, 2021.
- [GSTDA + 21] Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga, Ligong Han, Min Kyung Lee, and Matthew Lease. Human-ai collaboration with bandit feedback. arXiv preprint arXiv:2105.10614 , 2021. · arXiv
- [Han14] Steve Hanneke. Theory of active learning. Foundations and Trends in Machine Learning , 7(2-3), 2014.
- [JPDV21] Shalmali Joshi, Sonali Parbhoo, and Finale Doshi-Velez. Pre-emptive learningto-defer for sequential medical decision-making under uncertainty. arXiv preprint arXiv:2109.06312 , 2021. · arXiv
- [KAH + 17] Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 1915-1924. PMLR, 2017.
- [KHH12] Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS , volume 12, pages 467-474, 2012.
- [KLK21] Vijay Keswani, Matthew Lease, and Krishnaram Kenthapadi. Towards unbiased and accurate deferral to multiple experts. arXiv preprint arXiv:2102.13004 , 2021. · arXiv
- [KLL + 18] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics , 133(1):237-293, 2018.
- [KSS21] Gavin Kerrigan, Padhraic Smyth, and Mark Steyvers. Combining human predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems , 34, 2021.
- [LGB21] Jessie Liu, Blanca Gallego, and Sebastiano Barbieri. Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. arXiv preprint arXiv:2108.07392 , 2021. · arXiv
- [LH17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. · arXiv
- [LLT20] Vivian Lai, Han Liu, and Chenhao Tan. " why is' chicago'deceptive?" towards building model-driven tutorials for humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages 1-13, 2020.
- [LT91] Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.
- [MPZ18] David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems , pages 6150-6160, 2018.
- [MRT18] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.
- [MS20] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087. PMLR, 2020.
- [MSS22] Hussein Mozannar, Arvind Satyanarayan, and David Sontag. Teaching humans when to defer to a classifier via exemplars. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI) , 2022.
- [NVBR19] Alex Nowak-Vila, Francis Bach, and Alessandro Rudi. A general theory for structured prediction with smooth convex surrogates. arXiv preprint arXiv:1902.01958 , 2019. · arXiv
- [ODGR21] Nastaran Okati, Abir De, and Manuel Gomez-Rodriguez. Differentiable learning under triage. arXiv preprint arXiv:2103.08902 , 2021. · arXiv
- [PZP + 21] Melanie F Pradier, Javier Zazo, Sonali Parbhoo, Roy H Perlis, Maurizio Zazzi, and Finale Doshi-Velez. Preferential mixture-of-experts: Interpretable models that rely on human expertise as much as possible. arXiv preprint arXiv:2101.05360 , 2021. · arXiv
- [RBC + 19] Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 , 2019. · arXiv
- [RSG16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.
- [RY21] Naveen Raman and Michael Yee. Improving learning-to-defer algorithms through fine-tuning. arXiv preprint arXiv:2112.10768 , 2021. · arXiv
- [SM20] Kulin Shah and Naresh Manwani. Online active learning of reject option classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5652-5659, 2020.
- [SSMGR21] Eleni Straitouri, Adish Singla, Vahid Balazadeh Meresht, and Manuel GomezRodriguez. Reinforcement learning under algorithmic triage. arXiv preprint arXiv:2109.11328 , 2021. · arXiv
- [TAIK18] Sarah Tan, Julius Adebayo, Kori Inkpen, and Ece Kamar. Investigating human+ machine complementarity for recidivism predictions. arXiv preprint arXiv:1808.09123 , 2018. · arXiv
- [WHK20] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. arXiv preprint arXiv:2005.00582 , 2020. · arXiv
- [ZARS21] Jason Zhao, Monica Agrawal, Pedram Razavi, and David Sontag. Directing human attention in event localization for clinical timeline creation. In Machine Learning for Healthcare Conference , pages 80-102. PMLR, 2021.
- [ZC15] Chicheng Zhang and Kamalika Chaudhuri. Active learning from weak and strong labelers. In Advances in Neural Information Processing Systems , pages 703-711, 2015.
- [Zha04] Tong Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004.
- [ZSZR18] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N Rafferty. An overview of machine teaching. arXiv preprint arXiv:1801.05927 , 2018. · arXiv
Cited by
Sorted by citations
A benchmarking framework and dataset for learning to defer in human-AI decision-making
Alves, J. V., Leitão, D., Jesus, S., Sampaio, M. O. P., Liébana, J., Saleiro, P., Figueiredo, M. A. T., & Bizarro, P. (2025) · Scientific Data · doi:10.1038/s41597-025-04664-y · Cited by 10 · see citing works
- Authors
- Charusaie, M.-A., Mozannar, H., Sontag, D., & Samadi, S.
- Affiliations
- Max Planck Institute for Intelligent Systems (DE)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2207.09584
- OpenAlex
- W4320466172
- Cited by
- 6
- References
- 0
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 1 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
- International Max Planck Research School for Advanced Methods in Process and Systems Engineering
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- surrogate loss functions, active learning (machine learning), sample efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
