CoChat
Sign Up
Cited by 10Machine Learning and AlgorithmsMachine Learning and Data ClassificationExplainable Artificial Intelligence (XAI)

Learning to Defer with Limited Expert Predictions

Open access
Hemmer, Patrick, Thede, Lukas, Vössing, Michael, Jakubik, Johannes, Kühl, Niklas·2023·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v37i5.25742
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent research suggests that combining AI models with a human expert can exceed the performance of either alone. The combination of their capabilities is often realized by learning to defer algorithms that enable the AI to learn to decide whether to make a prediction for a particular instance or defer it to the human expert. However, to accurately learn which instances should be deferred to the human expert, a large number of expert predictions that accurately reflect the expert's capabilities are required—in addition to the ground truth labels needed to train the AI. This requirement shared by many learning to defer algorithms hinders their adoption in scenarios where the responsible expert regularly changes or where acquiring a sufficient number of expert predictions is costly. In this paper, we propose a three-step approach to reduce the number of expert predictions required to train learning to defer algorithms. It encompasses (1) the training of an embedding model with ground truth labels to generate feature representations that serve as a basis for (2) the training of an expertise predictor model to approximate the expert's capabilities. (3) The expertise predictor generates artificial expert predictions for instances not yet labeled by the expert, which are required by the learning to defer algorithms. We evaluate our approach on two public datasets. One with "synthetically" generated human experts and another from the medical domain containing real-world radiologists' predictions. Our experiments show that the approach allows the training of various learning to defer algorithms with a minimal number of human expert predictions. Furthermore, we demonstrate that even a small number of expert predictions per class is sufficient for these algorithms to exceed the performance the AI and the human expert can achieve individually.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v37i5.25742

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The only public dataset containing human predictions used in multi-expert learning-to-defer is the NIH Clinical Center X-ray dataset, aimed at detecting airspace opacity.

    “To our knowledge, the only public dataset containing human predictions used in multi-expert L2D is the NIH Clinical Center X-ray dataset5,13,14, aimed at detecting airspace opacity; each instance features the predictions of 3 radiologists, sampled from a team of 22.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [13]

    The NIH Chest X-rays section describes a dataset with individual predictions from three radiologists per image and says the authors used it to study airspace opacity. It does not say this is the only public dataset used in multi-expert learning-to-defer, or that the three radiologists were sampled from a team of 22; the authors selected one radiologist for their main experiments. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The NIH Clinical Center X-ray dataset features predictions of 3 radiologists per instance, sampled from a team of 22.

    “To our knowledge, the only public dataset containing human predictions used in multi-expert L2D is the NIH Clinical Center X-ray dataset5,13,14, aimed at detecting airspace opacity; each instance features the predictions of 3 radiologists, sampled from a team of 22.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [13]

    The NIH Chest X-rays section says that each image has predictions from three individual radiologists and that the authors focus their experiments on airspace opacity. It does not say those radiologists were sampled from a team of 22 (searched for “22” and the team description), so that part is unsupported by our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The assumption that ground-truth labels exist is common to all cited works and is considered necessary for an L2D algorithm to estimate the human's probability of correctness for a given instance.

    “the assumption that ground-truth labels exist is common to all cited works, and is considered necessary for an L2D algorithm to be able to learn to estimate the human's probability of correctness for a given instance.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [13]

    The Introduction says learning-to-defer algorithms require ground-truth labels and expert predictions. The Problem Formulation and Training of Expertise Predictor Model sections show that this paper uses ground truth to determine whether an expert's prediction is correct. The paper does not establish that the assumption is common to all works cited by the citing document, or that ground truth is necessary for every L2D algorithm to estimate instance-level human correctness. Page numbers are unavailable in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The state-of-the-art approach to managing assignments in human-AI collaboration is learning to defer.

    “The state-of-the-art approach to managing assignments in HAIC is learning to defer (L2D)4–13, in which algorithms choose whether to assign a task to a human expert or an ML model, leveraging their complementary strengths.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [13]

  • Learning to Defer algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.

    “Learning to Defer (L2D) algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Branson, S.; Van Horn, G.; and Perona, P. 2017. Lean crowd- sourcing: combining humans and machines in an online sys- tem. In Conference on Computer Vision and Pattern Recog- nition, 7474-7483.
  2. Buolamwini, J.; and Gebru, T. 2018. Gender shades: in- tersectional accuracy disparities in commercial gender clas- sification. In Conference on Fairness, Accountability and Transparency, 77-91.
  3. Charusaie, M.-A.; Mozannar, H.; Sontag, D.; and Samadi, S. 2022. Sample efficient learning of predictors that com- plement humans. In International Conference on Machine Learning, 2972-3005.
  4. Cremer, C. Z. 2021. Deep limitations? Examining expert disagreement over deep learning. Progress in Artificial In- telligence, 10(4): 449-464.
  5. De, A.; Koley, P.; Ganguly, N.; and Gomez-Rodriguez, M. 2020. Regression under human assistance. In AAAI Confer- ence on Artificial Intelligence, 2611-2620.
  6. De, A.; Okati, N.; Zarezade, A.; and Gomez-Rodriguez, M. 2021. Classification under human assistance. In AAAI Con- ference on Artificial Intelligence, 5905-5913.
  7. Farrell, C.-J. 2021. Identifying mislabelled samples: ma- chine learning models exceed human performance. Annals of Clinical Biochemistry, 58(6): 650-652.
  8. Gao, R.; Saar-Tsechansky, M.; De-Arteaga, M.; Han, L.; Lee, M. K.; and Lease, M. 2021. Human-AI collaboration with bandit feedback. In International Joint Conference on Artificial Intelligence, 1722-1728.
  9. Guan, M.; Gulshan, V.; Dai, A.; and Hinton, G. 2018. Who said what: modeling individual labelers improves classifica- tion. In AAAI Conference on Artificial Intelligence, 3109- 3118.
  10. Hemmer, P.; Schemmer, M.; Kühl, N.; Vössing, M.; and Satzger, G. 2022b. On the effect of information asymme- try in human-AI teams. In CHI Conference on Human Fac- tors in Computing Systems, Workshop on Human-Centered Explainable AI.
  11. Irvin, J.; Rajpurkar, P.; Ko, M.; Yu, Y.; Ciurea-Ilcus, S.; Chute, C.; Marklund, H.; Haghgoo, B.; Ball, R.; Shpan- skaya, K.; et al. 2019. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI Conference on Artificial Intelligence, 590-597.
  12. Jones, E.; Sagawa, S.; Koh, P. W.; Kumar, A.; and Liang, P. 2021. Selective classification can magnify disparities across groups. In International Conference on Learning Represen- tations.
  13. Koch, G.; Zemel, R.; Salakhutdinov, R.; et al. 2015. Siamese neural networks for one-shot image recognition. In Inter- national Conference on Machine Learning, Deep Learning Workshop.
  14. Lampert, T. A.; Stumpf, A.; and Ganc ¸arski, P. 2016. An empirical study into annotator agreement, ground truth es- timation, and algorithm evaluation. IEEE Transactions on Image Processing, 25(6): 2557-2572.
  15. Leitão, D.; Saleiro, P.; Figueiredo, M. A.; and Bizarro, P. 2022. Human-AI collaboration in decision-making: beyond learning to defer. In International Conference on Machine Learning, Workshop on Human-Machine Collaboration and Teaming.
  16. Li, J.; Xiong, C.; and Hoi, S. C. 2021. CoMatch: semi- supervised learning with contrastive graph regularization. In International Conference on Computer Vision, 9475-9484.
  17. Liao, Y.-H.; Kar, A.; and Fidler, S. 2021. Towards good practices for efficiently annotating large-scale image clas- sification datasets. In Conference on Computer Vision and Pattern Recognition, 4350-4359.
  18. Madras, D.; Pitassi, T.; and Zemel, R. 2018. Predict respon- sibly: improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems.
  19. Majkowska, A.; Mittal, S.; Steiner, D. F.; Reicher, J. J.; McKinney, S. M.; Duggan, G. E.; Eswaran, K.; Cameron Chen, P.-H.; Liu, Y.; Kalidindi, S. R.; et al. 2020. Chest radiograph interpretation with deep learning models: assessment with radiologist-adjudicated reference standards and population-adjusted evaluation. Radiology, 294(2): 421-431.
  20. Okati, N.; De, A.; and Rodriguez, M. 2021. Differentiable learning under triage. Advances in Neural Information Pro- cessing Systems, 9140-9151.
  21. Raghu, M.; Blumer, K.; Corrado, G.; Kleinberg, J.; Ober- meyer, Z.; and Mullainathan, S. 2019. The algorithmic automation problem: prediction, triage, and human effort. arXiv preprint arXiv:1903.12220. · arXiv
  22. Rajpurkar, P.; Irvin, J.; Zhu, K.; Yang, B.; Mehta, H.; Duan, T.; Ding, D.; Bagul, A.; Langlotz, C.; Shpanskaya, K.; et al. 2017. Chexnet: radiologist-level pneumonia detec- tion on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225. · arXiv
  23. Raman, N.; and Yee, M. 2021. Improving learning-to-defer algorithms through fine-tuning. In Advances in Neural Infor- mation Processing Systems, Workshop on Human and Ma- chine Decisions.
  24. Scott, T.; Ridgeway, K.; and Mozer, M. C. 2018. Adapted deep embeddings: a synthesis of methods for k-shot induc- tive transfer learning. In Advances in Neural Information Processing Systems.
  25. Snell, J.; Swersky, K.; and Zemel, R. 2017. Prototypical networks for few-shot learning. In Advances in Neural In- formation Processing Systems.
  26. Sohn, K.; Berthelot, D.; Carlini, N.; Zhang, Z.; Zhang, H.; Raffel, C. A.; Cubuk, E. D.; Kurakin, A.; and Li, C.-L. 2020. FixMatch: simplifying semi-supervised learning with con- sistency and confidence. In Advances in Neural Information Processing Systems, 596-608.
  27. Tan, M.; and Le, Q. 2019. EfficientNet: rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 6105-6114.
  28. Tian, Y.; Wang, Y.; Krishnan, D.; Tenenbaum, J. B.; and Isola, P. 2020. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, 266-282.
  29. Topol, E. J. 2019. High-performance medicine: the conver- gence of human and artificial intelligence. Nature Medicine, 25(1): 44-56.
  30. Tversky, A.; and Kahneman, D. 1974. Judgment under un- certainty: heuristics and biases. Science, 185(4157): 1124- 1131.
  31. Vinyals, O.; Blundell, C.; Lillicrap, T.; Kavukcuoglu, K.; and Wierstra, D. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems.
  32. Welinder, P.; and Perona, P. 2010. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition -Workshops, 25-32.
  33. Wilder, B.; Horvitz, E.; and Kamar, E. 2020. Learning to complement humans. In International Joint Conference on Artificial Intelligence, 1526-1533.
  34. Zagoruyko, S.; and Komodakis, N. 2016. Wide residual net- works. In British Machine Vision Conference, 87.1-87.

Cited by

Sorted by citations

Authors
Hemmer, P., Thede, L., Vössing, M., Jakubik, J., & Kühl, N.
Affiliations
Karlsruhe Institute of Technology (DE)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2023-06-26
Volume
37
Issue
5
Pages
6002-6011
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v37i5.25742
OpenAlex
W4382239840
Cited by
10
References
3
Citations by year
2026: 4 · 2025: 4 · 2024: 2
FWCI
1.78
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
Bundesministerium für Wirtschaft und Klimaschutz
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
learning to defer, human-AI collaboration, embedding models, feature representation, radiology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review