Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts
Open accessAbstract
Machine learning (ML) models are increasingly being used in application domains that often involve working together with human experts. In this context, it can be advantageous to defer certain instances to a single human expert when they are difficult to predict for the ML model. While previous work has focused on scenarios with one distinct human expert, in many real-world situations several human experts with varying capabilities may be available. In this work, we propose an approach that trains a classification model to complement the capabilities of multiple human experts. By jointly training the classifier together with an allocation system, the classifier learns to accurately predict those instances that are difficult for the human experts, while the allocation system learns to pass each instance to the most suitable team member—either the classifier or one of the human experts. We evaluate our proposed approach in multiple experiments on public datasets with “synthetic” experts and a real-world medical dataset annotated by multiple radiologists. Our approach outperforms prior work and is more accurate than the best human expert or a classifier. Furthermore, it is flexibly adaptable to teams of varying sizes and different levels of expert diversity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The NIH Clinical Center X-ray dataset's main drawback is the rare overlap in the images reviewed by different experts.
“This dataset’s main drawback is the rare overlap in the images reviewed by different experts.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
The NIH Dataset section identifies the chest X-ray dataset, and the Experimental Setup reports that insufficient overlap between radiologists’ labeled images prevented the authors from including a third radiologist in a team. This supports limited overlap as a drawback for their experiments, but the paper does not call it the dataset’s main drawback. (LLM, conf=0.85)
Experimental — accuracy not yet measuredHemmer et al. found only four expert pairs with more than 800 shared image predictions, limiting their experiments to 2-radiologist teams.
“Hemmer et al.5 found only four expert pairs with more than 800 shared image predictions, limiting their experiments to 2-radiologist teams.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as Hemmer et al. [5]
The Experimental Setup says the NIH experiments used four radiologist pairs sharing more than 800 labeled images and could not include a third radiologist because of insufficient overlap. The two-radiologist limit applies to those NIH experiments, not all experiments: Section 5.2 evaluates teams with 2 to 10 synthetic human experts. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe only public dataset containing human predictions used in multi-expert learning-to-defer is the NIH Clinical Center X-ray dataset, aimed at detecting airspace opacity.
“To our knowledge, the only public dataset containing human predictions used in multi-expert L2D is the NIH Clinical Center X-ray dataset5,13,14, aimed at detecting airspace opacity; each instance features the predictions of 3 radiologists, sampled from a team of 22.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
Section 5.3 (NIH Dataset; page not provided) says the authors used an NIH Clinical Center chest X-ray dataset, focused on airspace opacity, and had individual annotations from three radiologists per image drawn from a cohort of 22. The Introduction distinguishes this dataset from two other datasets used in prior work, but does not establish that it is the only public dataset with human predictions used in multi-expert learning-to-defer. (LLM, conf=0.87)
Experimental — accuracy not yet measuredTraining and testing multi-expert learning-to-defer algorithms requires every expert's prediction for every instance.
“Training and testing multi-expert L2D algorithms requires every expert’s prediction for every instance5,12.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
Section 3 (Problem Formulation; page unavailable) explicitly assumes access to all m experts’ predictions for every training instance. Section 5.3 (Experimental Setup; page unavailable) evaluates pairs of radiologists only on images both labeled, supporting the same requirement for this paper’s testing setup. The paper does not establish that every multi-expert learning-to-defer algorithm has this requirement. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLearning to Defer algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.
“Learning to Defer (L2D) algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
The NIH Clinical Center X-ray dataset features predictions of 3 radiologists per instance, sampled from a team of 22.
“To our knowledge, the only public dataset containing human predictions used in multi-expert L2D is the NIH Clinical Center X-ray dataset5,13,14, aimed at detecting airspace opacity; each instance features the predictions of 3 radiologists, sampled from a team of 22.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
The paper chose to create a total of 50 experts because this far surpasses the numbers previously considered in learning-to-defer research.
“We chose to create a total of 50 experts as this far surpasses the numbers previously considered in L2D research4–13.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
The state-of-the-art approach to managing assignments in human-AI collaboration is learning to defer.
“The state-of-the-art approach to managing assignments in HAIC is learning to defer (L2D)4–13, in which algorithms choose whether to assign a task to a human expert or an ML model, leveraging their complementary strengths.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 23 · Cited by 29
Research works printed but not linked: 4
- Abir De, Nastaran Okati, Ali Zarezade, and Manuel Gomez-Rodriguez. Classification under human assistance. In AAAI, 2021.
- Grønsund and Aanestad, 2020] Tor Grønsund and Margunn Aanestad. Augmenting the algorithm: emerging human- in-the-loop work configurations. The Journal of Strategic Information Systems, 2020.
- Patrick Hemmer, Max Schemmer, Michael Vössing, and Niklas Kühl. Human-ai comple- mentarity in hybrid intelligence systems: a structured lit- erature review. In PACIS, 2021.
- Nastaran Okati, Abir De, and Manuel Rodriguez. Differentiable learning under triage. In NeurIPS, 2021.
- Authors
- Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J., & Satzger, G.
- Affiliations
- Karlsruhe Institute of Technology (DE)
- Venue
- Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
- Published
- 2022-07
- Pages
- 2478-2484
- OpenAlex
- W4285603627
- Cited by
- 29
- References
- 23
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 9 · 2023: 5
- FWCI
- 3.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- human-AI teaming, human-AI collaboration, task allocation, medical image annotation, radiologist annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
