A benchmarking framework and dataset for learning to defer in human-AI decision-making

Probably not a substantial claim: may not pass S3 (load-bearing).

In some real-world applications of human-AI collaboration, a Machine Learning model is trained to perform the same classification task as the human expert.

“In some real-world applications of HAIC1,21, an ML model is trained to perform the same classification task as the human expert.”

• cites [1] De-Arteaga, M., Fogliato, R. & Chouldechova, A. A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12 https://doi.org/10.1145/3313831.3376638 (ACM, Honolulu HI USA, 2020). sourceDisputed

  The paper’s Predictive model section says the tool estimates the probability of a child’s future referral or out-of-home placement. The Discussion says call workers decide whether to screen a referral in for investigation. In the real-world application this paper studies, the model therefore predicts different outcomes from the decision the human experts make, rather than performing the same classification task. (LLM, conf=0.84)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites [21] Amarasinghe, K. et al. On the importance of application-grounded experimental design for evaluating explainable ml methods. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 20921–20929, https://doi.org/10.1609/aaai.v38i19.30082 (2024). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The expert's performance is significantly impacted by seeing the Machine Learning model's score along with the rest of the information pertaining to each instance.

“It has been shown that the expert’s performance is significantly impacted by seeing the ML model’s score along with the rest of the information pertaining to each instance1,21,22.”

• cites [1] De-Arteaga, M., Fogliato, R. & Chouldechova, A. A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12 https://doi.org/10.1145/3313831.3376638 (ACM, Honolulu HI USA, 2020). sourcePartly supported

  In ‘Change in call workers’ behavior’ (page unavailable), the paper reports that child-welfare call workers changed which cases they screened in after the tool was deployed. ‘Overrides of erroneous scores’ (page unavailable) reports higher post-deployment precision. These findings support an effect on decisions and one measure of performance, but the paper does not establish a statistically significant performance effect from seeing the score itself, separate from the tool’s mandatory-screen-in policy. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [21] Amarasinghe, K. et al. On the importance of application-grounded experimental design for evaluating explainable ml methods. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 20921–20929, https://doi.org/10.1609/aaai.v38i19.30082 (2024). sourceSupported
• cites [22] Levy, A., Agrawal, M., Satyanarayan, A. & Sontag, D. Assessing the impact of automated suggestions on decision making: Domain experts mediate model errors but take less initiative. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–13 https://doi.org/10.1145/3411764.3445522 (2021). sourceDisputed

  The paper’s Results (§6.2; page unavailable) say that displaying model-confidence indicators had no noticeable effect on users’ accuracy or their correction of incorrect suggestions. The Discussion (§7; page unavailable) likewise says presenting model confidence had little-to-no impact. Although Results (§5.2) finds a significant effect of annotation mode on recall, that is not a finding that seeing the model’s score significantly affected performance. (LLM, conf=0.87)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Learning to Defer algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.

“Learning to Defer (L2D) algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.”

• cites [4] Charusaie, M.-A., et al. (eds.) International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol. 162 of Proceedings of Machine Learning Research, 2972–3005 https://doi.org/10.48550/arXiv.2207.09584 (PMLR, 2022). sourceSupported
• cites [5] Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J. & Satzger, G. Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts. In Raedt, L. D. (ed.) Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, 2478–2484 https://doi.org/10.24963/ijcai.2022/344 (ijcai.org, 2022). sourceSupported
• cites [6] Raghu, M. et al. Direct uncertainty prediction for medical second opinions. In International Conference on Machine Learning, 5281–5290 https://doi.org/10.48550/arXiv.1807.01771 (PMLR, 2019). sourcePartly supported

  The Abstract (p. 1) and Introduction (p. 1) say the model identifies cases that may benefit from a doctor's second opinion, supporting the idea of directing selected cases to human experts. They do not describe Learning to Defer algorithms or select cases because a human is predicted to be more accurate than the AI model; the selection criterion is predicted disagreement among doctors. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [7] Raghu, M. et al. The Algorithmic Automation Problem: Prediction, Triage, and Human Effort. CoRR abs/1903.12220 https://doi.org/10.48550/arXiv.1903.12220 (2019). sourcePartly supported

  Section 4 (p. 8) says the paper can predict which instances humans or algorithms will handle better and divide work accordingly. Section 4.2.1 (p. 10) reports performance gains from that triage. The copy supports the deferral principle, but does not identify or evaluate the approach as “Learning to Defer” (L2D) algorithms. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [8] Mozannar, H. & Sontag, D. A. Consistent Estimators for Learning to Defer to an Expert. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, vol. 119 of Proceedings of Machine Learning Research, 7076–7087 https://doi.org/10.48550/arXiv.2006.01862 (PMLR, 2020). sourceSupported
• cites [9] Mozannar, H. et al. Who should predict? exact algorithms for learning to defer to humans. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), https://doi.org/10.48550/arXiv.2301.06197 (2023). sourceSupported
• cites [10] Madras, D., Pitassi, T. & Zemel, R. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In Advances in Neural Information Processing Systems, vol. 31 https://doi.org/10.48550/arXiv.1711.06664 (Curran Associates, Inc., 2018). sourceSupported
• cites [11] Verma, R. & Nalisnick, E. T. Calibrated Learning to Defer with One-vs-All Classifiers. In Chaudhuri, K. et al. (eds.) International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol. 162 of Proceedings of Machine Learning Research, 22184–22202 https://doi.org/10.48550/arXiv.2202.03673 (PMLR, 2022). sourceSupported
• cites [12] Verma, R., Barrejón, D. & Nalisnick, E. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, 11415–11434 https://doi.org/10.48550/arXiv.2210.16955 (PMLR, 2023). sourceSupported
• cites [13] Hemmer, P., Thede, L., Vössing, M., Jakubik, J. & Kühl, N. Learning to defer with limited expert predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 6002–6011 https://doi.org/10.1609/aaai.v37i5.25742 (2023). sourceSupported

May combine two claims.

For α=0, the probability of error is the same for all instances within a class, which is equivalent to class-dependent label noise.

“For α=0, the probability of error is the same for all instances within a class, which is equivalent to class-dependent label noise, as used in previous L2D testing4,8,11.”

• cites [4] Charusaie, M.-A., et al. (eds.) International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol. 162 of Proceedings of Machine Learning Research, 2972–3005 https://doi.org/10.48550/arXiv.2207.09584 (PMLR, 2022). sourceCould not verify

  The Problem Setting (p. 3) allows expert predictions to depend on both the instance and its label, but does not define an α parameter or class-dependent label noise. The Experimental Illustration (p. 13) describes a different synthetic error pattern. Searches for α=0 and class-dependent label noise found no passage establishing the claimed equivalence. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [8] Mozannar, H. & Sontag, D. A. Consistent Estimators for Learning to Defer to an Expert. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, vol. 119 of Proceedings of Machine Learning Research, 7076–7087 https://doi.org/10.48550/arXiv.2006.01862 (PMLR, 2020). sourceSupported
• cites [11] Verma, R. & Nalisnick, E. T. Calibrated Learning to Defer with One-vs-All Classifiers. In Chaudhuri, K. et al. (eds.) International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol. 162 of Proceedings of Machine Learning Research, 22184–22202 https://doi.org/10.48550/arXiv.2202.03673 (PMLR, 2022). sourceSupported

Classified as probably not a substantial claim.

Humans and AI have complementary strengths and weaknesses.

“Recently, a growing body of research has focused on human-AI collaboration (HAIC), with several authors arguing that humans and AI have complementary strengths and weaknesses1,2.”

• cites [1] De-Arteaga, M., Fogliato, R. & Chouldechova, A. A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12 https://doi.org/10.1145/3313831.3376638 (ACM, Honolulu HI USA, 2020). sourceSupported
• cites [2] Dellermann, D., Ebel, P., Soellner, M. & Leimeister, J. M. Hybrid. Intelligence. Business & Information Systems Engineering 61, 637–643, https://doi.org/10.1007/s12599-019-00595-2 (2019). sourceSupported

May combine two claims.

Hemmer et al. found only four expert pairs with more than 800 shared image predictions, limiting their experiments to 2-radiologist teams.

“Hemmer et al.5 found only four expert pairs with more than 800 shared image predictions, limiting their experiments to 2-radiologist teams.”

• cites Hemmer et al. [5] Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J. & Satzger, G. Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts. In Raedt, L. D. (ed.) Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, 2478–2484 https://doi.org/10.24963/ijcai.2022/344 (ijcai.org, 2022). sourcePartly supported

  The Experimental Setup says the NIH experiments used four radiologist pairs sharing more than 800 labeled images and could not include a third radiologist because of insufficient overlap. The two-radiologist limit applies to those NIH experiments, not all experiments: Section 5.2 evaluates teams with 2 to 10 synthetic human experts. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Detecting some conditions was much harder even for experienced doctors, resulting in lower intra-rater reliability κ∈[0.4,0.85].

“However, detecting some conditions was much harder even for experienced doctors, resulting in a lower intra-rater reliability κ∈[0.4,0.85].”

• cites [33] Remenyi, B. et al. Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the world heart federation evidence-based criteria. Heart Asia 11 https://doi.org/10.1136/heartasia-2019-011233 (2019). sourcePartly supported

  Table 2 (page unavailable in our copy) reports lower intra-rater κ values for some rheumatic-heart-disease features, including 0.37 for chordal thickening and 0.41 for excessive leaflet motion. The limitations section says the readers were cardiologists experienced in RHD. The claimed 0.4–0.85 range is not the reported range of median intra-rater values: some are below 0.4, while pathological mitral regurgitation reaches 0.92. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

Real-world experts' decisions depend on an instance's features and an auxiliary model score shown to them, a common practice in fraud prevention.

“However, this data is extremely useful to evaluate how real-world experts’ decisions depend on an instance’s features as well as an auxiliary model score shown to them, a common practice in fraud prevention21,28.”

• cites [21] Amarasinghe, K. et al. On the importance of application-grounded experimental design for evaluating explainable ml methods. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 20921–20929, https://doi.org/10.1609/aaai.v38i19.30082 (2024). sourceSupported
• cites [28] Han, J., Huang, Y., Liu, S. & Towey, K. Artificial intelligence for anti-money laundering: a review and extension. Digital Finance 2, 211–239, https://doi.org/10.1007/s42521-020-00023-1 (2020). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The concern of fairness was explored by the original authors in their paper, where they consider adding a fairness regularization term to the loss function of their L2D system.

“While this is a concern that goes unaddressed by most recent publications in this area, it was explored by the original authors in their paper10, where they consider adding a fairness regularization term to the loss function of their L2D system.”

• cites [10] Madras, D., Pitassi, T. & Zemel, R. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In Advances in Neural Information Processing Systems, vol. 31 https://doi.org/10.48550/arXiv.1711.06664 (Curran Associates, Inc., 2018). sourceSupported

May combine two claims.

The assumption that ground-truth labels exist is common to all cited works and is considered necessary for an L2D algorithm to estimate the human's probability of correctness for a given instance.

“the assumption that ground-truth labels exist is common to all cited works, and is considered necessary for an L2D algorithm to be able to learn to estimate the human's probability of correctness for a given instance.”

• cites [13] Hemmer, P., Thede, L., Vössing, M., Jakubik, J. & Kühl, N. Learning to defer with limited expert predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 6002–6011 https://doi.org/10.1609/aaai.v37i5.25742 (2023). sourcePartly supported

  The Introduction says learning-to-defer algorithms require ground-truth labels and expert predictions. The Problem Formulation and Training of Expertise Predictor Model sections show that this paper uses ground truth to determine whether an expert's prediction is correct. The paper does not establish that the assumption is common to all works cited by the citing document, or that ground truth is necessary for every L2D algorithm to estimate instance-level human correctness. Page numbers are unavailable in our copy. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [40] Tailor, D., Patra, A., Verma, R., Manggala, P. & Nalisnick, E. Learning to defer to a population: A meta-learning approach. In International Conference on Artificial Intelligence and Statistics, 3475–3483, https://doi.org/10.48550/arXiv.2403.02683 (PMLR, 2024). sourcePartly supported

  Section 2.1 defines L2D training data as features, labels, and human predictions; the Learning section uses whether a human prediction matches the true label to model correctness. These passages support the role of ground-truth labels in the methods discussed, but do not establish that all cited works assume them or that they are necessary for every L2D algorithm. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

In a more challenging diagnostic task, inter-rater reliability was on average κ=0.49, and the proportion of agreement between experts was pa=66.29%.

“It was found that in a more challenging diagnostic task, inter-rater reliability was, on average, κ=0.49. The proportion of agreement between experts was also reported at pa=66.29%.”

• cites [33] Remenyi, B. et al. Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the world heart federation evidence-based criteria. Heart Asia 11 https://doi.org/10.1136/heartasia-2019-011233 (2019). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

In the paper introducing the OvA multi-expert method, the authors report ECE values rarely surpassing 4% across three separate datasets.

“In the paper where the OvA multi-expert method is introduced12, the authors also report the ECE in three separate datasets, with values rarely surpassing 4%.”

• cites [12] Verma, R., Barrejón, D. & Nalisnick, E. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, 11415–11434 https://doi.org/10.48550/arXiv.2210.16955 (PMLR, 2023). sourcePartly supported

  Section 7.2 reports calibration error for the OvA method on HateSpeech, HAM10000, and Galaxy-Zoo, with results in the bottom row of Figure 1. The passages in our copy say OvA has the best calibration but do not establish that its ECE values rarely surpass 4%. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

A recent meta-analysis of hiring practices shows that discrimination against candidates with disabilities, older candidates, less physically attractive candidates, and candidates with salient racial or ethnic traits is still an issue in Europe and the United States.

“A recent meta-analysis of hiring practices34 shows that discrimination against candidates with disabilities, older candidates, less physically attractive candidates, and candidates with salient racial or ethnic traits is still an issue in Europe and the United States.”

• cites [34] Lippens, L., Vermeiren, S. & Baert, S. The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments. European Economic Review 151, 104315, https://doi.org/10.1016/j.euroecorev.2022.104315 (2023). sourceSupported

Classified as probably not a substantial claim.

For all capacity constraints, the deferral rate is set to 10/11, ensuring that on average each decision maker (each of the 10 humans or the classifier) predicts on 1/11 of the instances.

“For all capacity constraints, the deferral rate, that is, the fraction of instances that are deferred to the 10 expert team, is set to 10/11, to ensure that, on average, each decision maker (each of the 10 humans or the classifier) predicts on 1/11 of the instances.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Testing Scenarios are defined for each of the five teams.

“The Testing Scenarios are defined for each of the five teams, as we assume that the team of experts available in training is the same team that will be queried in testing”

• AssumptionThe authors state this as an assumption: The team of experts available in training is the same team that will be queried in testing.

Probably not a substantial claim: may not pass S3 (load-bearing).

The alerts.parquet file contains the subset of instances flagged by the Alert Model, i.e., instances with a model_score above the threshold determined in Alert Model, calculated to ensure maximal recall at 5% FPR.

“Finally, the file “alerts.parquet” contains the subset of instances flagged by the Alert Model, that is, the instances with a ‘model_score’ value above the threshold determined in Alert Model, calculated to ensure maximal recall at 5% FPR.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The expert_parameters.parquet file stores the value of each expert's set of parameters sampled in the data generation process.

“In the file “expert_parameters.parquet”, we store the value of each expert’s set of parameters, which were sampled in the data generation process detailed in Section Synthetic Expert Generation.”

May combine two claims.

OpenL2D allows users to generate realistic training datasets with limited expert predictions due to user-defined capacity constraints, as well as testing conditions subject to user-defined expert availability and capacity.

“(iii) generate realistic training datasets with limited expert predictions due to user-defined capacity constraints, as well as testing conditions subject to user-defined expert availability and capacity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The train.parquet file contains the instances pertaining to months 4-7 of the file alerts.parquet, with assignment and decision fields added.

“This file contains the instances pertaining to months 4–7 of the file ‘alerts.parquet’, described in the Subsection alert data. To each bank account opening application, the following two fields are added: ‘assignment’, which contains the identifier of the expert the application was deferred to, in the format ‘standard#{i}’); and “decision”, which contains expert i’s prediction for said instance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Base.csv file contains one million rows, each representing a feature-engineered bank account application.

“This file contains one million rows, each representing a feature-engineered bank account application with the following fields:”

Classified as probably not a substantial claim.

OpenL2D takes as input a tabular dataset for a binary classification task and generates supplementary data including synthetic expert predictions and training and testing conditions based on work capacity constraints.

“OpenL2D takes as an input a tabular dataset for a binary classification task, represented on the center left of the diagram. OpenL2D generates the supplementary data necessary to allow for thorough L2D research in said binary tabular dataset, including synthetic expert predictions, as well as training and testing conditions based on the work capacity constraints of the expert team.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 25 training scenarios were generated by combining 5 distinct teams of 10 experts with 5 different seeds for the distribution of cases throughout batches.

“Combining these 5 different teams with 5 different seeds for the distribution of cases throughout batches, we generate a total of 25 total training scenarios.”

May combine two claims.

The FiFAR dataset includes 25 constrained training scenarios with only one expert prediction per instance.

“Our dataset includes 25 constrained training scenarios with only one expert prediction per instance.”

Classified as probably not a substantial claim.

Month 8 is left for testing, on the subset of instances flagged by the Alert Model in that time period.

“Month 8 is left for testing, on the subset of instances flagged by the Alert Model in that time period.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The FiFAR dataset is available for download through the figshare platform.

“The FiFAR dataset31 has been made available for download through the figshare platform (see citation for DOI link).”

• cites [31] Alves, J. et al. Financial fraud alert review dataset. figshare https://doi.org/10.6084/m9.figshare.28351172. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

OpenL2D is designed to work with any tabular dataset for binary classification.

“Our framework is designed to work with on any tabular dataset for binary classification, where each instance i is represented by a feature vector xi and the label given by yi∈{0,1}.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The train_alert folder contains a sub-folder for each of the 25 training scenarios.

“The “train_alert” folder contains a sub-folder for each of the 25 training scenarios.”

Classified as probably not a substantial claim.

All training scenarios simulate a period from months 4 to 7 in which a random deferral system is deployed and alerted applications are randomly deferred to a team of experts.

“All scenarios aim to simulate a period from months 4 to 7 (see Fig. 2), where a random deferral system is deployed, meaning that the alerted applications are randomly deferred to a team of experts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The BAF_alert_model_score.parquet file contains the entirety of the instances pertaining to months 4-8 of the original Base.csv, with months 1-3 used in training, accompanied by case_id and model_score columns.

“The sub-directory “processed_data”, we include the file “BAF_alert_model_score.parquet”, which contains the entirety of the instances pertaining to months 4–8 of the original “Base.csv” (months 1–3 were used in training), accompanied by two additional columns:”

May combine two claims.

The private dataset of real transaction fraud analysts contains only a single expert's prediction per instance, making it impossible to calculate inter- and intra-rater agreement metrics.

“Note that this private dataset was not used in any of the previously mentioned sections as it contains only a single expert’s prediction per instance, making it impossible to calculate inter and intra-rater agreement metrics.”

Classified as probably not a substantial claim. May combine two claims.

Each training scenario represents a simulated period of time where a random deferral system is used, creating a realistic expert prediction set resulting from collection of a single prediction per instance.

“As mentioned in Section Training Scenarios, each scenario represents a simulated period of time where a random deferral system is used, thus creating a realistic expert prediction set, resulting from the collection of a single prediction per instance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This creates a set of relatively similar decision-making processes allowing for the possibility of some outliers.

“In this way, we create a set of relatively similar decision-making processes allowing for the possibility of some outliers (see Subsections “Inter-Rater Agreement” and “Feature Dependence and Algorithmic Bias”).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The test directory contains a set of 25 sub-directories pertaining to each of the testing scenarios.

“The directory “test” contains a set of 25 sub-directories pertaining to each of the testing scenarios.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The expert_predictions.parquet file contains every expert's prediction for every single alerted instance.

“The file “expert_predictions.parquet” contains every expert’s prediction for every single alerted instance.”

Classified as probably not a substantial claim. May combine two claims.

OpenL2D allows users to tweak FiFAR's experts and their capacity constraints to suit their benchmarks and generate complex synthetic experts in binary classification tabular datasets.

“Although FiFAR is designed to be used out-of-the-box, OpenL2D allows users to tweak FiFAR’s experts and their capacity constraints to suit their benchmarks and generate complex synthetic experts in binary classification tabular datasets, allowing for increased variety and complexity in L2D benchmarking.”

May combine two claims.

The private dataset contains missing ground-truth labels, making it impossible to evaluate expert performance distributions.

“Furthermore, the dataset contains missing ground-truth labels, thus making it impossible to evaluate expert performance distributions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For each testing scenario, due to the limited size of the test split (instances pertaining to month 8 in alerts.parquet), the entirety of the test set was considered as a single batch of size 4457.

“For each of these scenarios, due to the limited size of the test split (i.e. instances pertaining to month 8 in the file ‘alerts.parquet’, described in the Subsection alert data), the entirety of the test set was considered as a single batch of size 4457.”

Classified as probably not a substantial claim.

The framework generates an expert prediction table, the set of predictions of every expert for every instance in the dataset.

“Then, it generates the expert prediction table, which is the set of predictions of every expert for every instance in the dataset, represented in the top right.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prob_of_error.parquet file contains, for each case_id, the probability of false positive and false negative error for each expert, calculated according to Equation 1.

“The file “prob_of_error.parquet” is a table containing, for each entry identified by the’case_id’ value, the probability of false positive and false negative error for each expert n ∈0,…,49, calculated according to Equation 1.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For each team, five different capacity constraints are imposed in testing: one homogeneous and four variable with σd=0.2.

“For each team, five different capacity constraints are imposed in testing: one homogeneous and four variable with σd=0.2.”

May combine two claims.

Both OvA and DeCCaF obtain significantly better results than all non-L2D baselines, achieving an average 11% and 15% decrease, respectively, in misclassification cost over the Random Assignment baseline.

“both OvA and DeCCaF obtain significantly better results than all non-L2D baselines, with the methods achieving an average 11% and 15% decrease, respectively, in the misclassification cost over the Random Assignment baseline.”

May combine two claims.

Classifier h achieves a ROC AUC of 0.71 in testing, indicating it can correctly rank instances by probability of fraud.

“we use the ROC AUC, which has a value of 0.71, indicating that the classifier h is able to correctly rank instances according to the probability of fraud.”

May combine two claims.

Jointly modeling the team's behavior (DeCCaF) leads to a slight increase in average AUC (4% on average) and a significant improvement in calibration (17% on average) compared with OvA.

“jointly modeling the team's behavior leads to a slight increase in the average AUC (4% on average), and a significant improvement in the calibration of the estimates (17% on average).”

May combine two claims.

Full Rejection is the worst option, as all possible decision makers (experts and classifier h) have a lower misclassification cost over the test set.

“As expected, the worst option is Full Rejection, as all possible decision makers (experts and classifier h), have a lower misclassification cost over the test set.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The FiFAR dataset is publicly accessible at 10.6084/m9.figshare.28351172.

“The FiFAR dataset is publicly accessible at 10.6084/m9.figshare.28351172.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The set of decisions pertaining to each of the 20 real-world experts had an average of 10 K instances.

“As the set of decisions pertaining to each of the 20 real-world experts had an average of 10 K instances”

The quote only partly states this claim.

For the synthetic experts, the proportion gains are, across the board, higher for all features than for the real experts.

“In these results, we can see that the value of the proportion Gains are, across the board, higher for all features.”

May combine two claims.

Some experts within the team, such as experts #3 and #4, have a particularly unique set of decisions, yielding very low agreement with most of the team's members (an average of κ=0.33 and κ=0.28, respectively), but having high agreement between them (κ=0.71).

“In this set of results we can immediately notice how some of the experts within our team, such as experts #3 and #4, have a particularly unique set of decisions, yielding very low agreement with most of the team’s members (an average of κ=0.33 and κ=0.28, respectively), but having high agreement between them (κ=0.71).”

May combine two claims.

In both the real and the synthetic expert teams, the model score is the most important feature, followed by a similar distribution of feature importances.

“In both expert teams, the model score is the most important feature, followed by a similar distribution of feature importances.”

May combine two claims.

Our expert team is consistently biased against older customers, at varying degrees, with some experts being almost 50% more likely to incorrectly reject an older customer's application.

“Our expert team is consistently biased against older customers, at varying degrees, with some experts being almost 50% more likely to incorrectly reject an older customer’s application.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Panel b of Fig. 10 shows the expected misclassification cost distribution calculated over months 4-8 in the dataset.

“Panel ‘b’ of Fig. 10 shows the expected misclassification cost distribution calculated over months 4–8 in the dataset.”

May combine two claims.

The fraud prevalence within the alerts for Older customers is approximately 13%, while the prevalence for Younger customers is approximately 11%.

“The fraud prevalence within the alerts for the Older customers is ≈13%, while the prevalence for Younger customers is ≈11%.”

May combine two claims.

The agreement between experts varies substantially, with κ∈[0.06,0.79] and pa∈[0.32,0.95].

“Plotting the distribution for the two agreement metrics, in Fig. 9, we can see that the agreement between experts varies substantially, with κ∈[0.06,0.79], and pa∈[0.32,0.95].”

May combine two claims.

The BAF dataset contains 30 features, while the dataset of real-world financial analyst decisions contains 116.

“the BAF dataset contains 30 features, while the dataset of real-world financial analyst decisions contains 116”

Probably not a substantial claim: may not pass S3 (load-bearing).

The analysis notebook is completely dataset and generation agnostic.

“This notebook is completely dataset and generation agnostic”

May combine two claims.

An increase in the value of α will not always result in higher intra-rater reliability, particularly for lower values, but an increase in intra-rater reliability is expected as α rises.

“Should the values of β0, β1 be 0 or 1, α=0 would also result in a deterministic process, as such, an increase within in the value of α will not always result in higher intra-rater reliability, particularly for lower values, but we expect to see an increase in intra-rater reliability as the value of α rises.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The value of α can be seen as a measure of the consistency, or intra-rater reliability, of a decision maker, as larger values of α will result in experts who are more likely to produce the same decisions when faced with similar inputs.

“The value of α can therefore be seen as a measure of the consistency, or intra-rater reliability, of a decision maker, as larger values of α will result in experts who are more likely to produce the same decisions when faced with similar inputs.”

May combine two claims.

The generated experts will tend to produce false positives when a feature's value is positively correlated with the label, and false negatives when negatively correlated.

“This means that the generated experts will tend to produce false positives/negatives if a feature’s value is positively/negatively correlated with the label.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For very large α, the probability would saturate at the extremes of the sigmoid function, 0 or 1, resulting in a deterministic decision-making process.

“In turn, for very large α, the probability would saturate at the extremes of the range of the sigmoid function, 0 or 1, resulting in a deterministic decision-making process.”

May combine two claims.

To select the distribution of α, the paper conducted experiments to ensure that the intra-rater consistency was similar to that of real-world experts, selecting μα=15, σα=3.5.

“To select the distribution of the α parameter (see Equation (1)), we conducted a series of experiments in order to ensure that the intra-rater consistency was similar to that of real-world experts, selecting μα=15,σα=3.5.”

Probably not a substantial claim: may not pass S4 (based).

If the probabilities of error are closer to the edges of the interval (0,1), then the resulting sets of decisions are expected to be much more similar between them.

“If the probabilities of error are closer to the edges of the interval (0,1), then we expect the resulting sets of decisions to be much more similar between them.”

May combine two claims.

In a study involving seven experienced software professionals estimating effort for sixty software development tasks over three months, the mean difference of effort estimates of the same task by the same estimator was as much as 71%, with correlation between corresponding estimates being 0.7.

“The authors find that the mean difference of the effort estimates of the same task by the same estimator was as much as 71%, with the correlation between the corresponding estimates being 0.7.”

• cites [32] Grimstad, S. & Jørgensen, M. Inconsistency of expert judgment-based estimates of software development effort. Journal of Systems and Software 80, 1770–1777, https://doi.org/10.1016/j.jss.2007.03.001 (2007). sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

A large number of experts allows users to test the robustness of learning-to-defer systems to changes in the expert team by sub-sampling different, smaller sets of experts.

“Furthermore, a large number of experts allows users to test the robustness of L2D systems to changes in the expert team (i.e., teams that work on different shifts), by sub-sampling different, smaller sets of experts.”

May combine two claims.

These synthetic experts' decisions depend only on an instance's class, meaning the methods in these benchmarks are not challenged to learn nuanced and varied types of expertise.

“However, these synthetic experts’ decisions depend only on an instance’s class, meaning the methods in these benchmarks are not challenged to learn nuanced and varied types of expertise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The values pertaining to the distribution of the model score weight wM were manually chosen to produce a dependency on the model score similar to real-world fraud analysts.

“The values pertaining to the distribution of the model score weight wM were manually chosen to produce a dependency on the model score similar to real-world fraud analysts (see Subsection “Feature Dependence and Algorithmic Bias”).”

Classified as probably not a substantial claim.

The distribution of the protected attribute's weight wp was manually defined so that the experts exhibit varying levels of bias against older customer's applications.

“The distribution of the protected attribute’s weight wp was also manually defined, chosen so that the experts exhibit varying levels of bias against older customer’s applications (see subsection “Unfairness Against Social Groups”).”

Classified as probably not a substantial claim.

The train.parquet file simulates realistic data availability for L2D methods developed under the proposed framework.

“L2D methods developed under our proposed framework (i.e., a single expert prediction per instance, with work capacity constrained deferral) should take as input the ‘train.parquet’ file only, as it simulates realistic data availability.”

Classified as probably not a substantial claim.

The paper imposes bounds for the FPR and FNR of experts, ensuring that FNRj∈[0.01,1] and FPRj∈[0.01,1], in order to guarantee that the label flipping process produces both types of errors for all experts.

“We also impose bounds for the FPR and FNR of experts, ensuring that FNRj∈[0.01,1] and FPRj∈[0.01,1], in order to guarantee that the label flipping process produces both types of errors for all experts.”

Classified as probably not a substantial claim. May combine two claims.

The number of experts for testing capacity constraints are sampled using the same n_experts_seed values defined in training, guaranteeing that the sampled expert teams are the same in testing as in training.

“The number of experts for these capacity constraints are sampled by using the same n_experts_seed values defined in training. This guarantees that the sampled expert teams are the same in testing as in training.”

Classified as probably not a substantial claim. May combine two claims.

All other feature weights are sampled from N(0,0.05) to introduce further variation in the decision-making processes, but guaranteeing that the experts are not too dissimilar.

“All other feature weights are sampled from N(0,0.05), in order to introduce further variation in the decision-making processes, but guaranteeing that the experts are not too dissimilar.”

Classified as probably not a substantial claim. May combine two claims.

The paper selects the 15 features with the highest correlation with the label and sets their respective μ=ρ, σ=0.1, making each feature's influence on the probability of error scale according to their correlation with the label, but introducing variations in the decision-making processes of different experts.

“We then select the 15 features with the highest correlation with the label (either negative or positive), and set their respective μ=ρ,σ=0.1, making each feature’s influence on the probability of error scale according to their correlation with the label, but introducing variations in the decision-making processes of different experts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If an expert's probabilities of error are consistently approximately 0.5, the resulting decision process would be entirely random.

“If an expert’s probabilities of error are consistently ≈0.5, the resulting decision process would be entirely random.”

Probably not a substantial claim: may not pass S2 (checkable).

It is important to consider diverse expert behavior in learning-to-defer benchmarking.

“Our L2D benchmark reveals that performance rankings of L2D algorithms vary significantly based on the available experts, highlighting the need to consider diverse expert behavior in L2D benchmarking.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In a real-world scenario, the capacity of the classifier would be orders of magnitude larger than that of the human experts, as the cost of querying a model is much lower than that of querying a human expert.

“In a real-world scenario, the capacity of the classifier would be orders of magnitude larger than that of the human experts, as the cost of querying a model is much lower than that of querying a human expert.”

Classified as probably not a substantial claim.

The synthetic experts cannot replace, in any way, humans involved in HAIC systems, because real-world expert data would be necessary to train a system for real-world application.

“our synthetic experts can not replace, in any way, humans involved in HAIC systems, as it would be necessary to gather real-world expert data in order to train the system to be used in a real-world application.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

When implementing L2D methods, it is desirable for inter-expert agreement not to be perfect; if experts perfectly agree on every instance, there would be no benefit in attempting to intelligently assign within that expert team.

“When implementing L2D methods, it is desirable for inter-expert agreement not to be perfect. Should the experts perfectly agree on every instance, there would be no benefit in attempting to intelligently assign within that expert team.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Vector embeddings are, by their very nature, uninterpretable.

“vector embeddings are, by their very nature, uninterpretable.”

Probably not a substantial claim: may not pass S2 (checkable).

To measure fairness, we need to take the prevalence of each group into account.

“As such, to measure fairness, we need to take the prevalence of each group into account.”

May combine two claims.

Because the real-world dataset has no ground-truth labels and only one expert prediction per instance, the performance and intra-rater reliability of the real-world experts cannot be determined, and the source of the disparity in predictability cannot be identified.

“we are not able to determine the performance and intra-rater reliability of the real-world experts, making it impossible to identify the source of this disparity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A real-time, interactive visualization of the impact of each parameter change on the expert team is impossible due to computational restrictions.

“While a real-time, interactive visualization of the impact of each change on the expert team (i.e., by controlling each parameter with sliders and viewing a change's impact on key metrics such as inter and intra rater agreement) would be ideal, this is impossible due to computational restrictions.”

Classified as probably not a substantial claim. May combine two claims.

The dataset may be used to develop fairness-aware methods, as L2D systems can have an active role in intelligently assigning instances in a way that promotes fairness.

“Furthermore, it may also be used to develop fairness-aware methods, as L2D systems can have an active role in intelligently assigning instances in a way that promotes fairness (i.e., deferring the older customer’s applications to the unbiased experts).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The FiFAR dataset can be reproducibly re-generated by running the code detailed in the Code availability section.

“The FiFAR dataset can be reproducibly re-generated by running the code detailed in Section Code availability.”

May combine two claims.

The study shows that the Recall and FPR of highly trained experts varies drastically throughout the team, with FPR varying between ≈1% and ≈10%, while recall varies between ≈35% and ≈99%.

“While we cannot derive a cost structure for the classification task, and can not, therefore, evaluate the difference in performance throughout the team, this study shows that the Recall and FPR of highly trained experts varies drastically throughout the team, with FPR varying between ≈1% and ≈10%, while recall varies between ≈35% and ≈99%.”

• cites [25] Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316, 2402–2410, https://doi.org/10.1001/jama.2016.17216 (2016). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

If a team is comprised of experts with low intra-rater consistency, they will have more unpredictable behaviour, harming the performance of L2D systems.

“If a team is comprised of experts with low intra-rater consistency, they will have more unpredictable behaviour, harming the performance of L2D systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

As the classification task is highly imbalanced, κ values are expected to be lower, as the expected level of agreement by chance rises.

“As the classification task is highly imbalanced, κ values are expected to be lower, as the expected level of agreement by chance rises.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

In some use cases where domain-specific expertise is needed, annotation services may not be of use to practitioners, and it may be impossible or unfeasible to obtain real human expert data.

“in some use cases, where domain-specific expertise is needed, such annotation services may not be of use to practitioners, and it may be impossible/unfeasible to obtain real human expert data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the lending setting, even if the sampling process of loan applicants' income is unbiased and accurate, the data could still suffer from societal bias if some social group, for example women, systematically receive lower wages, making women less likely to be able to pay back a loan.

“In the lending setting, even if the sampling process of loan applicants’ income is unbiased and accurate, the data could still suffer from societal bias if some social group, for example women, systematically receive lower wages, making women less likely to be able to pay back a loan.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tweaking the parameters of the framework until the desired behavior is achieved is time-consuming, because the interaction between each variable is not trivial.

“We also acknowledge that the process of tweaking the parameters of our framework until the desired behavior is achieved is time-consuming, as the interaction between each variable is not trivial.”

Probably not a substantial claim: may not pass S4 (based).

The creation of realistic synthetic experts requires an understanding of human decision-making literature.

“A key restriction of our framework is that the creation of realistic synthetic experts requires an understanding of human decision-making literature.”

• AssumptionRests on an assumption: Human decision-making traits observed across a wide variety of domains include biases, imperfect inter- and intra-rater agreement, and greater dependence on features that correlate highly with the label.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In this case, a smaller fraction of women would have a positive ground truth label, meaning that a perfectly accurate algorithm would reject a higher proportion of women's applications when compared to men's applications, due to underlying socio-economic factors.

“In this case, a smaller fraction of women would have a positive ground truth label, meaning that a perfectly accurate algorithm would reject a higher proportion of women’s applications when compared to men’s applications, due to underlying socio-economic factor.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The framework motivates the use of more complex synthetic expert data only when real human predictions are unattainable, without posing a threat to current existing annotation services.

“we believe that our framework motivates the use of more complex synthetic expert data only when real human predictions are unattainable, without posing a threat to current existing annotation services.”

Probably not a substantial claim: may not pass S3 (load-bearing).

To ensure reproducibility, the Alert Model and all models used in the L2D benchmark are made available.

“In order to ensure reproducibility, the Alert Model, as well as all models used in the L2D benchmark are also made available.”

Classified as probably not a substantial claim.

OpenL2D and FiFAR are meant to be used as a benchmarking tool only, allowing the study of new L2D approaches on better synthetic data than currently used.

“OpenL2D and FiFAR are meant to be used as a benchmarking tool only, allowing for the study of new L2D approaches on better synthetic data than currently used.”