Towards Unbiased and Accurate Deferral to Multiple Experts
Open accessAbstract
Machine learning models are often implemented in cohort with humans in the pipeline, with the model having an option to defer to a domain expert in cases where it has low confidence in its inference. Our goal is to design mechanisms for ensuring accuracy and fairness in such prediction systems that combine machine learning model inferences and domain expert predictions. Prior work on "deferral systems" in classification settings has focused on the setting of a pipeline with a single expert and aimed to accommodate the inaccuracies and biases of this expert to simultaneously learn an inference model and a deferral system. Our work extends this framework to settings where multiple experts are available, with each expert having their own domain of expertise and biases. We propose a framework that simultaneously learns a classifier and a deferral system, with the deferral system choosing to defer to one or more human experts in cases of input where the classifier has low confidence. We test our framework on a synthetic dataset and a content moderation dataset with biased synthetic experts, and show that it significantly improves the accuracy and fairness of the final predictions, compared to the baselines. We also collect crowdsourced labels for the content moderation task to construct a real-world dataset for the evaluation of hybrid machine-human frameworks and show that our proposed framework outperforms baselines on this real-world dataset as well.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Learning to defer requires uncertainty quantification and confidence estimation for the underlying AI models.
“Learning to defer in human-AI settings is an active area of research [61], necessitating uncertainty quantification and confidence estimation for the underlying AI models.”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [61]
Learning to defer in human-AI settings is an active research area.
“Learning to defer in human-AI settings is an active area of research [61], necessitating uncertainty quantification and confidence estimation for the underlying AI models.”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
(2010) · Cited by 8,261 · see citing works
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
(2018) · Cited by 3,510 · see citing works
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709 · see citing works
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610 · see citing works
Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 2× · see citing works
“Prior studies on settings where human-in-the-loop frameworks have been implemented provide evidence of such complexities [16, 2, 26, 55].”
BackgroundFoundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122 · see citing works
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
The Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · see citing works
Predicting the Type and Target of Offensive Posts in Social Media
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., & Kumar, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1144 · Cited by 781 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
Edwards, L., & Veale, M. (2017) · doi:10.31228/osf.io/97upg · Cited by 524 · see citing works
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · see citing works
On Fairness and CalibrationPreprint
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J. M., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02012 · Cited by 509 · see citing works
Fairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · 2011 IEEE 11th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2011.83 · Cited by 475 · see citing works
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · see citing works
Augmenting the algorithm: Emerging human-in-the-loop work configurations
Grønsund, T., & Aanestad, M. (2020) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2020.101614 · Cited by 364 · cited 2× · see citing works
“Most machine-human pipelines employed in real-world applications have multiple human experts available to share the load and to cover different kinds of input samples [12, 29].”
BackgroundON MULTI‐CLASS COST‐SENSITIVE LEARNING
Zhou, Z., & Liu, X. (2010) · Computational Intelligence · doi:10.1111/j.1467-8640.2010.00358.x · Cited by 356 · see citing works
Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing
Raji, I. D., Gebru, T., Mitchell, M., Buolamwini, J., Lee, J., & Denton, R. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375820 · Cited by 299 · see citing works
Yan, Y., Rosales, R., Fung, G., & Dy, J. (2011) · Cited by 289 · see citing works
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · see citing works
Automation Bias in Intelligent Time Critical Decision Support Systems
Cummings, M. (2004) · AIAA 1st Intelligent Systems Technical Conference · doi:10.2514/6.2004-6313 · Cited by 269 · see citing works
Community-based bayesian aggregation models for crowdsourcing
Venanzi, M., Guiver, J., Kazai, G., Kohli, P., & Shokouhi, M. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2567989 · Cited by 247 · see citing works
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244 · see citing works
Automation bias in intelligent time critical decision support systems
Cummings, M. L. (2017) · Decision Making in Aviation · doi:10.4324/9781315095080-17 · Cited by 233 · see citing works
Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287586 · Cited by 231 · see citing works
Self-citation(2018) · Cited by 230 · see citing works
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · see citing works
On the Foundations of Noise-free Selective Classification
El‐Yaniv, R., & Wiener, Y. (2010) · Journal of Machine Learning Research · Cited by 170 · see citing works
Cortes, C., DeSalvo, G., & Mohri, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46379-7_5 · Cited by 162 · see citing works
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · see citing works
Toward situated interventions for algorithmic equity: lessons from the field
Katell, M., Young, M., Dailey, D., Herman, B., Guetler, V., Tam, A., Bintz, C., Raz, D., & Krafft, P. M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372874 · Cited by 153 · see citing works
REVISITING A 90-YEAR-OLD DEBATE: THE ADVANTAGES OF THE MEAN DEVIATION
Gorard, S. A. C. (2005) · British Journal of Educational Studies · doi:10.1111/j.1467-8527.2005.00304.x · Cited by 147 · see citing works
On quality control and machine learning in crowdsourcing
Lease, M. (2011) · National Conference on Artificial Intelligence · Cited by 132 · see citing works
Self-citationKnows what it knows: a framework for self-aware learning
Li, L. C., Littman, M. L., Walsh, T. J., & Strehl, A. L. (2010) · Machine Learning · doi:10.1007/s10994-010-5225-4 · Cited by 127 · see citing works
Wilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118 · see citing works
Deep Reinforcement Active Learning for Human-in-the-Loop Person Re-Identification
Liu, Z., Wang, J., Gong, S., Tao, D., & Lu, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00622 · Cited by 115 · see citing works
Direct Uncertainty Prediction for Medical Second Opinions
Raghu, M., Blumer, K., Sayres, R., Obermeyer, Z., Kleinberg, R., Mullainathan, S., & Kleinberg, J. (2018) · arXiv · doi:10.48550/arxiv.1807.01771 · Cited by 93 · see citing works
Dwork, C., & Ilvento, C. (2019) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2019.33 · Cited by 91 · see citing works
Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer
Madras, D., Pitassi, T., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1711.06664 · Cited by 90 · see citing works
Combining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 74 · see citing works
Self-citationMinimax group fairness: algorithms and experiments
Diana, E., Gill, W., Kearns, M., Kenthapadi, K., & Roth, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462523 · Cited by 74 · see citing works
Self-citationIdentifying and Accounting for Task-Dependent Bias in Crowdsourcing
Kamar, E. S., Kapoor, A., & Horvitz, E. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13238 · Cited by 63 · see citing works
The Algorithmic Automation Problem: Prediction, Triage, and Human Effort
Raghu, M., Blumer, K., Corrado, G., Kleinberg, J., Obermeyer, Z., & Mullainathan, S. (2019) · arXiv · doi:10.48550/arxiv.1903.12220 · Cited by 58 · see citing works
Minimax Pareto Fairness: A Multi Objective Perspective
Martínez, N., Bertrán, M., & Sapiro, G. (2020) · PubMed · doi:10.48550/arxiv.2011.01821 · Cited by 54 · see citing works
Deep Gamblers: Learning to Abstain with Portfolio TheoryPreprint
Ziyin, L., Wang, Z., Liang, P. P., Salakhutdinov, R. R., Morency, L., & Ueda, M. (2019) · arXiv (Cornell University) · Cited by 49 · see citing works
Duwe, G., & Rocque, M. (2017) · Criminology & Public Policy · doi:10.1111/1745-9133.12270 · Cited by 48 · see citing works
López-García, P., Masegosa, A. D., Osaba, E., Onieva, E., & Perallos, A. (2019) · Applied Intelligence · doi:10.1007/s10489-019-01423-6 · Cited by 46 · see citing works
Regression under Human Assistance
De, A., Koley, P., Ganguly, N., & Gomez-Rodriguez, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5645 · Cited by 40 · see citing works
Cortes, C., DeSalvo, G., & Mohri, M. (2016) · neural information processing systems · Cited by 39 · see citing works
Qiu, C., Squicciarini, A., Carminati, B., Caverlee, J., & Khare, D. R. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983830 · Cited by 31 · see citing works
The Limits of Abstract Evaluation Metrics
Olteanu, A., Talamadupula, K., & Varshney, K. R. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3098871 · Cited by 30 · see citing works
Why Are People’s Decisions Sometimes Worse with Computer Support?
Alberdi, E., Strigini, L., Povyakalo, A. A., & Ayton, P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04468-7_3 · Cited by 29 · see citing works
Predicting Next Label Quality: A Time-Series Model of Crowdwork
Jung, H., Park, Y., & Lease, M. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13165 · Cited by 27 · see citing works
Self-citationA Dataset and Classifier for Recognizing Social Media English
Blodgett, S. L., Wei, J. T.-Z., & O'Connor, B. (2017) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w17-4408 · Cited by 26 · see citing works
Who Said What: Modeling Individual Labelers Improves Classification
Guan, M. Y., Gulshan, V., Dai, A. M., & Hinton, G. E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11756 · Cited by 25 · see citing works
Online Learning with AbstentionPreprint
Cortes, C., DeSalvo, G., Gentile, C., Mohri, M., & Yang, S. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03478 · Cited by 24 · see citing works
ORES: Lowering Barriers with Participatory Machine Learning in WikipediaPreprint
Halfaker, A., & Geiger, R. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05189 · Cited by 22 · see citing works
FrugalML: How to Use ML Prediction APIs More Accurately and CheaplyPreprint
Chen, L., Zaharia, M., & Zou, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07512 · Cited by 20 · see citing works
Crowd Access Path Optimization: Diversity Matters
Nushi, B., Singla, A. K., Gruenheid, A., Zamanian, E., Krause, A., & Kossmann, D. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13228 · Cited by 17 · see citing works
Cheaper and Better: Selecting Good Workers for Crowdsourcing
Li, H., & Liu, Q. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13248 · Cited by 17 · see citing works
Multi-label crowd consensus via joint matrix factorization
Tu, J., Yu, G., Domeniconi, C., Wang, J., Xiao, G., & Guo, M. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-019-01386-7 · Cited by 17 · see citing works
Multi-lingual Concept Extraction with Linked Data and Human-in-the-Loop
Alba, A., Coden, A. R., Gentile, A. L., Gruhl, D., Ristoski, P., & Welch, S. (2017) · International Conference on Knowledge Capture · doi:10.1145/3148011.3148021 · Cited by 16 · see citing works
Optimizing AI for Teamwork.Preprint
Bansal, G., Nushi, B., Kamar, E. S., Horvitz, E., & Weld, D. S. (2020) · arXiv (Cornell University) · Cited by 16 · see citing works
Crowdsourced Task Routing via Matrix FactorizationPreprint
Jung, H. J., & Lease, M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1310.5142 · Cited by 8 · see citing works
Self-citationDigitized Trust in Human-in-the-Loop Health Research
Sutton, A., Samavi, R., Doyle, T. E., & Koff, D. (2018) · Conference on Privacy, Security and Trust · doi:10.1109/pst.2018.8514168 · Cited by 7 · see citing works
Trust for the Doctor-in-the-Loop
Kieseberg, P., Weippl, E. R., & Holzinger, A. (2016) · TUGraz OPEN Library (Graz University of Technology) · Cited by 7 · see citing works
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Poursabzi-Sangdeh, F., samira-samadi, Vaughan, J. W., & Wallach, H. (2020) · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 75 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Amanda Bower, Sarah N Kitchen, Laura Niss, Martin J Strauss, Alexander Vargas, and Suresh Venkatasubramanian. Fair pipelines. arXiv preprint arXiv:1707.00391, 2017. · arXiv
- Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Convex Analysis and Minimization Algorithms I: Fundamentals, volume 305. Springer science & business media, 2013.
- Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pages 3150-3158, 2018.
- Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.
- Genevieve Patterson, Grant Van Horn, Serge Belongie, Pietro Perona, and James Hays. Bootstrapping fine-grained classifiers: Active learning with a crowd in the loop. In NeurIPS Workshop on Crowdsourcing: Theory, Algorithms and Applications, 2013.
- Chenxi Qiu, Anna C Squicciarini, Barbara Carminati, James Caverlee, and Dev Rishi Khare. Crowdselect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 539-548, 2016.
Cited by
Sorted by citations
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · see citing works
Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
Hemmer, P., Westphal, M., Schemmer, M., Vetter, S., Vössing, M., & Satzger, G. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584052 · Cited by 111 · see citing works
Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579601 · Cited by 77 · see citing works
Towards Human-AI Teaming to Mitigate Alert Fatigue in Security Operations Centres
Chhetri, M. B., Tariq, S., Singh, R., Jalalvand, F., Paris, C. L., & Nepal, . (2024) · ACM Transactions on Internet Technology · doi:10.1145/3670009 · Cited by 70 · see citing works
Sustainable energy management in the AI era: a comprehensive analysis of ML and DL approaches
Javed, H., Eid, F., El–Sappagh, S., & Abuhmed, T. (2025) · Computing · doi:10.1007/s00607-025-01485-0 · Cited by 44 · see citing works
Diversity in sociotechnical machine learning systems
Fazelpour, S., & De‐Arteaga, M. (2022) · Big Data & Society · doi:10.1177/20539517221082027 · Cited by 46 · see citing works
Role of Human-AI Interaction in Selective Prediction
Bondi, E., Koster, R., Sheahan, H., Chadwick, M. J., Bachrach, Y., Cemgil, A. T., Paquet, U., & Dvijotham, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20465 · Cited by 2 · see citing works
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36 · see citing works
Self-citationHuman-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness
Donahue, K., Chouldechova, A., & Kenthapadi, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533221 · Cited by 33 · see citing works
Self-citationRegulating Government AI and the Challenge of Sociotechnical Design
Engstrom, D. F., & Haim, A. (2023) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-120522-091626 · Cited by 35 · see citing works
A 2 C: A modular multi-stage collaborative decision framework for human–AI teams
Tariq, S., Chhetri, M. B., Nepal, ., & Paris, C. L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127318 · Cited by 29 · see citing works
SenseMate: An Accessible and Beginner-Friendly Human-AI Platform for Qualitative Data Analysis
Overney, C., Saldías-Fuentes, B., Dimitrakopoulou, D., & Roy, D. K. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645194 · Cited by 21 · see citing works
Black, E., Naidu, R. C. A., Ghani, R., Rodolfa, K. T., Ho, D. E., & Heidari, H. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623259 · Cited by 19 · see citing works
Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Information
Spitzer, P., Holstein, J., Hemmer, P., Vössing, M., Kühl, N., Martin, D., & Satzger, G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710999 · Cited by 18 · see citing works
Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J., & Satzger, G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/344 · Cited by 29 · see citing works
A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity
Rastogi, C., Leqi, L., Holstein, K., & Heidari, H. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27554 · Cited by 29 · see citing works
Towards Understanding AI Delegation: The Role of Self-Efficacy and Visual Processing Ability
Westphal, M., Hemmer, P., Vössing, M., Schemmer, M., Vetter, S., & Satzger, G. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3696423 · Cited by 10 · see citing works
Hettiachchi, D., Holcombe-James, I., Livingstone, S., de Silva, A., Lease, M., Salim, F. D., & Sanderson, M. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27546 · Cited by 11 · see citing works
Self-citationQiao, H., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714103 · Cited by 10 · see citing works
When More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21994 · Cited by 9 · see citing works
Learning to Defer with Limited Expert Predictions
Hemmer, P., Thede, L., Vössing, M., Jakubik, J., & Kühl, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25742 · Cited by 10 · see citing works
Algorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 128 · see citing works
Racial/Ethnic Categories in AI and Algorithmic Fairness: Why They Matter and What They Represent
Mickel, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659050 · Cited by 7 · see citing works
Learning to Complement and to Defer to Multiple Users
Zhang, Z., Ai, W., Wells, K., Rosewarne, D., Do, T.-T., & Carneiro, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72992-8_9 · Cited by 7 · see citing works
Diniz, P., Yokoe, T., Öttl, F., Pereira, H., Henriques, R., & Samuelsson, K. (2025) · Knee Surgery, Sports Traumatology, Arthroscopy · doi:10.1002/ksa.70222 · Cited by 4 · see citing works
Learning Paradigms for Hybrid Decision-Making Systems
Punzi, C., Pellungrini, R., Setzu, M., Giannotti, F., & Pedreschi, D. (2026) · ACM Computing Surveys · doi:10.1145/3802522 · Cited by 11 · see citing works
Banerjee, R., Jenamani, R. K., Vasudev, S., Nanavati, A., Dimitropoulou, K., Dean, S., & Bhattacharjee, T. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11127795 · Cited by 11 · see citing works
Principled Approaches for Learning to Defer with Multiple Experts
Mao, A., Mohri, M., & Zhong, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63735-3_7 · Cited by 4 · see citing works
Analytical Review of Task Allocation Methods for Human and AI Model Collaboration
Ponomarev, A., & Agafonov, A. (2025) · Informatics and Automation · doi:10.15622/ia.24.1.9 · Cited by 3 · see citing works
Scheduling And Sizing of Campus Microgrid Considering Demand Response And Economic AnalysisPreprint
Pansota, M. S., Javed, H., Muqeet, H. A., Irfan, M., Shehzad, M., & Liaqat, R. (2021) · Research Square · doi:10.21203/rs.3.rs-877951/v1 · Cited by 3 · see citing works
Who should I trust? Cautiously learning with unreliable experts
Love, T., Ajoodha, R., & Rosman, B. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07808-y · Cited by 2 · see citing works
Lowering Costs and Increasing Benefits Through the Ensemble of LLMs and Machine Learning Models
Liu, Z., Yang, Q., & Zou, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5666-7_31 · Cited by 2 · see citing works
Investigating the Relative Strengths of Humans and Machine Learning in Decision-Making
Rastogi, C. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604738 · Cited by 2 · see citing works
Human-in/on-the-Loop AI: Enabling Human Controllability and Decision-Making in Spaceflight
Bensch, L., Bensch, O., Nilsson, T., Paradiso, J. A., Davidoff, S., Blanchard, L., Pataranutaporn, P., Weber, M., Kutz, J. N., & Flemisch, F. O. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_40 · Cited by 1 · see citing works
Algorithmic Decision‐Making Safeguarded by Human Knowledge
Chen, N., Hu, M., & Li, W. (2026) · Naval Research Logistics (NRL) · doi:10.1002/nav.70052 · Cited by 1 · see citing works
MAOP: Multi-Agent Orchestration Platform for Robotic Applications
Doshi, D., Schuetz, D., Ebert, F., & Raatz, A. (2025) · International Conference on Robotics, Control and Automation Engineering (RCAE) · doi:10.1109/rcae66389.2025.11355188 · Cited by 1 · see citing works
A Confidence-Aware Human–Machine Collaboration Framework for Harmful Meme Detection
Lin, X.-Y., Tseng, Y.-Y., Huang, S.-C., & Wang, M.-H. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3649665 · Cited by 1 · see citing works
Trade risk identification and prediction using experts’ knowledge and machine learning
Dundulienė, I., Alzbutas, R., & Baltrušonė, K. (2021) · IEEE International Conference on Technology and Entrepreneurship (ICTE) · doi:10.1109/icte51655.2021.9584575 · Cited by 1 · see citing works
Designing Closed-Loop Models for Task Allocation
Keswani, V., Celis, L. E., Kenthapadi, K., & Lease, M. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230072
Self-citationA Complementarity-Enhanced Mixture of Human-AI Teams for Decision-Making
Liang, H., Liu, J., Guo, B., & Yu, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05981-9_17
Optimizing Resources for On-the-Fly Label Estimation with Multiple Unknown Medical Experts
Bary, T., Godelaine, T., Abels, A., & Macq, B. M. M. (2025) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi67747.2025.11269527
Is Uncertainty Quantification a Viable Alternative to Learned Deferral?
Wundram, A. M., & Baumgartner, C. F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06593-3_4
When Efficiency Shapes Fairness: Cost-Optimised Human-in-the-Loop Routing
Breene, V., O'Hagan, A., & Costello, F. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812278
A Taxonomy of Rational Handoffs for AI AgentsPreprint
Jauhari, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6986861
- Authors
- Keswani, V., Lease, M., & Kenthapadi, K.
- Affiliations
- Yale University (US)
- The University of Texas at Austin (US)
- Amazon (United States) (US)
- Venue
- Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2021-07-21
- Pages
- 154-165
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W3131911651
- Cited by
- 45
- References
- 75
- Citations by year
- 2026: 5 · 2025: 12 · 2024: 8 · 2023: 9 · 2022: 9 · 2021: 2
- FWCI
- 6.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic bias, content moderation, classifier confidence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
