CoChat
Sign Up
PreprintCited by 90Ethics and Social Impacts of AIMachine Learning and AlgorithmsExplainable Artificial Intelligence (XAI)

Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer

Open access
Madras, David, Pitassi, Toniann, Zemel, Richard·2017·arXiv
DOI 10.48550/arxiv.1711.06664arXiv 1711.06664
Open this paper in CoChat — free accountOpen PDF

Abstract

In many machine learning applications, there are multiple decision-makers involved, both automated and human. The interaction between these agents often goes unaddressed in algorithmic development. In this work, we explore a simple version of this interaction with a two-stage framework containing an automated model and an external decision-maker. The model can choose to say "Pass", and pass the decision downstream, as explored in rejection learning. We extend this concept by proposing "learning to defer", which generalizes rejection learning by considering the effect of other agents in the decision-making process. We propose a learning algorithm which accounts for potential biases held by external decision-makers in a system. Experiments demonstrate that learning to defer can make systems not only more accurate but also less biased. Even when working with inconsistent or biased users, we show that deferring models still greatly improve the accuracy and/or fairness of the entire system.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1711.06664

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The state-of-the-art approach to managing assignments in human-AI collaboration is learning to defer.

    “The state-of-the-art approach to managing assignments in HAIC is learning to defer (L2D)4–13, in which algorithms choose whether to assign a task to a human expert or an ML model, leveraging their complementary strengths.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [10]

    Section 2.1 (p. 3) describes learning to defer as a model choosing between its own prediction and a downstream decision-maker’s prediction based on which is expected to have lower loss. Section 2.4 (p. 5) illustrates how the model and a human doctor may have complementary strengths. The paper proposes and evaluates this approach, but does not establish the broader claim that it is the state-of-the-art approach to human–AI task assignment. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Learning to Defer algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.

    “Learning to Defer (L2D) algorithms improve human-AI collaboration by deferring decisions to human experts when they are likely to be more accurate than the AI model.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [10]

  • The paper chose to create a total of 50 experts because this far surpasses the numbers previously considered in learning-to-defer research.

    “We chose to create a total of 50 experts as this far surpasses the numbers previously considered in L2D research4–13.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [10]

  • The concern of fairness was explored by the original authors in their paper, where they consider adding a fairness regularization term to the loss function of their L2D system.

    “While this is a concern that goes unaddressed by most recent publications in this area, it was explored by the original authors in their paper10, where they consider adding a fairness regularization term to the loss function of their L2D system.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. [1][1] Josh Attenberg, Panagiotis G Ipeirotis, and Foster J Provost. Beat the machine: Challenging workers to find the unknown unknowns. Human Computation , 11(11), 2011. -
  2. [2][2] Yahav Bechavod and Katrina Ligett. Learning Fair Classifiers: A RegularizationInspired Approach. Workshop on Fairness, Accountability, and Transparency in Machine Learning , June 2017. arXiv: 1707.00044. Figure 5: Each point is a single run in sweep over γ reject /γ defer . X-axis is the model's lowest accuracy over 4 subgroups, defined by the cross product of binarized (sensitive attribute, unreliable attribute), which are (race, age) and (age, gender) for COMPAS and Health respectively. Y-axis is model accuracy. Only best Y-value for each X-value shown. Solid line is learning to defer; dashed line is rejection learning. <!-- image --> - · arXiv
  3. [3][3] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. URL http://proceedings.mlr.press/v37/blundell15.html . -
  4. [4][4] Amanda Bower, Sarah N. Kitchen, Laura Niss, Martin J. Strauss, Alexander Vargas, and Suresh Venkatasubramanian. Fair Pipelines. Workshop on Fairness, Accountability, and Transparency in Machine Learning , July 2017. arXiv: 1707.00391. - · arXiv
  5. [5][5] Jenna Burrell. How the machine 'thinks': Understanding opacity in machine learning algorithms. Big Data &amp; Society , 3(1):2053951715622512, 2016. -
  6. [6][6] Lowell W Busenitz and Jay B Barney. Differences between entrepreneurs and managers in large organizations: Biases and heuristics in strategic decision-making. Journal of business venturing , 12(1):9-30, 1997. -
  7. [7][7] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data , 5(2):153-163, 2017. -
  8. [8][8] C. Chow. An optimum character recognition system using decision function. IEEE T. C. , 1957. -
  9. [9][9] C. Chow. On optimum recognition error and reject trade-off. IEEE T. C. , 1970. -
  10. [10][10] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory , pages 67-82. Springer, 2016. -
  11. [11][11] Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences , 108(17):6889-6892, 2011. -
  12. [12][12] Robyn M Dawes, David Faust, and Paul E Meehl. Clinical versus actuarial judgment. Science , 243(4899):1668-1674, 1989. -
  13. [13][13] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226. ACM, 2012. -
  14. [14][14] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature , 542(7639):115-118, 2017. -
  15. [16][16] Nina Grgić-Hlaca, Muhammad Bilal Zafar, Krishna P. Gummadi, and Adrian Weller. On Fairness, Diversity and Randomness in Algorithmic Decision Making. Workshop on Fairness, Accountability, and Transparency in Machine Learning , June 2017. arXiv: 1706.10208. - · arXiv
  16. [17][17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http://proceedings.mlr.press/v70/guo17a.html . -
  17. [18][18] Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in neural information processing systems , pages 3909-3917, 2016. -
  18. [19][19] Kelly Hannah-Moffat. Actuarial sentencing: An 'unsettled' proposition. Justice Quarterly , 30(2):270-296, 2013. -
  19. [20][20] Moritz Hardt, Eric Price, Nati Srebro, et al. Equality of opportunity in supervised learning. In Advances in neural information processing systems , pages 3315-3323, 2016. -
  20. [21][21] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991. -
  21. [22][22] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbelsoftmax. International Conference on Learning Representations , 2016. -
  22. [23][23] Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. In Advances in Neural Information Processing Systems , pages 325-333, 2016. -
  23. [24][24] F. Kamiran and T. Calders. Classifying without discriminating. In 2nd International Conference on Computer, Control and Communication, 2009. IC4 2009 , pages 1-6, February 2009. doi: 10.1109/IC4.2009.4909197. - · DOI · Paper page
  24. [25][25] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. Machine Learning and Knowledge Discovery in Databases , pages 35-50, 2012. -
  25. [26][26] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2014. -
  26. [28][28] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent Trade-Offs in the Fair Determination of Risk Scores. Innovations in Theoretical Computer Science Conference , September 2016. arXiv: 1609.05807. - · arXiv
  27. [29][29] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. International Conference on Learning Representations , 2017. -
  28. [30][30] Aditya Krishna Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference on Fairness, Accountability and Transparency , pages 107118, 2018. -
  29. [31][31] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems , pages 5680-5689, 2017. -
  30. [32][32] Kush R Varshney and Homa Alemzadeh. On the safety of machine learning: Cyberphysical systems, decision sciences, and data products. Big data , 5(3):246-255, 2017. -
  31. [33][33] Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, and Joseph E. Gonzalez. IDK Cascades: Fast Deep Learning by Learning not to Overthink. Conference on Uncertainty in Artificial Intelligence , June 2017. arXiv: 1706.00885. - · arXiv
  32. [34][34] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment &amp; disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference on World Wide Web , pages 1171-1180. International World Wide Web Conferences Steering Committee, 2017. -
  33. [35][35] Richard Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning Fair Representations. In PMLR , pages 325-333, February 2013. URL http://proceedings. mlr.press/v28/zemel13.html .

Cited by

Sorted by citations

Authors
Madras, D., Pitassi, T., & Zemel, R.
Affiliations
University of Toronto (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
31
Pages
6147-6157
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1711.06664
arXiv
1711.06664
OpenAlex
W2891947597
Cited by
90
References
0
Citations by year
2025: 12 · 2024: 13 · 2023: 17 · 2022: 13 · 2021: 18 · 2020: 11 · 2019: 3 · 2018: 2 · 2013: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
learning to defer, fairness, algorithmic bias, human-in-the-loop decision making, two-stage decision making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review