Formalising trade-offs beyond algorithmic fairness: lessons from ethical philosophy and welfare economics
Open accessAbstract
Abstract There is growing concern that decision-making informed by machine learning (ML) algorithms may unfairly discriminate based on personal demographic attributes, such as race and gender. Scholars have responded by introducing numerous mathematical definitions of fairness to test the algorithm, many of which are in conflict with one another. However, these reductionist representations of fairness often bear little resemblance to real-life fairness considerations, which in practice are highly contextual. Moreover, fairness metrics tend to be implemented within narrow and targeted fairness toolkits for algorithm assessments that are difficult to integrate into an algorithm’s broader ethical assessment. In this paper, we derive lessons from ethical philosophy and welfare economics as they relate to the contextual factors relevant for fairness. In particular we highlight the debate around the acceptability of particular inequalities and the inextricable links between fairness, welfare and autonomy. We propose Key Ethics Indicators (KEIs) as a way towards providing a more holistic understanding of whether or not an algorithm is aligned to the decision-maker’s ethical values.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These toolkits have been criticised for offering a 'reductionist understanding of fairness as mathematical conditions'.
“These toolkits have been criticised for offering a 'reductionist understanding of fairness as mathematical conditions' [71, p. 1], and reflect a longer history of attempts to reduce (un)fairness to a metric [72].”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [71]
Many of the collected documents are drafted by public, private or not-for-profit organisations and constitute 'grey literature' not typically found in academic databases.
“Many of these documents are drafted by public, private or not-for-profit organisations and constitute 'grey literature' not typically found in academic databases [71].”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [71]
Fairness is often contextual: a seemingly 'fair' model designed for one context might be misleading, inaccurate, or harmful when applied to a different context.
“However, as existing literature makes clear, fairness is often contextual [57,90,91,131,136]: a seemingly”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [90]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Black Feminist Thought: Knowledge, Consciousness, and the Politics of Empowerment.
Brewer, R. M., & Collins, P. H. (1992) · Contemporary Sociology · doi:10.2307/2074808 · Cited by 19,445 · see citing works
Crenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510 · see citing works
Rawls, J. (1971) · Princeton University Press eBooks · doi:10.2307/j.ctv19fvzzk.68 · Cited by 9,464 · see citing works
Johnson, K., & Nozick, R. (1975) · The Western Political Quarterly · doi:10.2307/447864 · Cited by 7,692 · see citing works
Nozick, R. (2018) · Princeton University Press eBooks · doi:10.2307/j.ctv19fvzzk.69 · Cited by 6,689 · see citing works
Rawls, J. (1958) · The Philosophical Review · doi:10.2307/2182612 · Cited by 5,583 · see citing works
Justice as Fairness: A Restatement
Metz, T., Rawls, J., & Kelly, E. I. (2002) · The Philosophical Review · doi:10.2307/1556442 · Cited by 3,586 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Broome, J., & Sen, A. K. (1993) · The Economic Journal · doi:10.2307/2234727 · Cited by 3,250 · see citing works
Spheres of Justice: A Defense of Pluralism and Equality.
Daniels, N., & Walzer, M. (1985) · The Philosophical Review · doi:10.2307/2184729 · Cited by 2,667 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
A Taxonomy of Organizational Justice Theories
Greenberg, J. (1987) · Academy of Management Review · doi:10.5465/amr.1987.4306437 · Cited by 2,309 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Kant: The Metaphysics of Morals
Kant, I., & Sullivan, R. J. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809644 · Cited by 1,942 · see citing works
Gauthier, D. (1987) · doi:10.1093/0198249926.001.0001 · Cited by 1,933 · see citing works
Lewis, D. (1973) · The Journal of Philosophy · doi:10.2307/2025310 · Cited by 1,921 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
A Unified Framework of Five Principles for AI in Society
Floridi, L., & Cowls, J. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.8cd550d1 · Cited by 1,631 · cited 5× · see citing works
“It complements the other four principles by helping us understand the good or harm an algorithmic system is actually doing to society, in which ways, and why [20].”
InfluentialSelf-citationKant, I., & Wood, A. W. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813306.013 · Cited by 1,567 · see citing works
Hill Collins, P. (2002) · Routledge · doi:10.4324/9780203900055 · Cited by 1,532 · see citing works
What is Equality? Part 1: Equality of Welfare
Dworkin, R. M. (2018) · Routledge eBooks · doi:10.4324/9781315199795-6 · Cited by 1,183 · see citing works
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · see citing works
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges
Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., & Zhu, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32236-6_51 · Cited by 758 · see citing works
Fairness, Responsibility, and Welfare
Fleurbaey, M. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:osobl/9780199215911.001.0001 · Cited by 695 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · see citing works
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444 · see citing works
Scheffler, S. (1984) · eYLS (Yale Law School) · doi:10.1093/0198235119.001.0001 · Cited by 357 · see citing works
On the apparent conflict between individual and group fairness
Binns, R. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372864 · Cited by 312 · see citing works
50 Years of Test (Un)fairness: Lessons for Machine Learning
Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287600 · Cited by 297 · see citing works
Delayed Impact of Fair Machine Learning
Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., & Hardt, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/862 · Cited by 236 · see citing works
A Framework for Understanding Unintended Consequences of Machine Learning
Suresh, H., & Guttag, J. V. (2019) · Cited by 229 · see citing works
Parfit, D. (1991) · KU ScholarWorks (University of Kansas) · Cited by 215 · see citing works
Kagan, S. (2012) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199895595.001.0001 · Cited by 179 · see citing works
Cole, N. S. (1973) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1973.tb00802.x · Cited by 168 · see citing works
Wachter, S., Mittelstadt, B., & Russell, C. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3792772 · Cited by 167 · see citing works
Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150 · see citing works
Self-citationA Moral Framework for Understanding Fair ML through Economic Models of Equality of Opportunity
Heidari, H., Loi, M., Gummadi, K. P., & Krause, A. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287584 · Cited by 141 · see citing works
“Justice” and “fairness” are not the same thing
Goldman, B. M., & Cropanzano, R. (2014) · Journal of Organizational Behavior · doi:10.1002/job.1956 · Cited by 134 · see citing works
50 Years of Test (Un)fairness: Lessons for Machine LearningPreprint
Hutchinson, B., & Mitchell, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10104 · Cited by 118 · see citing works
Algorithmic Fairness in Mortgage Lending: from Absolute Conditions to Relational Trade-offs
Lee, M. S. A., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-020-09529-4 · Cited by 106 · see citing works
Self-citationFairness Through Computationally-Bounded Awareness
Kim, M. P., Reingold, O., & Rothblum, G. N. (2018) · arXiv · doi:10.48550/arxiv.1803.03239 · Cited by 97 · see citing works
Long, H. S., , A., & Sinclair, A. (1963) · The Classical World · doi:10.2307/4345110 · Cited by 92 · see citing works
, A. (1962) · Oxford University Press eBooks · doi:10.1093/oseo/instance.00259307 · Cited by 81 · see citing works
Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AIPreprint
Wachter, S., Mittelstadt, B., & Russell, C. (2020) · doi:10.31228/osf.io/jq2b6 · Cited by 65 · see citing works
Determinants of the Locations of Payday Lenders, Pawnshops and Check-Cashing Outlets
Prager, R. A. (2009) · Finance and Economics Discussion Series · doi:10.17016/feds.2009.33 · Cited by 54 · see citing works
Lee, M. S. A., Floridi, L., & Denev, A. (2021) · Philosophical studies series · doi:10.1007/978-3-030-81907-1_20 · Cited by 38 · see citing works
Self-citationEthics and the business of bioscience
Eaton, M. L. (2004) · Choice Reviews Online · doi:10.5860/choice.42-1594 · Cited by 36 · see citing works
Bankruptcy: Is It Enough to Forgive or Must We Also Forget?
Elul, R., & Gottardi, P. (2015) · American Economic Journal Microeconomics · doi:10.1257/mic.20130139 · Cited by 36 · see citing works
The Sensitivity of Counterfactual Fairness to Unmeasured ConfoundingPreprint
Kilbertus, N., Ball, P., Kusner, M. J., Weller, A., & Silva, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01040 · Cited by 29 · see citing works
Ethics and the Business of Bioscience
Eaton, M. L. (2004) · Stanford University Press eBooks · doi:10.1515/9781503619609 · Cited by 29 · see citing works
Explainable and Non-explainable Discrimination in Classification
Kamiran, F., & Žliobaitė, I. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-30487-3_8 · Cited by 27 · see citing works
The Input FallacyPreprint
Gillis, T. B. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3571266 · Cited by 20 · see citing works
The Norms of Algorithmic Credit ScoringPreprint
Aggarwal, N. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3569083 · Cited by 19 · see citing works
The Landscape and Gaps in Open Source Fairness ToolkitsPreprint
Lee, M. S. A., & Singh, J. P. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3695002 · Cited by 10 · see citing works
Self-citationRisk Identification Questionnaire for Unintended Bias in Machine Learning Development LifecyclePreprint
Lee, M. S. A., & Singh, J. P. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3777093 · Cited by 6 · see citing works
Self-citationAequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577
Flagged first, then by citations
Research works printed but not linked: 10
- Aggarwal, N.: Law and autonomous systems series: algorithmic credit scoring and the regulation of consumer credit markets. Uni- versity of Oxford Business Law Blog, Oxford (2018)
- Arya, V., Bellamy, R.K.E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S.C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A, et al.: One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv: 1909. 03012 (2019)
- Bellamy, R.K.E., Dey, K., Hind, M., Hoffman, S.C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A. et al.: AI fairness 360: An extensible toolkit for detecting, understand- ing, and mitigating unwanted algorithmic bias. arXiv preprint arXiv: 1810. 01943 (2018)
- Donovan, K.P., Park, E.: Perpetual bebt in dilicon Savannah. Bos- ton Review, Boston (2019)
- Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fair- ness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference. ACM, pp 214-226 (2012)
- Friedler, S.A., Scheidegger, C., Venkatasubramanian, S.: On the (im) possibility of fairness. arXiv preprint arXiv: 1609. 07236 (2016)
- Gajane, P., Pechenizkiy, M.: On formalizing fairness in prediction with machine learning. arXiv preprint arXiv: 1710. 03184 (2017)
- Gillis, T.B.: False dreams of algorithmic fairness: the case of credit pricing. Available at SSRN 3571266 (2020)
- Kleinberg, J., Mullainathan, S., Raghavan, M.: Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv: 1609. 05807 (2016)
- Lewis, D.: Causation. J. Philos. 70(17), 556-567 (1973)
Cited by
Sorted by citations
Putting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253 · see citing works
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157 · see citing works
“Implement both mathematical and ethically grounded fairness definitions [67, 129] 17.”
BackgroundInfluentialExploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · see citing works
Self-citationA translational perspective towards clinical AI fairness
Liu, M., Ning, Y., Teixayavong, S., Mertens, M., Xu, J., Ting, D. S. W., Cheng, L. T.-E., Ong, J. C. L., Teo, Z. L., Tan, T. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00918-4 · Cited by 82 · see citing works
Lewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30 · see citing works
Self-citation
- Authors
- Lee, M. S. A., Floridi, L., & Singh, J.
- Affiliations
- University of Cambridge (GB)
- University of Oxford (GB)
- Corresponding author
- Lee, Michelle Seng Ah
- Venue
- AI and Ethics
- Published
- 2021-06-12
- Volume
- 1
- Issue
- 4
- Pages
- 529-544
- ISSN
- 2730-5953 (ISSN-L)
- 2730-5961
- OpenAlex
- W3166308075
- Cited by
- 71
- References
- 58
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 16 · 2023: 18 · 2022: 14 · 2021: 5
- FWCI
- 16.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- AI and Ethics (publishedVersion, cc-by)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo
- Funders
- Engineering and Physical Sciences Research Council (EP/P024394/1)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Political Science and International Relations › Political Philosophy and Ethics
- Keywords
- algorithmic fairness, ethical philosophy, welfare economics, autonomy, fairness metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
