Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record, with a small difference in title or year; 59 of the 59 references it cites were found in a registry or an index; 12 of the 60 sources its claims cite were checked for support.

Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree

Open access
Weerts, Hilde, Xenidis, Raphaële, Tarissan, Fabien, Olsen, Henrik Palmer, Pechenizkiy, Mykola · 2023 · 2023 ACM Conference on Fairness Accountability and Transparency

Abstract

Concerns regarding unfairness and discrimination in the context of artificial intelligence (AI) systems have recently received increased attention from both legal and computer science scholars. Yet, the degree of overlap between notions of algorithmic bias and fairness on the one hand, and legal notions of discrimination and equality on the other, is often unclear, leading to misunderstandings between computer science and law. What types of bias and unfairness does the law address when it prohibits discrimination? What role can fairness metrics play in establishing legal compliance? In this paper, we aim to illustrate to what extent European Union (EU) non-discrimination law coincides with notions of algorithmic fairness proposed in computer science literature and where they differ. The contributions of this paper are as follows. First, we analyse seminal examples of algorithmic unfairness through the lens of EU non-discrimination law, drawing parallels with EU case law. Second, we set out the normative underpinnings of fairness metrics and technical interventions and compare these to the legal reasoning of the Court of Justice of the EU. Specifically, we show how normative assumptions often remain implicit in both disciplinary approaches and explain the ensuing limitations of current AI practice and non-discrimination law. We conclude with implications for AI practitioners and regulators.

Overview

CoChat Summary

TL;DR

The paper argues that EU non-discrimination law and algorithmic fairness overlap but are not equivalent, because legal discrimination analysis is contextual and normative rather than reducible to a fairness metric or threshold. It analyses three algorithmic unfairness examples and shows that fairness metrics and fair-ML techniques embed implicit normative choices that cannot alone establish legal compliance.

Key findings

  • EU non-discrimination law distinguishes direct and indirect discrimination: direct discrimination generally cannot be justified except for limited derogations, while indirect discrimination can be objectively justified if it pursues a legitimate aim and is proportionate. The boundary is contested; the CJEU has not always consistently distinguished the two, e.g. treating pregnancy as directly linked to sex in *Dekker* but not treating country of birth as inseparably linked to ethnic origin in *Jyske Finans*. In *Seymour-Smith*, statistics showing 77.4% of men and 68.9% of women could meet a two-year employment requirement did not, on the face of it, show a considerably smaller percentage of women.
  • In the Dutch childcare benefits scandal, the Dutch DPA found that using nationality in a risk classification model was discriminatory under, among others, Art. 21 of the Charter and the GDPR fairness principle; it analysed the case through indirect discrimination because nationality alone was insufficient and a less discriminatory alternative existed. Parents were wrongly accused over the course of a decade, and redacted files made evidence difficult.
  • Amazon’s recruitment algorithm penalised gender-associated words; this could be proxy discrimination under indirect discrimination or direct discrimination if the criteria are “inextricably linked” with sex or gender. If framed as indirect discrimination, accuracy on a test set may be accepted as justification, but accuracy on observed data may be inadequate under measurement bias and may not generalise due to out-of-sample predictions or concept drift.
  • *Gender Shades* found commercial facial recognition systems failed disproportionately for darker-skinned women compared with lighter-skinned men; such cases may fall under indirect discrimination if within the material scope, with the burden of proof shifting, as in a Dutch student complaint. EU law has not explicitly recognised intersectional discrimination (*Parris*). Fairness metrics are incompatible when base rates differ: demographic parity, equalised odds, and equal calibration cannot be satisfied simultaneously, and individual fairness is typically at odds with demographic parity when all input features are used in the similarity metric. The law is not a decision tree; compliance cannot translate into a single threshold or fairness metric.

Methods

  • Conceptual and doctrinal legal analysis with cross-disciplinary comparison; no empirical experiment is reported. The paper analyses three seminal algorithmic unfairness examples—the Dutch childcare benefits scandal, Amazon’s recruitment algorithm, and *Gender Shades*—through the lens of EU non-discrimination law, drawing parallels with CJEU case law.
  • Sources are EU non-discrimination law (Treaties, Charter, directives, and CJEU case law) and algorithmic fairness literature on fairness metrics and fairness-aware machine learning techniques. The paper compares the normative underpinnings of those metrics and interventions with the legal reasoning of the Court of Justice of the EU.
  • Main analysis identifies elements of a discrimination case—decision “on grounds of” a protected characteristic, personal and material scope, evidence of less favourable treatment or particular disadvantage, and objective justification—and applies them by analogy to algorithmic cases, while examining implicit normative assumptions in fairness metrics and fair-ML.

Limitations

  • Stated: The paper says future work is necessary to uncover the moral implications of design choices in machine learning development; legal scholars often have an idealised view of what fair-ML techniques can achieve; applicants face difficulty providing prima facie evidence in opaque algorithmic systems; the CJEU rarely relies on statistical evidence and has not explicitly recognised intersectional discrimination.
  • Stated: The analysis focuses on three seminal examples and EU non-discrimination law; case law in some areas, such as disparate product safety and performance in goods and services, is scarce and does not provide immediate analogies.
  • Evident from the text: It is a conceptual and legal analysis without empirical validation or quantitative measurement of legal compliance; the examples are illustrative rather than a systematic sample, and the paper does not provide a decision procedure or threshold for compliance.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3593013.3594044

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 11 Not valid · 35 Not checked · 70 Own results and interpretations

50 more claims

Create a free account to see all 127 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 27
  • Probably not a substantial claim. 56
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 58 references

Research works printed but not linked: 37

  1. Case 236/09. 2018. Test-Achats. EU:C:2011:100.
  2. Case 406/15. 2017. Petya Milkova v Izpalnitelen direktor na Agentsiata za privatizatsia i sledprivatizatsionen kontrol. EU:C:2017:198.
  3. 10.1093/indlaw · doi:10.1093/indlaw
  4. Agathe Balayn and Seda Gürses. 2021. Beyond Debiasing: Regulating AI and its inequalities. EDRi Report. https://edri. org/wp-content/uploads/2021/09/EDRi_Beyond-Debiasing-Report_Online. pdf (2021).
  5. Eugenia Caracciolo di Torella. 2022. Directive 2004/113/EC on Gender Equality in Goods and Services – In search of the potential of a forgotten Directive. Publications Office of the European Union, Luxembourg.
  6. Case 109/88. 1989. Handels- og Kontorfunktionærernes Forbund I Danmark v Dansk Arbejdsgiverforening, acting on behalf of Danfoss. EU:C:1989:383.
  7. Case 144/04. 2005. Werner Mangold v Rüdiger Helm. EU:C:2005:709.
  8. Case 152/11. 2012. Johann Odar v Baxter Deutschland GmbH. EU:C:2012:772.
  9. Case 157/15. 2021. Samira Achbita and Centrum voor gelijkheid van kansen en voor racismebestrijding v G4S Secure Solutions NV. EU:C:2017:203.
  10. Case 167/97. 1999. Regina v Secretary of State for Employment, ex parte Nicole Seymour-Smith and Laura Perez. EU:C:1999:60.
  11. Case 170/84. 1986. Bilka - Kaufhaus GmbH v Karin Weber von Hartz. EU:C:1986:204.
  12. Case 177/88. 1990. Elisabeth Johanna Pacifica Dekker v Stichting Vormingscentrum voor Jong Volwassenen (VJV-Centrum) Plus. EU:C:1990:383.
  13. Case 193/17. 2019. Cresco Investigation. EU:C:2019:43.
  14. Case 223/19. 2020. YS v. NK. EU:C:2020:753.
  15. Case 262/88. 1990. Douglas Harvey Barber v Guardian Royal Exchange Assurance Group. EU:C:1990:209.
  16. Case 26/62. 1963. Van Gend en Loos. EU:C:1963:1.
  17. Case 312/17. 2018. Surjit Singh Bedi v Bundesrepublik Deutschland and Bundesrepublik Deutschland in Prozessstandschaft für das Vereinigte Königreich von Großbritannien und Nordirland. EU:C:2018:734.
  18. Case 409/16. 2017. Ypourgos Esoterikon and Ypourgos Ethnikis paideias kai Thriskevmaton v Maria-Eleni Kalliri. EU:C:2017:767.
  19. Case 409/95. 1997. Hellmut Marschall v Land Nordrhein-Westfalen. EU:C:1997:533.
  20. Case 443/15. 2016. David L. Parris v Trinity College Dublin and Others. EU:C:2016:897.
  21. Case 450/93. 1995. Eckhard Kalanke v Freie Hansestadt Bremen. EU:C:1995:322.
  22. Case 457/17. 2018. Heiko Jonny Maniero v Studienstiftung des deutschen Volkes eV. EU:C:2018:912.
  23. Case 457/17. 2018. Opinion of Advocate General Sharpston in Heiko Jonny Maniero v Studienstiftung des deutschen Volkes eV. EU:C:2018:697.
  24. Case 668/15. 2017. Jyske Finans A/S v Ligebehandlingsnævnet, acting on behalf of Ismar Huskic. EU:C:2017:278.
  25. Case 68/17. 2018. IR v. JQ. EU:C:2018:696.
  26. Council of European Union. 1975. Council Directive 75/117/EEC of 10 February 1975 on the approximation of the laws of the Member States relating to the application of the principle of equal pay for men and women. Official Journal L 45 (1975), 19–20.
  27. Council of European Union. 2000. Council Directive 2000/78/EC of 27 November 2000 establishing a general framework for equal treatment in employment and occupation. Official Journal L 303 (2000), 16–22.
  28. Council of European Union. 2000. Racial Equality Directive. Council Directive 2000/43/EC of 29 June 2000 implementing the principle of equal treatment between persons irrespective of racial or ethnic origin. Official Journal L 180 (2000), 22–26.
  29. Council of European Union. 2006. Directive 2006/54/EC of the European Parliament and of the Council of 5 July 2006 on the implementation of the principle of equal opportunities and equal treatment of men and women in matters of employment and occupation. Official Journal L 204 (2006), 23––36.
  30. Council of European Union. 2012. Treaty on European Union. Official Journal C 326 (2012), 13–390.
  31. Joined Cases C-569/16 and C-570/16. 2018. Bauer. EU:C:2018:871.
  32. Joined Cases C-804/18 and C-341/19. 2021. IX v WABE eV and MH Müller Handels GmbH v MJ. EU:C:2021:594.
  33. Pauline T Kim. 2022. Race-aware algorithms: Fairness, nondiscrimination and affirmative action. Cal. L. Rev. 110 (2022), 1539.
  34. Pablo Martínez-Ramil. 2022. Discriminatory algorithms. A proportionate means of achieving a legitimate aim?Journal of Ethics and Legal Technologies 4, 1 (2022).
  35. NOS Nieuws. 2019. Ouders zwartgelakte dossiers: ’Ik weet nog steeds niet wat ik fout heb gedaan’. NOS Nieuws (2019). https://nos.nl/artikel/2314288-ouders-zwartgelakte-dossiers-ik-weet-nog-steeds-niet-wat-ik-fout-heb-gedaan
  36. Elanor Sharpston. 2022. Shadow Opinion of former Advocate-General Sharpston: headscarves at work (Cases C-804/18 and C-341/19). Last visited: the 7th of July (2022).
  37. Raphaële Xenidis. 2018. Multiple discrimination in EU anti-discrimination law: towards redressing complex inequality? In EU anti-discrimination law beyond gender, Uladzislau Belavusau and Kristin Henrard (Eds.). Oxford University Press, Oxford, Chapter 2, 41–74.

Cited by

See all 34

Metadata

Authors
Weerts, H., Xenidis, R., Tarissan, F., Olsen, H. P., & Pechenizkiy, M.
Affiliations
  • Eindhoven University of Technology (NL)
  • Centre National de la Recherche Scientifique (FR)
  • University of Copenhagen (DK)
Venue
2023 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2023-06-12
Pages
805-816
OpenAlex
W4378446601
Cited by
34
References
58
Citations by year
2026: 9 · 2025: 10 · 2024: 12 · 2023: 3
FWCI
8.52
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
  • TU/e Research Portal (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
  • HAL (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • European Commission (898937, HORIZON2020)
  • Agence Nationale de la Recherche (ANR-22-EXES-0014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Discrimination and Equality Law
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
algorithmic discrimination, EU anti-discrimination law, algorithmic fairness, fairness metrics, Court of Justice of the European Union, EU case law, AI regulation, legal compliance, normative assumptions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree. https://app.cochat.ai/papers/10.1145/3593013.3594044
The paper
Weerts, H., Xenidis, R., Tarissan, F., Olsen, H. P., & Pechenizkiy, M. (2023). Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree. 2023 ACM Conference on Fairness Accountability and Transparency. https://doi.org/10.1145/3593013.3594044

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.