CoChat

Protected Grounds and the System of Non-Discrimination Law in the Context of Algorithmic Decision-Making and Artificial Intelligence

Open access
Gerards, Janneke, Zuiderveen Borgesius, Frederik · 2021 · SSRN Electronic Journal

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

Abstract

Algorithmic decision-making and similar types of artificial intelligence (AI) may lead to improvements in all sectors of society, but can also have discriminatory effects. While current non-discrimination law offers people some protection, algorithmic decision-making presents the law with several challenges. For instance, algorithms can generate new categories of people based on seemingly innocuous characteristics, such as web browser preference or apartment number, or more complicated categories combining many data points. Such new types of differentiation could evade non-discrimination law, as browser type and house number are not protected characteristics, but such differentiation could still be unfair, for instance if it reinforces social inequality. This paper explores which system of non-discrimination law can best be applied to algorithmic decision-making, considering that algorithms can differentiate on the basis of characteristics that do not correlate with protected grounds of discrimination such as ethnicity or gender. The paper analyses the current loopholes in the protection offered by non-discrimination law and explores the best way for lawmakers to approach algorithmic differentiation. While we focus on Europe, the conceptual and theoretical focus of the paper can make it useful for scholars and policymakers from other regions too, as they encounter similar problems with algorithmic decision-making.

Overview

Integrity signals

Registry

Crossref · 10.2139/ssrn.3723873

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 34 references

Research works printed but not linked: 95

  1. Latanya Sweeney, 2013, Discrimination in OnlineAd Delivery, CoMM. ACM
  2. Latanya Sweeney, 2013, Discriminationin Online Ad Delivery, CoMM. ACM
  3. Latanya Sweeney, 2013, Discriminationin Online Ad Delivery, CoMM. ACM
  4. Latanya Sweeney, 2013, Discrimination in OnlineAd Delivery, CoMM. ACM
  5. Jennifer Valentino-Devries, 2012, Trust but Verify: A Guide to Algorithms and the Law, 31 HARV, J. L. & TECH
  6. We leave aside, for now, the question whether the bank's behaviour is socially desirable. We merely want to illustrate that algorithmic price differentiation can reinforce financial inequality
  7. David Caplovitz, 1963, CONSUMER PRACTICES OF LOW-INCOME FAMILIES
  8. Virginia Eubanks, 2016, See also Tal Zarsky, The Trouble with Algorithmic Decisions: An Analytic Road Map to Examine Efficiency and Fairness in Automated and Opaque Decision Making, 41 SCI, TECH. & HUM. VALUES, 118, 124
  9. 2013, The ECtHR's caselaw on the topic is not consistent, however; see Janneke H. Gerards, The Discrimination Grounds of Article 14 of the European Convention on Human Rights, 13 HUM, RTS. L. REV
  10. See Also Aalt Willem, 1999, Standards of Review for Discrimination: The Scope of Review by the Courts, NON-DISCRIMINATION LAW: COMPARATIVE PERSPECTIVES, 25
  11. As discussed supra
  12. Christa See Generally, A CASE STUDY INTO THE DEVELOPMENT OF THE LEGAL CONCEPT OF INDIRECT DISCRIMINATION UNDER EC LAW
  13. 2005, SOCIAL EUROPE SERIES
  14. Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 129
  15. Elisabeth Holzleithner, 2004, Dis/Entangling Grounds of Discrimination, 927
  16. 2016, Amending or Retaining Article 1 of the Netherlands Constitution: In Case of Doubt, Do Not Pass, Titia Loenen, Wijzigen of handhaven artikel 1 Grondwet: Bij twijfel niet inhalen
  17. Titia Loenen, 2016, Amending or Retaining Article 1 of the Netherlands Constitution: In Case of Doubt, Do Not Pass, Wijzigen of handhaven artikel 1 Grondwet: Bij twijfel niet inhalen
  18. Titia Loenen, 2016, Amending or Retaining Article 1 of the Netherlands Constitution: In Case of Doubt, Do Not Pass, Wijzigen of handhaven artikel 1 Grondwet: Bij twijfel niet inhalen
  19. Titia Loenen, 2016, Amending or Retaining Article 1 of the Netherlands Constitution: In Case of Doubt, Do Not Pass, Wijzigen of handhaven artikel 1 Grondwet: Bij twijfel niet inhalen
  20. On the importance of this symbolic value for protecting against discrimination
  21. See Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 135
  22. H Janneke, 2016, Artikel 1 Grondwet -goede gronden voor wijziging? [Article 1 of the Netherlands Constitution -Good Grounds for Change?
  23. 2012, Jakob Cornides, Three Case Studies on 'Anti-Discrimination', 23 EUR, J. INT'L L, 517
  24. 1969, For an example of easy acceptance of 'biological' inequalities between men and women, see e, THE CONCEPT OF EQUALITY, 54, 64
  25. 2012, In more detail, see Jakob Cornides, Three Case Studies on 'Anti-Discrimination', 23 EUR
  26. 2005, there is a debate on whether appearance or weight should perhaps be considered protected grounds of discrimination; see e.g. Deborah Carr & Michael A. Friedman, Is Obesity Stigmatizing? Body Weight, Perceived Discrimination, and Psychological Well-Being in the United States, J. HEALTH AND SOC. BEHAV, 46
  27. On 'suspect' grounds, see infra
  28. Comparing the semi-closed lists of grounds of the two provisions shows that 'ethnic origin' (next to 'race' and 'national origin'), 'belief' (next to 'religion'), 'sexual orientation', 'disability', 'age' and 'genetic features' have been added
  29. 2020, Andrew Altman, Discrimination, THE STANFORD ENCYCLOPEDIA OF PHILOSOPHY � 7
  30. Andrew Altman, 2020, Discrimination, THE STANFORD ENCYCLOPEDIA OF PHILOSOPHY
  31. 2007, This is even more true in the EU Non-Discrimination Directives, many of which are limited to a number of particular grounds, such as either gender or race/ethnicity; see e.g. Janneke H. Gerards, Discrimination Grounds, CASES, MATERIALS AND TEXT ON NATIONAL, SUPRANATIONAL AND INTERNATIONAL NON-DISCRIMINATION LAW, 33
  32. 2007, This is clearly true for the EU Non-Discrimination Directives with their closed system of exemptions, but problematic examples also can be found in national law; see Janneke H. Gerards, Discrimination Grounds, CASES, MATERIALS AND TEXT ON NATIONAL, SUPRANATIONAL AND INTERNATIONAL NON-DISCRIMINATION LAW, 33
  33. 2008, Approaching the Intersection of Sex, Religion and Race under the European Convention on Human Rights and EC Equality Law, in EUROPEAN UNION NON-DISCRIMINATION LAW, Titia Loenen, 313
  34. 2018, EUROPEAN FUNDAMENTAL RIGHTS AGENCY & EUROPEAN COURT OF HUMAN RIGHTS
  35. Timo Makkonen, 2002, BRINGING THE EXPERIENCES OF THE MOST MARGINALIZED TO THE FORE, 36
  36. H Janneke, 2007, Discrimination Grounds, CASES, MATERIALS AND TEXT ON NATIONAL, SUPRANATIONAL AND INTERNATIONAL NON-DISCRIMINATION LAW, 33
  37. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 141
  38. 2017, For further criticism of the indirect discrimination approach taken in the Achbita v. G4S Secure Solutions NV case, see e.g. Eva Brems, European Court of Justice Allows Bans on Religious Dress in the Workplace, Springer 2020), 135
  39. Eleanor Spaventa, 2017, The approach was received more favorably by Mark Bell, Leaving Religion at the Door? The European Court of Justice and Religious Symbols in the Workplace, What is the Point of Minimum Harmonization of Fundamental Rights? Some Further Reflections on the Achbita Case
  40. See Also Aalt Willem, 1999, Standards of Review for Discrimination: The Scope of Review by the Courts, NON-DISCRIMINATION LAW: COMPARATIVE PERSPECTIVES, 25
  41. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 138
  42. 1998, See e.g. Janneke H. Gerards, Nieuwe ronde, nieuwe kansen: naar een 'semi-open' systeem van gelijkebehandelingswetgeving?, -85 of the (then) Dutch Equal Treatment Commission (since 2012 the Netherlands Human Rights Institute
  43. 2015, Geoffrey L�ger v. Ministre des Affaires sociales, de la Sant� et des Droits des femmes
  44. 2020, See Decisions 1998-137 and 2007-85 of the (then) Dutch Equal Treatment Commission (since 2012 the Netherlands Human Rights Institute
  45. Janneke H Gerards, 2011, Nieuwe ronde, nieuwe kansen: naar een 'semi-open' systeem van gelijkebehandelingswetgeving? [New Round, New Chances: Toward a 'Semi-Open' System of Equal Treatment Law?] 36 NEDERLANDS TIJDSCHRIFT VOOR DE MENSENRECHTEN, NETH. J. HUM. RTS
  46. 2020, See Decisions 1998-137 and 2007-85 of the (then) Dutch Equal Treatment Commission (since 2012 the Netherlands Human Rights Institute
  47. 2020, Decisions 1998-137 and 2007-85 of the (then) Dutch Equal Treatment Commission (since 2012 the Netherlands Human Rights Institute
  48. 2020, Decisions 1998-137 and 2007-85 of the (then) Dutch Equal Treatment Commission (since 2012 the Netherlands Human Rights Institute
  49. See Janneke, 2011, Nieuwe ronde, nieuwe kansen: naar een 'semi-open' systeem van gelijkebehandelingswetgeving? [New Round, New Chances: Toward a 'Semi-Open' System of Equal Treatment Law?] 36 NEDERLANDS TIJDSCHRIFT VOOR DE MENSENRECHTEN, NETH. J. HUM. RTS
  50. 2000, See also e.g. Dagmar Schiek, A New Framework on Equal Treatment of Persons in EC Law? Directives, 290
  51. 2000, ); and see Dagmar Schiek, A New Framework on Equal Treatment of Persons in EC Law? Directives, RTS. L. REV, 99
  52. 2011, App. No. 56328/07, � 45, 1
  53. 2020, Andrew Altman, Discrimination, THE STANFORD ENCYCLOPEDIA OF PHILOSOPHY �, 4
  54. 2013, As is witnessed by the ECtHR deeming this ground "suspect"; see e, 36
  55. Peter Petkoff, 1997, For the application of these notions to the context of choice/nonchoice grounds, Forum Internum and Forum Externum in Canon Law and Public International Law with a Particular Reference to the Jurisprudence of the European Court of Human Rights, 7 RELIGION & HUM. RTS, 183
  56. Pilvi R�m�, 2019, Towards the Core of Conscience. A Study on Conscientious Objection in the Workplace Under the Freedom of Thought, Conscience and Religion
  57. 1993, ) and Dagmar Schiek, A New Framework on Equal Treatment of Persons in EC Law? Directives, 36516
  58. E G Pilvi R�m�, 2019, Towards the Core of Conscience. A Study on Conscientious Objection in the Workplace Under the Freedom of Thought, Conscience and Religion
  59. Carolyn Evans, 2001, HUMAN RIGHTS, 76
  60. See Pilvi R�m�, 2019, Towards the Core of Conscience. A Study on Conscientious Objection in the Workplace Under the Freedom of Thought, Conscience and Religion
  61. H Janneke, 2008, For a further discussion of different types of religious expression, see e.g. Javier Mart�nez-Torr�n, Manifestations of Religion or Belief in the Case Law of the European Court of Human Rights, 12 RELIGION & HUM, RTS, 147, 120
  62. 2013, On headscarves, 10
  63. C- Case, 2017, Achbita v. G4S Secure Solutions NV
  64. 2017, On blasphemous speech, 38450, 1
  65. See Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 147
  66. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 147
  67. 2020, Andrew Altman, Discrimination, THE STANFORD ENCYCLOPEDIA OF PHILOSOPHY �, 4
  68. 2007, Christopher McCrudden & Haris Kountouros, Human Rights and European Equality Law, EQUALITY LAW IN AN ENLARGED EUROPEAN UNION. UNDERSTANDING THE ARTICLE 13 DIRECTIVES, 4
  69. 2002, See also TIMO MAKKONEN, MULTIPLE, COMPOUND AND INTERSECTIONAL DISCRIMINATION: BRINGING THE EXPERIENCES OF THE MOST MARGINALIZED TO THE FORE 2 (Institute For Human Rights
  70. 2015, PROTECTING VULNERABLE GROUPS: THE EUROPEAN HUMAN RIGHTS FRAMEWORK
  71. Valeska David, 2018, Crossing Divides and Seeking the Whole: An Integrated View of Cultural Difference and Economic Disadvantage in Regional, Human Rights Courts
  72. For a different, more individualised type of conceptualisation, see Martha A. Fineman, Equality, Autonomy, and the Vulnerable Subject in Law and Politics, VULNERABILITY: REFLECTIONS ON A NEW ETHICAL FOUNDATION FOR LAW AND POLITICS, 20
  73. 2020, See also the overview of different rationales in Andrew Altman, Discrimination, 4
  74. However, some recurring problems in open systems are the risk of judicial bias, and the lack of legal certainty
  75. See supra
  76. See Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 153
  77. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 153
  78. See Janneke, 2013, The Discrimination Grounds of Article 14 of the European Convention on Human Rights, 13 HUM, RTS. L. REV, 99
  79. 2007, ) and Christopher McCrudden & Haris Kountouros, Human Rights and European Equality Law, EQUALITY LAW IN AN ENLARGED EUROPEAN UNION. UNDERSTANDING THE ARTICLE 13 DIRECTIVES, 73
  80. See Janneke, 2013, The Discrimination Grounds of Article 14 of the European Convention on Human Rights, 13 HUM, RTS. L. REV, 99
  81. Samantha Besson, 2008, Never Shall the Twain Meet?, 8 HUM, 647, 653
  82. H Janneke, 2013, The Discrimination Grounds of Article 14 of the European Convention on Human Rights, 13 HUM, RTS. L. REV, 99
  83. H Janneke, 2013, The Discrimination Grounds of Article 14 of the European Convention on Human Rights, 13 HUM, RTS. L. REV, 99
  84. H Janneke, 2013, The ground of discrimination may certainly be relevant to the intensity of review and the applicability of the very weighty reasons test, The Discrimination Grounds of Article 14 of the European Convention on Human Rights, 13 HUM. RTS. L. REV. 99, 117
  85. Erwin Chemerinsky, 1983, Defense of Equality: A Reply to Professor Westen
  86. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 154
  87. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 154
  88. See Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 155
  89. H See Also Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 129
  90. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 166
  91. H See Janneke, Nieuwe ronde, nieuwe kansen: naar een 'semi-open' systeem van gelijkebehandelingswetgeving?, New Round, New Chances: Toward a 'Semi-Open' System of Equal Treatment Law?] 36 NEDERLANDS TIJDSCHRIFT VOOR DE MENSENRECHTEN
  92. See supra, Section 5.1
  93. H Janneke, 2008, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 166
  94. H Janneke, Gronden van discriminatie -de wenselijkheid van open en gesloten opsommingen [Grounds of Discrimination -The Desirability of Open and Closed Lists of Grounds, 129
  95. 2008, Critically on the current set-up of the EU non-discrimination directives, see also e.g. Dagmar Schiek, Broadening the Scope and the Norms of

Cited by

Metadata

Authors
Gerards, J., & Zuiderveen Borgesius, F.
Affiliations
Utrecht University (NL)
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2021
ISSN
1556-5068 (ISSN-L)
OpenAlex
W3144953237
Cited by
72
References
32
Citations by year
2026: 2 · 2025: 6 · 2024: 10 · 2023: 48 · 2022: 3 · 2021: 3
Type
posted content
Language
en
Open access
Locations
  • SSRN Electronic Journal (acceptedVersion)
  • Colorado Law Scholarly Commons (University of Colorado Colorado Springs) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Topics
  • Social Sciences › Law › Discrimination and Equality Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
algorithmic decision-making, artificial intelligence, anti-discrimination law, protected grounds, algorithmic discrimination, automatic differentiation, social inequality, legal loopholes, European Union law, algorithmic fairness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.