PreprintCited by 3,048Ethics and Social Impacts of AIDiscrimination and Equality LawAI and HR Technologies

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Big Data's Disparate Impact

Open access
Barocas, Solon, Selbst, Andrew D.·2014·SSRN Electronic Journal

Abstract

Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court. This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying data Addressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”

Overview

Integrity signals

Registry

Crossref · 10.2139/ssrn.2477899

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 20 references

Research works printed but not linked: 24

  1. Int'l J, 2008, ETHICS, 37, 48
  2. Insurance offers the most obvious example of this: the rate that a person pays for car insurance, for 201. See supra text accompanying notes 106-07. 202. See supra Part I
  3. Stacy E Seicshnaydre, 2007, Is the Road to Disparate Impact Paved with Good Intentions?, Stuck on State of Mind in Antidiscrimination Law, 42 WAKE FOREST L. REV. 1141, 1142
  4. Supra discussion accompanying note 101
  5. Labi, Director of Research for Philadelphia's Adult Probation and Parole Department), 5
  6. 2004, In a sense, computer scientists have unwittingly furnished the kind of evidence that social scientists routinely seek: the particular contours of inequality, SOCIAL INEQUALITY (Kathryn M
  7. Dwork, at 215; cf. Wax, supra note 146, at 711 (noting intractable problems due to a "validity-diversity tradeoff" in employment metrics). 250. Peck, supra note 181
  8. ; Wall Id, 2012, Other companies have not held back from considering this information for the very same purposes, See Joseph Walker, Meet the New Boss: Big Data
  9. Helen Norton, 2010, The Supreme Court's Post-Racial Turn Towards a Zero-Sum Understanding of Equality, 52 WM, & MARY L. REV, 197, 40
  10. M Owen, 1976, Groups and the Equal Protection Clause, 5 PHIL. & PUB. AFF, 107
  11. See Norton, supra note 253. 255. Id. at 209. 256. Id. at 206. 257. Primus, supra note 98, at 520-23. 258. Id.; Perry, supra note 138
  12. 1971, The objective of Congress the Act, practices, procedures, or tests neutral on their face, and even neutral in terms of intent, cannot be maintained if they operate to 'freeze' the status quo of prior discriminatory employment practices, 429
  13. See Tex, Dep't of Hous. & Cmty Aff. v. Inclusive Cmtys. Project, Inc., No. 13-1371, slip op. at 8 (Sup. Ct. 2015) (quoting Griggs, 401 U.S. at 431). 261. Primus, supra note 98
  14. See Bagenstos, 1976, 229, 246
  15. 2009, 267. Id. 268. Id. 269. Id. at 563. 270. Primus, supra note 92
  16. Cardozo L, 2013, 2157, 2162
  17. While the decision was formally about Title VII only, and thus amenable to statutory resolution, the reasoning applied equally well to a future equal protection claim, endangering the future of disparate impact, 1385
  18. A Bradley, 2012, The Anticlassification Turn in Employment Discrimination Law, 63 ALA, L. REV, 955
  19. J Norton ; Scalia, 2016, concurring) ("[The Court's] resolution of this dispute merely postpones the evil day on which the Court will have to confront the question: Whether, or to what extent, are the disparateimpact provisions of Title VII of the Civil Rights Act of 1964 consistent with the Constitution's guarantee of equal protection?"). But the Supreme Court seemed to pull back from the brink last term, approving of the use of disparate impact in a new setting-the Fair Housing Act-and engaging deeply with the constitutional issues that Ricci raised, settling them for now. Samuel R. Bagenstos, Disparate Impact and the Role of Classification and Motivation in Equal Protection Law After Inclusive Communities, Justice Scalia stated as much in his concurrence. Ricci, 557 U.S. at, 594, 11
  20. Title VII does not prohibit an employer from considering, before administering a test or practice, how to design that test or practice in order to provide a fair opportunity for all individuals, regardless of their race. And when, during the test-design stage, an employer invites comments to ensure the test is fair
  21. We cannot stress enough the import of these caveats. Certainty is a strong and unlikely precondition, and the status quo should not be taken as a given, as we explain below. 285. See generally Jolls
  22. See Steven, 1985, The Disparate Impact Model of Discrimination: Theory and Limits, AM. U. L. REV, 799, 809
  23. Solon Barocas, 2014, Putting Data to Work, DATA AND DISCRIMINATION: COLLECTED ESSAYS, 58
  24. Claire Cain Miller, 2015, Can an Algorithm Hire Better than a Human?, N.Y. TIMES

Cited by

Sorted by citations

Items 901–1,000 of 1,105

Metadata

Authors
Barocas, S., & Selbst, A. D.
Affiliations
  • Cornell University (US)
  • University of California, Los Angeles (US)
Corresponding author
Barocas, Solon
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2014
Volume
104
Issue
3
Pages
671
ISSN
1556-5068 (ISSN-L)
OpenAlex
W3123374861
Cited by
3,048
References
20
Citations by year
2026: 258 · 2025: 362 · 2024: 391 · 2023: 370 · 2022: 318 · 2021: 372 · 2020: 389 · 2019: 278 · 2018: 143 · 2017: 100 · 2016: 47 · 2015: 12 · 2014: 5
Type
posted content
Language
en
Open access
Locations
  • SSRN Electronic Journal (acceptedVersion)
  • bepress Legal Repository (submittedVersion)
  • California Law Review
  • UC Berkeley
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Discrimination and Equality Law
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
Keywords
big data, algorithmic discrimination, indirect discrimination, data mining, Title VII, employment discrimination, anti-subordination, anti-discrimination law, predictive analytics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review