Big Data's Disparate Impact
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court. This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying data Addressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Where ADM involves personal data, GDPR may help; an array of information should be provided to those whose personal data is being processed 26 , including, in some cases, the existence of ADM and information about the logic involved 27 (the socalled 'right to an explanation' 28 ).”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as [28]
In the citing paper
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [5]
In the citing paper
“Many AI audits focus on technical implementation of principles; are primarily quantitative in nature; and do not include relevant stakeholders or consider use context or socioeconomic impacts [7].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [7]
In the citing paper
“Assumptions in the design of data processing pipelines can affect datasets, altering their content, structure, organization, or representation (Barocas and Selbst, 2016; Poirier, 2018).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“the selection criteria may include features held at a lower rate by members of certain groups (Barocas and Selbst, 2016), and it may over or under-emphasize certain categories of users such as those that are highly-active on a target topic (Cohen and Ruths, 2013)”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
14 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
(2014) · Development, Growth & Differentiation · doi:10.1111/dgd.12079 · Cited by 902
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764
Feature Extraction, Construction and Selection
(1998) · Springer eBooks · doi:10.1007/978-1-4615-5725-8 · Cited by 573
Gaydar: Facebook Friendships Expose Sexual Orientation
Jernigan, C., & Mistree, B. F. T. (2009) · First Monday · doi:10.5210/fm.v14i10.2611 · Cited by 222
Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures
Calders, T., & Žliobaitė, I. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_3 · Cited by 128
Implementing Anti-discrimination Policies in Statistical Profiling Models
Pope, D. G., & Sydnor, J. R. (2011) · American Economic Journal: Economic Policy · doi:10.1257/pol.3.3.206 · Cited by 87
Disparate Impact under Title VII: An Objective Theory of Discrimination
Rutherglen, G. A. (1987) · Virginia Law Review · doi:10.2307/1072940 · Cited by 39
Defining and minimizing the risk of complications during carotid artery interventions
Henry, M., Ohki, T., Polydorou, A., Strigaris, K., & Kiskinis, D. (2004) · CRC Press eBooks · doi:10.3109/9780203490891-59 · Cited by 20
Foreword: Mental Health Law and the Movement toward Voluntary Treatment
Wexler, D. B. (1974) · California Law Review · doi:10.2307/3479745 · Cited by 10
Research works printed but not linked: 24
- Int'l J, 2008, ETHICS, 37, 48
- Insurance offers the most obvious example of this: the rate that a person pays for car insurance, for 201. See supra text accompanying notes 106-07. 202. See supra Part I
- Stacy E Seicshnaydre, 2007, Is the Road to Disparate Impact Paved with Good Intentions?, Stuck on State of Mind in Antidiscrimination Law, 42 WAKE FOREST L. REV. 1141, 1142
- Supra discussion accompanying note 101
- Labi, Director of Research for Philadelphia's Adult Probation and Parole Department), 5
- 2004, In a sense, computer scientists have unwittingly furnished the kind of evidence that social scientists routinely seek: the particular contours of inequality, SOCIAL INEQUALITY (Kathryn M
- Dwork, at 215; cf. Wax, supra note 146, at 711 (noting intractable problems due to a "validity-diversity tradeoff" in employment metrics). 250. Peck, supra note 181
- ; Wall Id, 2012, Other companies have not held back from considering this information for the very same purposes, See Joseph Walker, Meet the New Boss: Big Data
- Helen Norton, 2010, The Supreme Court's Post-Racial Turn Towards a Zero-Sum Understanding of Equality, 52 WM, & MARY L. REV, 197, 40
- M Owen, 1976, Groups and the Equal Protection Clause, 5 PHIL. & PUB. AFF, 107
- See Norton, supra note 253. 255. Id. at 209. 256. Id. at 206. 257. Primus, supra note 98, at 520-23. 258. Id.; Perry, supra note 138
- 1971, The objective of Congress the Act, practices, procedures, or tests neutral on their face, and even neutral in terms of intent, cannot be maintained if they operate to 'freeze' the status quo of prior discriminatory employment practices, 429
- See Tex, Dep't of Hous. & Cmty Aff. v. Inclusive Cmtys. Project, Inc., No. 13-1371, slip op. at 8 (Sup. Ct. 2015) (quoting Griggs, 401 U.S. at 431). 261. Primus, supra note 98
- See Bagenstos, 1976, 229, 246
- 2009, 267. Id. 268. Id. 269. Id. at 563. 270. Primus, supra note 92
- Cardozo L, 2013, 2157, 2162
- While the decision was formally about Title VII only, and thus amenable to statutory resolution, the reasoning applied equally well to a future equal protection claim, endangering the future of disparate impact, 1385
- A Bradley, 2012, The Anticlassification Turn in Employment Discrimination Law, 63 ALA, L. REV, 955
- J Norton ; Scalia, 2016, concurring) ("[The Court's] resolution of this dispute merely postpones the evil day on which the Court will have to confront the question: Whether, or to what extent, are the disparateimpact provisions of Title VII of the Civil Rights Act of 1964 consistent with the Constitution's guarantee of equal protection?"). But the Supreme Court seemed to pull back from the brink last term, approving of the use of disparate impact in a new setting-the Fair Housing Act-and engaging deeply with the constitutional issues that Ricci raised, settling them for now. Samuel R. Bagenstos, Disparate Impact and the Role of Classification and Motivation in Equal Protection Law After Inclusive Communities, Justice Scalia stated as much in his concurrence. Ricci, 557 U.S. at, 594, 11
- Title VII does not prohibit an employer from considering, before administering a test or practice, how to design that test or practice in order to provide a fair opportunity for all individuals, regardless of their race. And when, during the test-design stage, an employer invites comments to ensure the test is fair
- We cannot stress enough the import of these caveats. Certainty is a strong and unlikely precondition, and the status quo should not be taken as a given, as we explain below. 285. See generally Jolls
- See Steven, 1985, The Disparate Impact Model of Discrimination: Theory and Limits, AM. U. L. REV, 799, 809
- Solon Barocas, 2014, Putting Data to Work, DATA AND DISCRIMINATION: COLLECTED ESSAYS, 58
- Claire Cain Miller, 2015, Can an Algorithm Hire Better than a Human?, N.Y. TIMES
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“The sources for bias, as indicated in [392] , can be traced to: • Skewed data: Bias within the data acquisition process.”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“The sources for bias, as indicated in [392] , can be traced to: • Skewed data: Bias within the data acquisition process.”
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611
“…for e.g., welfare service provision or addressing anti-social behaviour, it can also amplify discriminatory biases that are already present in human-led assessments: predictive algorithms, in fact, can favour groups that are better represented in the training data (Barocas & Selbst, 2016).”
BackgroundThe ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
“However, proxies for protected attributes are not easy to predict or detect (Romei and Ruggieri, 2014; Zarsky, 2016), particularly when algorithms access linked datasets (Barocas and Selbst, 2015).”
Influential
Metadata
- Authors
- Barocas, S., & Selbst, A. D.
- Affiliations
- Cornell University (US)
- University of California, Los Angeles (US)
- Corresponding author
- Barocas, Solon
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2014
- Volume
- 104
- Issue
- 3
- Pages
- 671
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W3123374861
- Cited by
- 3,048
- References
- 16
- Citations by year
- 2026: 258 · 2025: 362 · 2024: 391 · 2023: 370 · 2022: 318 · 2021: 372 · 2020: 389 · 2019: 278 · 2018: 143 · 2017: 100 · 2016: 47 · 2015: 12 · 2014: 5
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- bepress Legal Repository (submittedVersion)
- California Law Review
- UC Berkeley
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Discrimination and Equality Law
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- big data, algorithmic discrimination, indirect discrimination, data mining, Title VII, employment discrimination, anti-subordination, anti-discrimination law, predictive analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Big Data's Disparate Impact. https://app.cochat.ai/papers/10.2139/ssrn.2477899
- The paper
- Barocas, S., & Selbst, A. D. (2014). Big Data's Disparate Impact. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2477899
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.