The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Big Data's Disparate Impact
Open accessAbstract
Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court. This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying data Addressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Where ADM involves personal data, GDPR may help; an array of information should be provided to those whose personal data is being processed 26 , including, in some cases, the existence of ADM and information about the logic involved 27 (the socalled 'right to an explanation' 28 ).”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as [28]
In the citing paper
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [5]
In the citing paper
“Many AI audits focus on technical implementation of principles; are primarily quantitative in nature; and do not include relevant stakeholders or consider use context or socioeconomic impacts [7].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [7]
In the citing paper
“Assumptions in the design of data processing pipelines can affect datasets, altering their content, structure, organization, or representation (Barocas and Selbst, 2016; Poirier, 2018).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“the selection criteria may include features held at a lower rate by members of certain groups (Barocas and Selbst, 2016), and it may over or under-emphasize certain categories of users such as those that are highly-active on a target topic (Cohen and Ruths, 2013)”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“both scientists (Dwork and Mulligan, 2013; Barocas and Selbst, 2016) and journalists (Hill, 2014; Kirchner, 2015) have pressed for greater scrutiny of the use of social data against possible ethical pitfalls, such as breaching users privacy (Goroff, 2015), or enabling racial, socioeconomic or gender-based profiling (Barocas and Selbst, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“The reliance on automated decision making processes based on statistical methods, can inherit, propagate, or even amplify the biases and prejudice present in the training data with respect to various factors such as race, age, gender or socioeconomic groups (Crawford and Schultz, 2014; Barocas and Selbst, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“Privacy breaches can have harmful outcomes (Barocas and Selbst, 2016) like stalking, discrimination, black-mailing or identity theft (Gross and Acquisti, 2005).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“In fact, an attribute may implicitly encode a protected factor, as occurs with postal code in credit rating [392].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [392]
In the citing paper
“Second, it is by now well-known that 'objective' big data analyses are likely to reproduce the biases in the data they learn from (thus violating the right to non-discrimination) [62].”
From AI for the Common Good?! Pitfalls, challenges, and ethics pen-testing (2019) · cites this paper as [62]
In the citing paper
“The prescribed audit is more accurately described as a 'disparate impact' audit[3, 4, 20], as it measures a specific form of algorithmic bias [31].”
From Auditing Work: Exploring the New York City algorithmic bias audit regime (2024) · cites this paper as [4]
In the citing paper
“Taken together, this literature establishes two claims the remainder of the paper relies on: that algorithmic bias in hiring is real and largely traceable to data and design rather than malice, and that fixing it cannot be left to the technology alone.”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“Removing the protected attribute from the model therefore does not remove the bias; the model reconstructs the attribute from its correlates [2] [3].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“Antidiscrimination law generally holds that an employer cannot escape liability by outsourcing a discriminatory practice; if a selection procedure produces a disparate impact, the employer remains responsible even when an outside vendor designed it [2].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“Yet a substantial body of scholarship shows that such systems can reproduce and even intensify historical discrimination, because models trained on past hiring data learn the patterns embedded in that data, including patterns that track protected characteristics through seemingly neutral proxy variables [2] [3].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“fairness and the legal standard of disparate impact are not purely statistical concepts, and involve ethical, political and constitutional concerns15.”
From Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities (2020) · cites this paper as [15]
In the citing paper
“There is widespread agreement on the need for algorithmic fairness, particularly to mitigate the risks of direct and indirect discrimination (under US law, 'disparate treatment' and 'disparate impact', respectively) due to algorithmic decisions (Barocas and Selbst 2016;Grgić-Hlača et al. 2018;Green and Chen 2019).”
From The ethics of algorithms: key problems and solutions (2022) · cites this paper as Barocas and Selbst 2016
In the citing paper
“For instance, when sourcing, specific groups have been found excluded from viewing job ads based on age [10] or gender [12].”
From What does it mean to 'solve' the problem of discrimination in hiring?: social, technical and legal perspectives from the UK on automated hiring systems (2020) · cites this paper as [12]
Citations
Cites
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
(2014) · Development Growth & Differentiation · doi:10.1111/dgd.12079 · Cited by 902
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764
Feature Extraction, Construction and Selection
(1998) · Springer eBooks · doi:10.1007/978-1-4615-5725-8 · Cited by 576
Gaydar: Facebook Friendships Expose Sexual Orientation
Jernigan, C., & Mistree, B. F. T. (2009) · First Monday · doi:10.5210/fm.v14i10.2611 · Cited by 222
Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures
Calders, T., & Žliobaitė, I. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_3 · Cited by 128
Implementing Anti-discrimination Policies in Statistical Profiling Models
Pope, D. G., & Sydnor, J. R. (2011) · American Economic Journal: Economic Policy · doi:10.1257/pol.3.3.206 · Cited by 87
Disparate Impact under Title VII: An Objective Theory of Discrimination
Rutherglen, G. A. (1987) · Virginia Law Review · doi:10.2307/1072940 · Cited by 44
Defining and minimizing the risk of complications during carotid artery interventions
Henry, M., Ohki, T., Polydorou, A., et al. (2004) · CRC Press eBooks · doi:10.3109/9780203490891-59 · Cited by 20
Foreword: Mental Health Law and the Movement toward Voluntary Treatment
Wexler, D. B. (1974) · California Law Review · doi:10.2307/3479745 · Cited by 10
Flagged first, then by citations
Research works printed but not linked: 24
- Int'l J, 2008, ETHICS, 37, 48
- Insurance offers the most obvious example of this: the rate that a person pays for car insurance, for 201. See supra text accompanying notes 106-07. 202. See supra Part I
- Stacy E Seicshnaydre, 2007, Is the Road to Disparate Impact Paved with Good Intentions?, Stuck on State of Mind in Antidiscrimination Law, 42 WAKE FOREST L. REV. 1141, 1142
- Supra discussion accompanying note 101
- Labi, Director of Research for Philadelphia's Adult Probation and Parole Department), 5
- 2004, In a sense, computer scientists have unwittingly furnished the kind of evidence that social scientists routinely seek: the particular contours of inequality, SOCIAL INEQUALITY (Kathryn M
- Dwork, at 215; cf. Wax, supra note 146, at 711 (noting intractable problems due to a "validity-diversity tradeoff" in employment metrics). 250. Peck, supra note 181
- ; Wall Id, 2012, Other companies have not held back from considering this information for the very same purposes, See Joseph Walker, Meet the New Boss: Big Data
- Helen Norton, 2010, The Supreme Court's Post-Racial Turn Towards a Zero-Sum Understanding of Equality, 52 WM, & MARY L. REV, 197, 40
- M Owen, 1976, Groups and the Equal Protection Clause, 5 PHIL. & PUB. AFF, 107
- See Norton, supra note 253. 255. Id. at 209. 256. Id. at 206. 257. Primus, supra note 98, at 520-23. 258. Id.; Perry, supra note 138
- 1971, The objective of Congress the Act, practices, procedures, or tests neutral on their face, and even neutral in terms of intent, cannot be maintained if they operate to 'freeze' the status quo of prior discriminatory employment practices, 429
- See Tex, Dep't of Hous. & Cmty Aff. v. Inclusive Cmtys. Project, Inc., No. 13-1371, slip op. at 8 (Sup. Ct. 2015) (quoting Griggs, 401 U.S. at 431). 261. Primus, supra note 98
- See Bagenstos, 1976, 229, 246
- 2009, 267. Id. 268. Id. 269. Id. at 563. 270. Primus, supra note 92
- Cardozo L, 2013, 2157, 2162
- While the decision was formally about Title VII only, and thus amenable to statutory resolution, the reasoning applied equally well to a future equal protection claim, endangering the future of disparate impact, 1385
- A Bradley, 2012, The Anticlassification Turn in Employment Discrimination Law, 63 ALA, L. REV, 955
- J Norton ; Scalia, 2016, concurring) ("[The Court's] resolution of this dispute merely postpones the evil day on which the Court will have to confront the question: Whether, or to what extent, are the disparateimpact provisions of Title VII of the Civil Rights Act of 1964 consistent with the Constitution's guarantee of equal protection?"). But the Supreme Court seemed to pull back from the brink last term, approving of the use of disparate impact in a new setting-the Fair Housing Act-and engaging deeply with the constitutional issues that Ricci raised, settling them for now. Samuel R. Bagenstos, Disparate Impact and the Role of Classification and Motivation in Equal Protection Law After Inclusive Communities, Justice Scalia stated as much in his concurrence. Ricci, 557 U.S. at, 594, 11
- Title VII does not prohibit an employer from considering, before administering a test or practice, how to design that test or practice in order to provide a fair opportunity for all individuals, regardless of their race. And when, during the test-design stage, an employer invites comments to ensure the test is fair
- We cannot stress enough the import of these caveats. Certainty is a strong and unlikely precondition, and the status quo should not be taken as a given, as we explain below. 285. See generally Jolls
- See Steven, 1985, The Disparate Impact Model of Discrimination: Theory and Limits, AM. U. L. REV, 799, 809
- Solon Barocas, 2014, Putting Data to Work, DATA AND DISCRIMINATION: COLLECTED ESSAYS, 58
- Claire Cain Miller, 2015, Can an Algorithm Hire Better than a Human?, N.Y. TIMES
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“The sources for bias, as indicated in [392] , can be traced to: • Skewed data: Bias within the data acquisition process.”
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
“However, proxies for protected attributes are not easy to predict or detect (Romei and Ruggieri, 2014; Zarsky, 2016), particularly when algorithms access linked datasets (Barocas and Selbst, 2015).”
InfluentialKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Metadata
- Authors
- Barocas, S., & Selbst, A. D.
- Affiliations
- Cornell University (US)
- University of California, Los Angeles (US)
- Corresponding author
- Barocas, Solon
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2014
- Volume
- 104
- Issue
- 3
- Pages
- 671
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W3123374861
- Cited by
- 3,048
- References
- 20
- Citations by year
- 2026: 258 · 2025: 362 · 2024: 391 · 2023: 370 · 2022: 318 · 2021: 372 · 2020: 389 · 2019: 278 · 2018: 143 · 2017: 100 · 2016: 47 · 2015: 12 · 2014: 5
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- bepress Legal Repository (submittedVersion)
- California Law Review
- UC Berkeley
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Discrimination and Equality Law
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- big data, algorithmic discrimination, indirect discrimination, data mining, Title VII, employment discrimination, anti-subordination, anti-discrimination law, predictive analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
