The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Big Data's Disparate Impact
Open accessAbstract
Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court. This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying data Addressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Where ADM involves personal data, GDPR may help; an array of information should be provided to those whose personal data is being processed 26 , including, in some cases, the existence of ADM and information about the logic involved 27 (the socalled 'right to an explanation' 28 ).”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as [28]
In the citing paper
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [5]
In the citing paper
“Many AI audits focus on technical implementation of principles; are primarily quantitative in nature; and do not include relevant stakeholders or consider use context or socioeconomic impacts [7].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [7]
In the citing paper
“Assumptions in the design of data processing pipelines can affect datasets, altering their content, structure, organization, or representation (Barocas and Selbst, 2016; Poirier, 2018).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“the selection criteria may include features held at a lower rate by members of certain groups (Barocas and Selbst, 2016), and it may over or under-emphasize certain categories of users such as those that are highly-active on a target topic (Cohen and Ruths, 2013)”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“both scientists (Dwork and Mulligan, 2013; Barocas and Selbst, 2016) and journalists (Hill, 2014; Kirchner, 2015) have pressed for greater scrutiny of the use of social data against possible ethical pitfalls, such as breaching users privacy (Goroff, 2015), or enabling racial, socioeconomic or gender-based profiling (Barocas and Selbst, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“The reliance on automated decision making processes based on statistical methods, can inherit, propagate, or even amplify the biases and prejudice present in the training data with respect to various factors such as race, age, gender or socioeconomic groups (Crawford and Schultz, 2014; Barocas and Selbst, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“Privacy breaches can have harmful outcomes (Barocas and Selbst, 2016) like stalking, discrimination, black-mailing or identity theft (Gross and Acquisti, 2005).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In the citing paper
“In fact, an attribute may implicitly encode a protected factor, as occurs with postal code in credit rating [392].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [392]
In the citing paper
“Second, it is by now well-known that 'objective' big data analyses are likely to reproduce the biases in the data they learn from (thus violating the right to non-discrimination) [62].”
From AI for the Common Good?! Pitfalls, challenges, and ethics pen-testing (2019) · cites this paper as [62]
In the citing paper
“The prescribed audit is more accurately described as a 'disparate impact' audit[3, 4, 20], as it measures a specific form of algorithmic bias [31].”
From Auditing Work: Exploring the New York City algorithmic bias audit regime (2024) · cites this paper as [4]
In the citing paper
“Taken together, this literature establishes two claims the remainder of the paper relies on: that algorithmic bias in hiring is real and largely traceable to data and design rather than malice, and that fixing it cannot be left to the technology alone.”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“Removing the protected attribute from the model therefore does not remove the bias; the model reconstructs the attribute from its correlates [2] [3].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“Antidiscrimination law generally holds that an employer cannot escape liability by outsourcing a discriminatory practice; if a selection procedure produces a disparate impact, the employer remains responsible even when an outside vendor designed it [2].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“Yet a substantial body of scholarship shows that such systems can reproduce and even intensify historical discrimination, because models trained on past hiring data learn the patterns embedded in that data, including patterns that track protected characteristics through seemingly neutral proxy variables [2] [3].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
In the citing paper
“fairness and the legal standard of disparate impact are not purely statistical concepts, and involve ethical, political and constitutional concerns15.”
From Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities (2020) · cites this paper as [15]
In the citing paper
“There is widespread agreement on the need for algorithmic fairness, particularly to mitigate the risks of direct and indirect discrimination (under US law, 'disparate treatment' and 'disparate impact', respectively) due to algorithmic decisions (Barocas and Selbst 2016;Grgić-Hlača et al. 2018;Green and Chen 2019).”
From The ethics of algorithms: key problems and solutions (2022) · cites this paper as Barocas and Selbst 2016
In the citing paper
“For instance, when sourcing, specific groups have been found excluded from viewing job ads based on age [10] or gender [12].”
From What does it mean to 'solve' the problem of discrimination in hiring?: social, technical and legal perspectives from the UK on automated hiring systems (2020) · cites this paper as [12]
Citations
Cites
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
(2014) · Development Growth & Differentiation · doi:10.1111/dgd.12079 · Cited by 902
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764
Feature Extraction, Construction and Selection
(1998) · Springer eBooks · doi:10.1007/978-1-4615-5725-8 · Cited by 576
Gaydar: Facebook Friendships Expose Sexual Orientation
Jernigan, C., & Mistree, B. F. T. (2009) · First Monday · doi:10.5210/fm.v14i10.2611 · Cited by 222
Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures
Calders, T., & Žliobaitė, I. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_3 · Cited by 128
Implementing Anti-discrimination Policies in Statistical Profiling Models
Pope, D. G., & Sydnor, J. R. (2011) · American Economic Journal: Economic Policy · doi:10.1257/pol.3.3.206 · Cited by 87
Disparate Impact under Title VII: An Objective Theory of Discrimination
Rutherglen, G. A. (1987) · Virginia Law Review · doi:10.2307/1072940 · Cited by 44
Defining and minimizing the risk of complications during carotid artery interventions
Henry, M., Ohki, T., Polydorou, A., et al. (2004) · CRC Press eBooks · doi:10.3109/9780203490891-59 · Cited by 20
Foreword: Mental Health Law and the Movement toward Voluntary Treatment
Wexler, D. B. (1974) · California Law Review · doi:10.2307/3479745 · Cited by 10
Flagged first, then by citations
Research works printed but not linked: 24
- Int'l J, 2008, ETHICS, 37, 48
- Insurance offers the most obvious example of this: the rate that a person pays for car insurance, for 201. See supra text accompanying notes 106-07. 202. See supra Part I
- Stacy E Seicshnaydre, 2007, Is the Road to Disparate Impact Paved with Good Intentions?, Stuck on State of Mind in Antidiscrimination Law, 42 WAKE FOREST L. REV. 1141, 1142
- Supra discussion accompanying note 101
- Labi, Director of Research for Philadelphia's Adult Probation and Parole Department), 5
- 2004, In a sense, computer scientists have unwittingly furnished the kind of evidence that social scientists routinely seek: the particular contours of inequality, SOCIAL INEQUALITY (Kathryn M
- Dwork, at 215; cf. Wax, supra note 146, at 711 (noting intractable problems due to a "validity-diversity tradeoff" in employment metrics). 250. Peck, supra note 181
- ; Wall Id, 2012, Other companies have not held back from considering this information for the very same purposes, See Joseph Walker, Meet the New Boss: Big Data
- Helen Norton, 2010, The Supreme Court's Post-Racial Turn Towards a Zero-Sum Understanding of Equality, 52 WM, & MARY L. REV, 197, 40
- M Owen, 1976, Groups and the Equal Protection Clause, 5 PHIL. & PUB. AFF, 107
- See Norton, supra note 253. 255. Id. at 209. 256. Id. at 206. 257. Primus, supra note 98, at 520-23. 258. Id.; Perry, supra note 138
- 1971, The objective of Congress the Act, practices, procedures, or tests neutral on their face, and even neutral in terms of intent, cannot be maintained if they operate to 'freeze' the status quo of prior discriminatory employment practices, 429
- See Tex, Dep't of Hous. & Cmty Aff. v. Inclusive Cmtys. Project, Inc., No. 13-1371, slip op. at 8 (Sup. Ct. 2015) (quoting Griggs, 401 U.S. at 431). 261. Primus, supra note 98
- See Bagenstos, 1976, 229, 246
- 2009, 267. Id. 268. Id. 269. Id. at 563. 270. Primus, supra note 92
- Cardozo L, 2013, 2157, 2162
- While the decision was formally about Title VII only, and thus amenable to statutory resolution, the reasoning applied equally well to a future equal protection claim, endangering the future of disparate impact, 1385
- A Bradley, 2012, The Anticlassification Turn in Employment Discrimination Law, 63 ALA, L. REV, 955
- J Norton ; Scalia, 2016, concurring) ("[The Court's] resolution of this dispute merely postpones the evil day on which the Court will have to confront the question: Whether, or to what extent, are the disparateimpact provisions of Title VII of the Civil Rights Act of 1964 consistent with the Constitution's guarantee of equal protection?"). But the Supreme Court seemed to pull back from the brink last term, approving of the use of disparate impact in a new setting-the Fair Housing Act-and engaging deeply with the constitutional issues that Ricci raised, settling them for now. Samuel R. Bagenstos, Disparate Impact and the Role of Classification and Motivation in Equal Protection Law After Inclusive Communities, Justice Scalia stated as much in his concurrence. Ricci, 557 U.S. at, 594, 11
- Title VII does not prohibit an employer from considering, before administering a test or practice, how to design that test or practice in order to provide a fair opportunity for all individuals, regardless of their race. And when, during the test-design stage, an employer invites comments to ensure the test is fair
- We cannot stress enough the import of these caveats. Certainty is a strong and unlikely precondition, and the status quo should not be taken as a given, as we explain below. 285. See generally Jolls
- See Steven, 1985, The Disparate Impact Model of Discrimination: Theory and Limits, AM. U. L. REV, 799, 809
- Solon Barocas, 2014, Putting Data to Work, DATA AND DISCRIMINATION: COLLECTED ESSAYS, 58
- Claire Cain Miller, 2015, Can an Algorithm Hire Better than a Human?, N.Y. TIMES
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“The sources for bias, as indicated in [392] , can be traced to: • Skewed data: Bias within the data acquisition process.”
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
“However, proxies for protected attributes are not easy to predict or detect (Romei and Ruggieri, 2014; Zarsky, 2016), particularly when algorithms access linked datasets (Barocas and Selbst, 2015).”
InfluentialKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Privacy in the age of medical big data
Price, W. N., & Cohen, I. G. (2019) · Nature Medicine · doi:10.1038/s41591-018-0272-7 · Cited by 1,522
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338
A SWOT analysis of ChatGPT: Implications for educational practice and research
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2195846 · Cited by 1,384
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Gorwa, R., Binns, R., & Katzenbach, C. (2020) · Big Data & Society · doi:10.1177/2053951719897945 · Cited by 1,097
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
“…formal definitions of fairness into philosophical or jurisprudential notions in part because the authors of this article lack the expertise and in part because that multidisciplinary conversation is just beginning (Barocas and Selbst 2016; Ferguson 2015; Jans-sen and Kuk 2016; Kroll et al. 2017).”
BackgroundInherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943
“Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”
BackgroundMachine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
“Big Data Surveillance: The Case of Policing
Brayne, S. (2017) · American Sociological Review · doi:10.1177/0003122417725865 · Cited by 596
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Self-citationFerrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875
“Can be complex and require large amounts of additional data [32].”
BackgroundMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“…automated decision making processes based on statistical methods, can inherit, propagate, or even amplify the biases and prejudice present in the training data with respect to various factors such as race, age, gender or socioeconomic groups (Crawford and Schultz, 2014; Barocas and Selbst, 2016).”
BackgroundInfluentialEthical Implications and Accountability of Algorithms
Martin, K. (2019) · Journal of Business Ethics · doi:10.1007/s10551-018-3921-3 · Cited by 757
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783
Governing artificial intelligence: ethical, legal and technical opportunities and challenges
Cath, C. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0080 · Cited by 745
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709
(2019) · doi:10.1080/1369118x.2019.1573914 · Cited by 719
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Self-citationZafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632
Consumer-lending discrimination in the FinTech Era
Bartlett, R., Morse, A., Stanton, R., & Wallace, N. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.05.047 · Cited by 659
Fairness of Exposure in Rankings
Singh, A., & Joachims, T. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220088 · Cited by 673
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598
Sex and gender analysis improves science and engineering
Tannenbaum, C., Ellis, R. P., Eyssel, F., Zou, J., & Schiebinger, L. L. (2019) · Nature · doi:10.1038/s41586-019-1657-6 · Cited by 625
There is a blind spot in AI research
Crawford, K., & Calo, R. (2016) · Nature · doi:10.1038/538311a · Cited by 580
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
“So models based on such variables can still negatively impact individuals or classes of individuals [74].”
BackgroundWhere fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535
A Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
Veale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512
Artificial intelligence, systemic risks, and sustainability
Galaz, V., Centeno, M. A., Callahan, P. W., Causevic, A., Patterson, T., Brass, I., Baum, S., Farber, D., Fischer, J., Garcia, D., et al. (2021) · Technology in Society · doi:10.1016/j.techsoc.2021.101741 · Cited by 497
Köchling, A., & Wehner, M. C. (2020) · Business Research · doi:10.1007/s40685-020-00134-w · Cited by 511
Artificial Intelligence in Society
OECD (2019) · doi:10.1787/eedfee77-en · Cited by 487
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496
Algorithmic Bias in Autonomous Systems
Danks, D., & London, A. J. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/654 · Cited by 472
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
BackgroundRegulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466
“Generally, it applies, in the US as well as the EU, in a technology-neutral way [4, 109-112, 113 ; but LGBTI, para.”
BackgroundDurán, J. M., & Jongsma, K. R. (2021) · Journal of Medical Ethics · doi:10.1136/medethics-2020-106820 · Cited by 446
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443
Can fintech improve the efficiency of commercial banks? —An analysis based on big data
Wang, Y., Xiuping, S., & Zhang, Q. (2020) · Research in International Business and Finance · doi:10.1016/j.ribaf.2020.101338 · Cited by 459
SoK: Towards the Science of security and privacy in machine learning
Papernot, N., McDaniel, P. D., Sinha, A., & Wellman, M. P. (2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 427
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
Ethics of AI-enabled recruiting and selection: a review and research agenda
Hunkenschroer, A. L., & Luetge, C. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05049-6 · Cited by 447
Cheong, B. C. (2024) · Frontiers in Human Dynamics · doi:10.3389/fhumd.2024.1421273 · Cited by 445
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419
“Machine learning is playing a growing role in decision making on issues of broad societal importance (Munoz et al., 2016; Barocas and Selbst, 2016).”
BackgroundSituating methods in the magic of Big Data and AI
Elish, M. C., & boyd, D. (2017) · Communication Monographs · doi:10.1080/03637751.2017.1375130 · Cited by 488
Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404
Fairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676
Algorithms in practice: Comparing web journalism and criminal justice
Christin, A. (2017) · Big Data & Society · doi:10.1177/2053951717718855 · Cited by 379
Discrimination in the Age of Algorithms
Kleinberg, J., Ludwig, J., Mullainathan, S., & Sunstein, C. R. (2018) · Journal of Legal Analysis · doi:10.1093/jla/laz001 · Cited by 401
Social Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373
The Intuitive Appeal of Explainable Machines
Selbst, A. D., & Barocas, S. (2018) · doi:10.2139/ssrn.3126971 · Cited by 392
Self-citationGender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384
Artificial Intelligence and Journalism
Broussard, M., Diakopoulos, N. A., Guzman, A. L., Abebe, R., Dupagne, M., & Chuan, C. (2019) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699019859901 · Cited by 418
Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330691 · Cited by 428
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep LearningPreprint
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 592
van Noordt, C., & Misuraca, G. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101714 · Cited by 356
Measuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348
The Challenges of Algorithm-Based HR Decision-Making for Personal Integrity
Leicht-Deobald, U., Busch, T., Schank, C., Weibel, A., Schafheitle, S., Wildhaber, I., & Kasper, G. (2019) · Journal of Business Ethics · doi:10.1007/s10551-019-04204-w · Cited by 347
Algorithmic Prediction in Policing: Assumptions, Evaluation, and Accountability
Moses, L. B., & Chan, J. B. L. (2016) · Policing & Society · doi:10.1080/10439463.2016.1253695 · Cited by 453
Ten simple rules for responsible big data research
Zook, M., Barocas, S., boyd, D., Crawford, K., Keller, E., Gangadharan, S. P., Goodman, A. A., Hollander, R. D., Koenig, B. A., Metcalf, J., et al. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005399 · Cited by 394
Self-citationAlgorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334
Katzenbach, C., & Ulbricht, L. (2019) · Internet Policy Review · doi:10.14763/2019.4.1424 · Cited by 359
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322
When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring
van den Broek, E., Sergeeva, A., & Huysman, M. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16559 · Cited by 330
The ethnographer and the algorithm: beyond the black box
Christin, A. (2020) · Theory and Society · doi:10.1007/s11186-020-09411-3 · Cited by 327
Inherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 350
Fourcade, M., & Healy, K. (2016) · Socio-Economic Review · doi:10.1093/ser/mww033 · Cited by 474
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337
Predict and Surveil: Data, Discretion, and the Future of Policing
Brayne, S. (2020) · doi:10.1093/oso/9780190684099.001.0001 · Cited by 334
Bias and Discrimination in AI: A Cross-Disciplinary Perspective
Ferrer, X., Nuenen, T. V., Such, J. M., Cote, M., & Criado, N. (2021) · IEEE Technology and Society Magazine · doi:10.1109/mts.2021.3056293 · Cited by 308
“The Castaneda rule, which states that the number of people of the protected group selected from a relevant population cannot be smaller than 3 standard deviations the number expected in a random selection, is also used [21].”
BackgroundFairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316
Königstorfer, F., & Thalmann, S. (2020) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2020.100352 · Cited by 299
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 559
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287
Paulus, J. K., & Kent, D. M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0304-9 · Cited by 274
AI in education: learner choice and fundamental rights
Berendt, B., Littlejohn, A., & Blakemore, M. (2020) · Learning, Media and Technology · doi:10.1080/17439884.2020.1786399 · Cited by 281
Regan, P. M., & Jesse, J. (2019) · Ethics and Information Technology · doi:10.1007/s10676-018-9492-2 · Cited by 286
Technologies of Crime Prediction: The Reception of Algorithms in Policing and Criminal Courts
Brayne, S., & Christin, A. (2021) · Social Problems · doi:10.1093/socpro/spaa004 · Cited by 279
Metadata
- Authors
- Barocas, S., & Selbst, A. D.
- Affiliations
- Cornell University (US)
- University of California, Los Angeles (US)
- Corresponding author
- Barocas, Solon
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2014
- Volume
- 104
- Issue
- 3
- Pages
- 671
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W3123374861
- Cited by
- 3,048
- References
- 20
- Citations by year
- 2026: 258 · 2025: 362 · 2024: 391 · 2023: 370 · 2022: 318 · 2021: 372 · 2020: 389 · 2019: 278 · 2018: 143 · 2017: 100 · 2016: 47 · 2015: 12 · 2014: 5
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- bepress Legal Repository (submittedVersion)
- California Law Review
- UC Berkeley
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Discrimination and Equality Law
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- big data, algorithmic discrimination, indirect discrimination, data mining, Title VII, employment discrimination, anti-subordination, anti-discrimination law, predictive analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
