Big Data's Disparate Impact
Open accessAbstract
Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court. This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying data Addressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Where ADM involves personal data, GDPR may help; an array of information should be provided to those whose personal data is being processed, including, in some cases, the existence of ADM and information about the logic involved.
“Where ADM involves personal data, GDPR may help; an array of information should be provided to those whose personal data is being processed 26 , including, in some cases, the existence of ADM and information about the logic involved 27 (the socalled 'right to an explanation' 28 ).”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as [28]
Performance metrics are one of the most value-laden parts of the machine learning process.
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [5]
Many AI audits focus on technical implementation of principles, are primarily quantitative, and do not include relevant stakeholders or consider use context or socioeconomic impacts.
“Many AI audits focus on technical implementation of principles; are primarily quantitative in nature; and do not include relevant stakeholders or consider use context or socioeconomic impacts [7].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [7]
Assumptions in the design of data processing pipelines can affect datasets, altering their content, structure, organization, or representation.
“Assumptions in the design of data processing pipelines can affect datasets, altering their content, structure, organization, or representation (Barocas and Selbst, 2016; Poirier, 2018).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
In user-based data collections, the selection criteria may include features held at a lower rate by members of certain groups, and may over or under-emphasize certain categories of users such as those that are highly-active on a target topic.
“the selection criteria may include features held at a lower rate by members of certain groups (Barocas and Selbst, 2016), and it may over or under-emphasize certain categories of users such as those that are highly-active on a target topic (Cohen and Ruths, 2013)”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
Scientists and journalists have pressed for greater scrutiny of the use of social data against possible ethical pitfalls, such as breaching users' privacy or enabling racial, socioeconomic or gender-based profiling.
“both scientists (Dwork and Mulligan, 2013; Barocas and Selbst, 2016) and journalists (Hill, 2014; Kirchner, 2015) have pressed for greater scrutiny of the use of social data against possible ethical pitfalls, such as breaching users privacy (Goroff, 2015), or enabling racial, socioeconomic or gender-based profiling (Barocas and Selbst, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
The reliance on automated decision making processes based on statistical methods can inherit, propagate, or even amplify the biases and prejudice present in the training data with respect to various factors such as race, age, gender or socioeconomic groups.
“The reliance on automated decision making processes based on statistical methods, can inherit, propagate, or even amplify the biases and prejudice present in the training data with respect to various factors such as race, age, gender or socioeconomic groups (Crawford and Schultz, 2014; Barocas and Selbst, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
Privacy breaches can have harmful outcomes like stalking, discrimination, black-mailing or identity theft.
“Privacy breaches can have harmful outcomes (Barocas and Selbst, 2016) like stalking, discrimination, black-mailing or identity theft (Gross and Acquisti, 2005).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Barocas and Selbst, 2016
An attribute may implicitly encode a protected factor, as occurs with postal code in credit rating.
“In fact, an attribute may implicitly encode a protected factor, as occurs with postal code in credit rating [392].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [392]
'Objective' big data analyses are likely to reproduce the biases in the data they learn from, thus violating the right to non-discrimination.
“Second, it is by now well-known that 'objective' big data analyses are likely to reproduce the biases in the data they learn from (thus violating the right to non-discrimination) [62].”
From AI for the Common Good?! Pitfalls, challenges, and ethics pen-testing (2019) · cites this paper as [62]
The prescribed audit under LL 144 is more accurately described as a 'disparate impact' audit, as it measures a specific form of algorithmic bias.
“The prescribed audit is more accurately described as a 'disparate impact' audit[3, 4, 20], as it measures a specific form of algorithmic bias [31].”
From Auditing Work: Exploring the New York City algorithmic bias audit regime (2024) · cites this paper as [4]
Algorithmic bias in hiring is real and largely traceable to data and design rather than malice.
“Taken together, this literature establishes two claims the remainder of the paper relies on: that algorithmic bias in hiring is real and largely traceable to data and design rather than malice, and that fixing it cannot be left to the technology alone.”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
Removing a protected attribute from a model does not remove bias; the model reconstructs the attribute from its correlates.
“Removing the protected attribute from the model therefore does not remove the bias; the model reconstructs the attribute from its correlates [2] [3].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
Antidiscrimination law generally holds that an employer cannot escape liability by outsourcing a discriminatory practice; if a selection procedure produces a disparate impact, the employer remains responsible even when an outside vendor designed it.
“Antidiscrimination law generally holds that an employer cannot escape liability by outsourcing a discriminatory practice; if a selection procedure produces a disparate impact, the employer remains responsible even when an outside vendor designed it [2].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
AI recruitment tools can reproduce and intensify historical discrimination because models trained on past hiring data learn patterns in that data, including patterns that track protected characteristics through seemingly neutral proxy variables.
“Yet a substantial body of scholarship shows that such systems can reproduce and even intensify historical discrimination, because models trained on past hiring data learn the patterns embedded in that data, including patterns that track protected characteristics through seemingly neutral proxy variables [2] [3].”
From Rejected by Code: Algorithmic Bias in AI-Driven Recruitment and the Case for HR-IT Governance Frameworks (2026) · cites this paper as [2]
Fairness and the legal standard of disparate impact are not purely statistical concepts, and involve ethical, political and constitutional concerns.
“fairness and the legal standard of disparate impact are not purely statistical concepts, and involve ethical, political and constitutional concerns15.”
From Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities (2020) · cites this paper as [15]
There is widespread agreement on the need for algorithmic fairness, particularly to mitigate the risks of direct and indirect discrimination due to algorithmic decisions.
“There is widespread agreement on the need for algorithmic fairness, particularly to mitigate the risks of direct and indirect discrimination (under US law, 'disparate treatment' and 'disparate impact', respectively) due to algorithmic decisions (Barocas and Selbst 2016;Grgić-Hlača et al. 2018;Green and Chen 2019).”
From The ethics of algorithms: key problems and solutions (2022) · cites this paper as Barocas and Selbst 2016
When sourcing, specific groups have been found excluded from viewing job ads based on age or gender.
“For instance, when sourcing, specific groups have been found excluded from viewing job ads based on age [10] or gender [12].”
From What does it mean to 'solve' the problem of discrimination in hiring?: social, technical and legal perspectives from the UK on automated hiring systems (2020) · cites this paper as [12]
Cites
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829 · see citing works
(2014) · Development Growth & Differentiation · doi:10.1111/dgd.12079 · Cited by 902 · see citing works
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · see citing works
Feature Extraction, Construction and Selection
(1998) · Springer eBooks · doi:10.1007/978-1-4615-5725-8 · Cited by 576 · see citing works
Gaydar: Facebook Friendships Expose Sexual Orientation
Jernigan, C., & Mistree, B. F. T. (2009) · First Monday · doi:10.5210/fm.v14i10.2611 · Cited by 222 · see citing works
Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures
Calders, T., & Žliobaitė, I. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_3 · Cited by 128 · see citing works
Implementing Anti-discrimination Policies in Statistical Profiling Models
Pope, D. G., & Sydnor, J. R. (2011) · American Economic Journal: Economic Policy · doi:10.1257/pol.3.3.206 · Cited by 87 · see citing works
Disparate Impact under Title VII: An Objective Theory of Discrimination
Rutherglen, G. A. (1987) · Virginia Law Review · doi:10.2307/1072940 · Cited by 44 · see citing works
Defining and minimizing the risk of complications during carotid artery interventions
Henry, M., Ohki, T., Polydorou, A., et al. (2004) · CRC Press eBooks · doi:10.3109/9780203490891-59 · Cited by 20 · see citing works
Foreword: Mental Health Law and the Movement toward Voluntary Treatment
Wexler, D. B. (1974) · California Law Review · doi:10.2307/3479745 · Cited by 10 · see citing works
Data as Representation: Beyond Anonymity In E-Research Ethics1
Carusi, A., & Uk, O. Q. (2014) · Cited by 6 · see citing works
City of Los Angeles Department of Water and Power v. Manhart, 435 U.S. 702 (1978)
Havard, C. J., & Thomas, T. A. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316411254.014 · Cited by 6 · see citing works
A New Cryoslide Technique for Cataract Removal
Croll, M., & Croll, L. J. (1966) · American Journal of Ophthalmology · doi:10.1016/0002-9394(66)91681-3 · Cited by 5 · see citing works
Primus, R. A. (2010) · bepress Legal Repository · Cited by 5 · see citing works
Rosenthal, L. (2013) · Chapman University Digital Commons (Chapman University) · Cited by 3 · see citing works
Space telescope duo to see in 3D
(2016) · The New Scientist · doi:10.1016/s0262-4079(16)32024-3 · Cited by 2 · see citing works
(2017) · UNP - Nebraska eBooks · doi:10.2307/j.ctt1d4v0kg.19 · Cited by 1 · see citing works
(1994) · Neurosurgical Review · doi:10.1007/bf00418434 · Cited by 1 · see citing works
N/A (2015) · Studies in Indian Politics · doi:10.1177/2321023015575206 · Cited by 1 · see citing works
(2014) · doi:10.5594/smpte.rdd25.2014 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
“The sources for bias, as indicated in [392] , can be traced to: • Skewed data: Bias within the data acquisition process.”
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611 · see citing works
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 3,432 · see citing works
“However, proxies for protected attributes are not easy to predict or detect (Romei and Ruggieri, 2014; Zarsky, 2016), particularly when algorithms access linked datasets (Barocas and Selbst, 2015).”
InfluentialKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · see citing works
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · see citing works
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · see citing works
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Privacy in the age of medical big data
Price, W. N., & Cohen, I. G. (2019) · Nature Medicine · doi:10.1038/s41591-018-0272-7 · Cited by 1,522 · see citing works
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · see citing works
A SWOT analysis of ChatGPT: Implications for educational practice and research
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2195846 · Cited by 1,384 · see citing works
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149 · see citing works
Gorwa, R., Binns, R., & Katzenbach, C. (2020) · Big Data & Society · doi:10.1177/2053951719897945 · Cited by 1,097 · see citing works
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043 · see citing works
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · see citing works
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · see citing works
“Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”
BackgroundBig Data Surveillance: The Case of Policing
Brayne, S. (2017) · American Sociological Review · doi:10.1177/0003122417725865 · Cited by 884 · see citing works
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845 · see citing works
Self-citationFerrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875 · see citing works
“Can be complex and require large amounts of additional data [32].”
BackgroundMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · see citing works
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
“…automated decision making processes based on statistical methods, can inherit, propagate, or even amplify the biases and prejudice present in the training data with respect to various factors such as race, age, gender or socioeconomic groups (Crawford and Schultz, 2014; Barocas and Selbst, 2016).”
BackgroundInfluentialEthical Implications and Accountability of Algorithms
Martin, K. (2019) · Journal of Business Ethics · doi:10.1007/s10551-018-3921-3 · Cited by 757 · see citing works
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · see citing works
Governing artificial intelligence: ethical, legal and technical opportunities and challenges
Cath, C. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0080 · Cited by 745 · see citing works
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · see citing works
(2019) · doi:10.1080/1369118x.2019.1573914 · Cited by 719 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Consumer-lending discrimination in the FinTech Era
Bartlett, R., Morse, A., Stanton, R., & Wallace, N. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.05.047 · Cited by 659 · see citing works
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687 · see citing works
Self-citationFairness of Exposure in Rankings
Singh, A., & Joachims, T. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220088 · Cited by 607 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598 · see citing works
Sex and gender analysis improves science and engineering
Tannenbaum, C., Ellis, R. P., Eyssel, F., Zou, J., & Schiebinger, L. L. (2019) · Nature · doi:10.1038/s41586-019-1657-6 · Cited by 587 · see citing works
There is a blind spot in AI research
Crawford, K., & Calo, R. (2016) · Nature · doi:10.1038/538311a · Cited by 580 · see citing works
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
“So models based on such variables can still negatively impact individuals or classes of individuals [74].”
BackgroundWhere fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535 · see citing works
A Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
Veale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512 · see citing works
Artificial intelligence, systemic risks, and sustainability
Galaz, V., Centeno, M. A., Callahan, P. W., Causevic, A., Patterson, T., Brass, I., Baum, S., Farber, D., Fischer, J., Garcia, D., et al. (2021) · Technology in Society · doi:10.1016/j.techsoc.2021.101741 · Cited by 497 · see citing works
Köchling, A., & Wehner, M. C. (2020) · Business Research · doi:10.1007/s40685-020-00134-w · Cited by 511 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
Artificial Intelligence in Society
OECD (2019) · doi:10.1787/eedfee77-en · Cited by 487 · see citing works
Algorithmic Bias in Autonomous Systems
Danks, D., & London, A. J. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/654 · Cited by 472 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · see citing works
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
BackgroundRegulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · see citing works
“Generally, it applies, in the US as well as the EU, in a technology-neutral way [4, 109-112, 113 ; but LGBTI, para.”
BackgroundDurán, J. M., & Jongsma, K. R. (2021) · Journal of Medical Ethics · doi:10.1136/medethics-2020-106820 · Cited by 446 · see citing works
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444 · see citing works
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · see citing works
Can fintech improve the efficiency of commercial banks? —An analysis based on big data
Wang, Y., Xiuping, S., & Zhang, Q. (2020) · Research in International Business and Finance · doi:10.1016/j.ribaf.2020.101338 · Cited by 459 · see citing works
SoK: Towards the Science of security and privacy in machine learning
Papernot, N., McDaniel, P. D., Sinha, A., & Wellman, M. P. (2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 427 · see citing works
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
Ethics of AI-enabled recruiting and selection: a review and research agenda
Hunkenschroer, A. L., & Luetge, C. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05049-6 · Cited by 447 · see citing works
Cheong, B. C. (2024) · Frontiers in Human Dynamics · doi:10.3389/fhumd.2024.1421273 · Cited by 445 · see citing works
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419 · see citing works
“Machine learning is playing a growing role in decision making on issues of broad societal importance (Munoz et al., 2016; Barocas and Selbst, 2016).”
BackgroundSituating methods in the magic of Big Data and AI
Elish, M. C., & boyd, D. (2017) · Communication Monographs · doi:10.1080/03637751.2017.1375130 · Cited by 414 · see citing works
Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420 · see citing works
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · see citing works
Fairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676 · see citing works
Algorithms in practice: Comparing web journalism and criminal justice
Christin, A. (2017) · Big Data & Society · doi:10.1177/2053951717718855 · Cited by 379 · see citing works
Discrimination in the Age of Algorithms
Kleinberg, J., Ludwig, J., Mullainathan, S., & Sunstein, C. R. (2018) · Journal of Legal Analysis · doi:10.1093/jla/laz001 · Cited by 401 · see citing works
Social Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373 · see citing works
The Intuitive Appeal of Explainable Machines
Selbst, A. D., & Barocas, S. (2018) · doi:10.2139/ssrn.3126971 · Cited by 392 · see citing works
Self-citationGender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381 · see citing works
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384 · see citing works
Artificial Intelligence and Journalism
Broussard, M., Diakopoulos, N. A., Guzman, A. L., Abebe, R., Dupagne, M., & Chuan, C. (2019) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699019859901 · Cited by 418 · see citing works
Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330691 · Cited by 382 · see citing works
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep LearningPreprint
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 460 · see citing works
van Noordt, C., & Misuraca, G. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101714 · Cited by 356 · see citing works
The Challenges of Algorithm-Based HR Decision-Making for Personal Integrity
Leicht-Deobald, U., Busch, T., Schank, C., Weibel, A., Schafheitle, S., Wildhaber, I., & Kasper, G. (2019) · Journal of Business Ethics · doi:10.1007/s10551-019-04204-w · Cited by 347 · see citing works
Algorithmic Prediction in Policing: Assumptions, Evaluation, and Accountability
Moses, L. B., & Chan, J. B. L. (2016) · Policing & Society · doi:10.1080/10439463.2016.1253695 · Cited by 416 · see citing works
Ten simple rules for responsible big data research
Zook, M., Barocas, S., boyd, D., Crawford, K., Keller, E., Gangadharan, S. P., Goodman, A. A., Hollander, R. D., Koenig, B. A., Metcalf, J., et al. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005399 · Cited by 358 · see citing works
Self-citationMeasuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348 · see citing works
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334 · see citing works
Katzenbach, C., & Ulbricht, L. (2019) · Internet Policy Review · doi:10.14763/2019.4.1424 · Cited by 359 · see citing works
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322 · see citing works
When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring
van den Broek, E., Sergeeva, A., & Huysman, M. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16559 · Cited by 330 · see citing works
The ethnographer and the algorithm: beyond the black box
Christin, A. (2020) · Theory and Society · doi:10.1007/s11186-020-09411-3 · Cited by 327 · see citing works
Inherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 347 · see citing works
Fourcade, M., & Healy, K. (2016) · Socio-Economic Review · doi:10.1093/ser/mww033 · Cited by 474 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
Predict and Surveil: Data, Discretion, and the Future of Policing
Brayne, S. (2020) · doi:10.1093/oso/9780190684099.001.0001 · Cited by 334 · see citing works
Bias and Discrimination in AI: A Cross-Disciplinary Perspective
Ferrer, X., Nuenen, T. V., Such, J. M., Cote, M., & Criado, N. (2021) · IEEE Technology and Society Magazine · doi:10.1109/mts.2021.3056293 · Cited by 308 · see citing works
“The Castaneda rule, which states that the number of people of the protected group selected from a relevant population cannot be smaller than 3 standard deviations the number expected in a random selection, is also used [21].”
BackgroundFairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316 · see citing works
Königstorfer, F., & Thalmann, S. (2020) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2020.100352 · Cited by 299 · see citing works
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 288 · see citing works
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · see citing works
Paulus, J. K., & Kent, D. M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0304-9 · Cited by 274 · see citing works
AI in education: learner choice and fundamental rights
Berendt, B., Littlejohn, A., & Blakemore, M. (2020) · Learning, Media and Technology · doi:10.1080/17439884.2020.1786399 · Cited by 281 · see citing works
Regan, P. M., & Jesse, J. (2019) · Ethics and Information Technology · doi:10.1007/s10676-018-9492-2 · Cited by 286 · see citing works
Technologies of Crime Prediction: The Reception of Algorithms in Policing and Criminal Courts
Brayne, S., & Christin, A. (2021) · Social Problems · doi:10.1093/socpro/spaa004 · Cited by 279 · see citing works
- Authors
- Barocas, S., & Selbst, A. D.
- Affiliations
- Cornell University (US)
- University of California, Los Angeles (US)
- Corresponding author
- Barocas, Solon
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2014
- Volume
- 104
- Issue
- 3
- Pages
- 671
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W3123374861
- Cited by
- 3,048
- References
- 20
- Citations by year
- 2026: 258 · 2025: 362 · 2024: 391 · 2023: 370 · 2022: 318 · 2021: 372 · 2020: 389 · 2019: 278 · 2018: 143 · 2017: 100 · 2016: 47 · 2015: 12 · 2014: 5 · 2013: 1 · 2012: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- bepress Legal Repository (submittedVersion)
- California Law Review
- UC Berkeley
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Law › Discrimination and Equality Law
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Keywords
- big data, algorithmic discrimination, indirect discrimination, data mining, Title VII, employment discrimination, anti-subordination, anti-discrimination law, predictive analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
