Towards a critical race methodology in algorithmic fairness
Open accessThe DOI matches its Crossref record; 121 of the 123 references it cites were found in a registry or an index; 15 of the 131 sources its claims cite were checked for support.
Abstract
We examine the way race and racial categories are adopted in algorithmic fairness frameworks. Current methodologies fail to adequately account for the socially constructed nature of race, instead adopting a conceptualization of race as a fixed attribute. Treating race as an attribute, rather than a structural, institutional, and relational phenomenon, can serve to minimize the structural aspects of algorithmic unfairness. In this work, we focus on the history of racial categories and turn to critical race theory and sociological work on race and ethnicity to ground conceptualizations of race for fairness research, drawing on lessons from public health, biomedical research, and social survey research. We argue that algorithmic fairness researchers need to take into account the multidimensionality of race, take seriously the processes of conceptualizing and operationalizing race, focus on social processes which produce racial inequality, and consider perspectives of those most affected by sociotechnical systems.
Overview
CoChat Summary
TL;DR
This paper argues that algorithmic fairness research treats race too much like a fixed attribute, when it is better understood as socially constructed, multidimensional, and tied to histories of racial classification and oppression. The authors call for researchers to center operationalization choices, focus on racism rather than race alone, and consider the perspectives of affected communities.
Key findings
- The paper identifies a recurring problem in fairness work: racial categories are often used as if they are “single dimensional” and interchangeable, which the authors say erases social, economic, and political complexity and can reify race.
- In the COMPAS case, the authors note that the original ProPublica story had “some 700 citations” on Google Scholar, but the underlying Broward County racial schema included categories such as Black, White, Hispanic, Asian, Native American, and Other, while omitting “Native Hawaiian or Other Pacific Islander” and treating “Hispanic” as a race rather than an ethnicity.
- The authors argue that race is multidimensional, giving examples including racial identity, self-classification, observed race, reflected race, and phenotype; they say different dimensions are associated with different outcomes and measurements.
- They recommend disaggregated analysis and transparent reporting of definitions, categories, measurements, and motivations, and they say multiple measures of race should be collected if possible.
Methods
- This is a conceptual and critical review paper, not an empirical study.
- The authors synthesize histories of racial classification and scientific racism, and draw lessons from public health, biomedical research, and social survey research.
- They review algorithmic fairness literature, especially group fairness and disaggregated analysis, to articulate implications for how race should be conceptualized and operationalized.
Limitations
- The paper does not present original empirical data or experiments; this is evident from the text, which is a review/argument piece rather than a study with a sample or quantitative results.
- The authors explicitly note that category choices are often constrained by available census or bureaucratic taxonomies, which is a practical limitation for the approaches they recommend.
- Not stated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 Valid · 16 Not valid · 54 Not checked · 85 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While an area of major concern for critical race and technology scholarship, the use of racial categories in algorithmic fairness research (i.e. the research community which has emerged around venues like FA T* and AIES) has largely gone unquestioned.”
What the sources say
- Not matchedcites [22] [22] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In FAT*, pages 77-91, 2018. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [12] [12] Sebastian Benthall and Bruce D. Haynes. Racial categories in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, 2019. - source
The introduction (p. 1) says that comparatively little attention is given to how protected-class labels are assigned and what their use means for fair machine-learning design. The discussion (p. 9) likewise describes group membership as typically treated as a simple fact. These passages support the claim that the field has largely left its use of racial categories unquestioned. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In fact, the widespread uncritical adoption of racial categories, standardized or not, can erode awareness of the social and political histories of racial taxonomies and reify racial categories as natural kinds [19, 30, 42, 43, 78, 100, 111].”
What the sources say
- Supportedcites [19] [19] Lundy Braun, Anne Fausto-Sterling, Duana Fullwiley, Evelynn Hammonds, Alondra Nelson, William Quivers, Susan Reverby, and Alexandra E Shields. Racial categories in medical practice: How useful are they? PLoS medicine, 4:e271, 10 2007. - source
In “From Census Categories to Research Plans,” the paper says poorly defined racial categories became reified through widespread use of census categories, and that routine use makes those categories seem almost natural. “Racial Categories Are Historical, Not Natural” explains their historically specific, often hidden social and political meanings. Together, these passages support the citing sentence’s paraphrase. (LLM, conf=0.91)
- Not matchedcites [30] [30] Troy Duster. Race and reification in science. Science, 307(5712):1050-1051, 2005. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [42] [42] Duana Fullwiley. The molecularization of race: Institutionalizing human difference in pharmacogenetics practice. Science as Culture, 16(1):1-30, 2007. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [43] [43] Duana Fullwiley. The biologistical construction of race. Social studies of science, 38:695-735, 11 2008. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [78] [78] Amade M'charek. Beyond fact or fiction: On the materiality of race in practice. Cultural Anthropology, 28:420-442, 07 2013. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [100] [100] Phia Salter and Glenn Adams. Toward a critical race psychology. Social and Personality Psychology Compass, 7, 11 2013. - sourceExperimental — accuracy not yet measured
The Synergistic foundations section cautions against essentializing identity or reifying problematic categories, supporting the concern about treating racial categories as fixed. The paper does not specifically say that widespread adoption of racial categories, standardized or otherwise, erodes awareness of the social and political histories of racial taxonomies or makes those categories appear to be natural kinds. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [111] [111] Andrew Smart, Richard Tutton, Paul Martin, George T.H. Ellison, and Richard Ashcroft. The standardization of race and ethnicity in biomedical science editorials and uk biobanks. Social Studies of Science, 38(3):407-423, 2008. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When race is utilized as a descriptive category, the choices about what categories to use and how to assign individuals to categories significantly impacts the results of analysis [18, 102, 114, 115].”
What the sources say
- Partly supportedcites [18] [18] Jenifer Bratter and Bridget K Gorman. Does multiracial matter? a study of racial disparities in self-rated health. Demography, 48:127-52, 02 2011. - sourceExperimental — accuracy not yet measured
The Self-Rated Health Disparities Using the Best-Race Identity section (page unavailable) finds that keeping multiracial respondents in separate categories produces a substantially better-fitting model than assigning them to single-race categories. But it also reports few differences in disparity patterns and only slight changes in odds ratios. The source therefore supports the importance of classification choices for this health analysis, not the unqualified claim that they significantly change analytical results generally. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [102] [102] Aliya Saperstein. Double-checking the race box: Examining inconsistency between survey measures of observed and self-reported race. Social Forces, 85(1):57-74, 2006. - source
- Supportedcites [114] [114] Edward Telles, René D. Flores, and Fernando Urrea-Giraldo. Pigmentocracies: Educational inequality, skin color and census ethnoracial identification in eight latin american countries. Research in Social Stratification and Mobility, 40:39 - 58, 2015. - source
The paper states that choosing census ethnoracial categories is consequential for counts and inequality estimates. In §9, Conclusions, it also reports that estimates based on skin color were more consistent and robust than those based on ethnoracial identification, supporting the claim that how people are categorized affects analytical results. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [115] [115] Edward E. Telles and Nelson Lim. Does it matter who answers the race question? racial classification and income inequality in brazil. Demography, 35(4):465-474, 1998. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Frameworks for algorithmic audit studies [22, 92] and standardized model reporting guidelines [81] have highlighted the importance of reporting model statistics disaggregated by groups defined along cultural, demographic, or phenotypic lines.”
What the sources say
- Not matchedcites [22] [22] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In FAT*, pages 77-91, 2018. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [92] [92] Inioluwa Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In AIES, 2019. - source
- Supportedcites [81] [81] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In FAT, 2019. - source
The citation to this paper supports the standardized model-reporting-guidelines part of the sentence. The Introduction (p. 2) advocates breaking down quantitative model-performance results by cultural, demographic, and phenotypic groups, and Quantitative Analyses (p. 6) says results should be disaggregated. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These categories look familiar - they are nearly identical to the way that the US Census defines race.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
63 more claims
Create a free account to see all 168 claims and to verify the 7 references still open.
- It may state two separate claims as one. 46
- Probably not a substantial claim. 78
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The call for a global lens in AI accountability is not new [84,155], but the ethical principles in AI are often interpreted, prioritised, contextualised, and implemented differently across the globe [98].”
From Re-imagining Algorithmic Fairness in India and Beyond (2021) · cites this paper as [84]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The labeling of groups may grant legitimacy to these boundaries, define relational differences between groups, and reinforce social hierarchies and power imbalances, often with very real and material consequences that can segregate, marginalize, and oppress (Beukeboom and Burgers 2019; Hanna et al. 2020).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Hanna et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Delineating between social groups is often required to assess disparities, yet can simultaneously legitimize social constructions, reinforce power differentials, and enable systems of oppression (Hanna et al. 2020).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Hanna et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These strategies tend to ignore or conceal distinct mechanisms of oppression that operate differently for each social group (Hanna et al. 2020).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Hanna et al. 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We highlight that social groups are often socially constructed, a form of classification with delineations that are not static and may be contested (Hanna et al. 2020).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Hanna et al. 2020
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed
Fukuyama, F., & Scott, J. C. (1998) · Foreign Affairs · doi:10.2307/20048980 · Cited by 8,396
American Apartheid: Segregation and the Making of the Underclass.
Fernandez, R. M., Massey, D. S., & Denton, N. A. (1993) · Contemporary Sociology · doi:10.2307/2074502 · Cited by 6,193
Sorting Things Out: Classification and Its Consequences
Bowker, G. C., & Star, S. L. (1999) · doi:10.7551/mitpress/6352.001.0001 · Cited by 6,188
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“There has also been significant debate about the appropriate use of racial variables in studies aimed at identifying causal effects [46, 54, 55, 58, 67].”
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Toward a Critical Race Theory of Education
Ladson-Billings, G., & Tate, W. F. (1995) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146819509700104 · Cited by 3,881 · cited 1×
“"that all difference is both analogous and equivalent" [68].”
Background(2018) · Cited by 3,510
Racial Formation in the United States
Omi, M., & Winant, H. (2014) · doi:10.4324/9780203076804 · Cited by 3,510 · cited 3×
“Projects that "misrecognize" race as natural [27] are driven by the hegemonic nature of racial projects [86].”
BackgroundInfluentialThe Social Construction of What?
Dupre, J., & Hacking, I. (2000) · The Journal of Philosophy · doi:10.2307/2678463 · Cited by 3,444 · cited 1×
“As Hacking [48] has said, social constructivism is the act of "making up people.”
BackgroundFoucault, M. (2005) · doi:10.4324/9780203996645 · Cited by 3,281
Research works printed but not linked: 17
- Julia Angwin and Jeff Larson. Propublica responds to company's critique of machine bias story. ProPublica, 29, 2016.
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, 2016.
- Geoffrey C. Bowker and Susan Leigh Star. Sorting Things Out. MIT Press, 2000.
- Patricia Hill Collins. Fighting words: Black women and the search for justice, volume 7. U of Minnesota Press, 1998.
- William Dieterich, Christina Mendoza, and Tim Brennan. COMPAS risk scales: Demonstrating accuracy equity and predictive parity. Northpoint Inc, 2016.
- Dave Elder-Vass. Towards a realist social constructionism. Sociologia, problemas e práticas, (70):9--24, 2012.
- Diana Elliott, Rob Santos, Steven Martin, and Charmaine Runes. Assessing miscounts in the 2020 census. 2019.
- Duana Fullwiley. The biologistical construction of race. Social studies of science, 38:695--735, 11 2008.
- Michael Herzfeld. The social production of indifference. University of Chicago Press 1993.
- Akil Kokayi Khalfani, Tukufu Zuberi, Sulaiman Bah, and Pali J Lehohla. Race and population statistics in south africa. In Tukufu Zuberi and Eduardo Bonilla-Silva, editors, White logic, white methods: racism and methodology, chapter 4, pages 63--92. Rowman & Littlefield Publishers, 2008.
- Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. How we analyzed the compas recidivism algorithm. ProPublica, 9, 2016.
- Adam Liptak. Supreme court leaves census question on citizenship in doubt. The New York Times, 2019.
- Katherine McKittrick. Demonic Grounds: Black Women and the Cartographies of Struggle. University of Minnesota Press, ned - new edition edition, 2006.
- Michael A Omi. The changing meaning of race. America becoming: Racial trends and their consequences, 1:243--263, 2001.
- Ramya Rajagopalan and Joan H Fujimura. Making history via dna, making dna from history. Genetics and the unsettled past: The collision of DNA, race, and history, page 143, 2012.
- Rashida Richardson, Jason Schultz, and Kate Crawford. Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. New York University Law Review, 2019.
- Blaise Agüera y Arcas, Margaret Mitchell, and Alexander Todorov. Physiognomy's new clothes. Medium, 2017.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
“to their different social positions (Hanna et al., 2020)—particularly groups at the intersections of multiple axes of oppression—we recommend that researchers and practitioners center work analyzing “bias” in NLP systems around the lived”
BackgroundDecolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912
“Benjamin, 2019; Buolamwini and Gebru, 2018; Lum and Isaac, 2016; Noble, 2018; Eubanks, 2018), exploitative or extractive data practices (Gray and Suri, 2019), and applications of AI that dispossess the identity and resources of vulnerable populations (Green, 2019; Keyes, 2018; Hanna et al., 2019; Stark, 2019).”
BackgroundBaker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“As feminist theory (D’ignazio & Klein, 2020), sociocultural theory (Ferrero & Barujel, 2019), and critical race theory (Hanna et al., 2020) begin to influence and improve data science, we need the ideas from these communities in AIED research as well.”
BackgroundAnil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
“Delineating between social groups is often required to assess disparities, yet can simultaneously legitimize social constructions, reinforce power differentials, and enable systems of oppression (Hanna et al., 2020).”
BackgroundMethodInfluentialMunn, L. (2023) · AI and Ethics · doi:10.1007/s43681-022-00209-w · Cited by 355
“These are complex questions which have shifted substantially over time and which intersect with race, gender, and culture [29, 58].”
BackgroundSociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“When such classification relies on phenotypes, this dimension of representational harm essentializes historically contingent identities [85] through which classification systems entrench and produce meaning about what they represent [100, 111].”
BackgroundSelf-citationJacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
“However, there are important questions about the legitimacy of treating attributes like race as variables subject to intervention (e.g., Kohler-Hausmann, 2018; Hanna et al., 2020), and Kilbertus et al.”
BackgroundRe-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228
“However these categories are culturally and historically situated [85].”
BackgroundScheuerman, M. K., Wade, K., Lustig, C., & Brubaker, J. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392866 · Cited by 193
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
An empirical characterization of fair machine learning for clinical risk prediction
Pfohl, S. R., Foryciarz, A., & Shah, N. H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103621 · Cited by 164
“Age Group Race and Ethnicity Cohort construction Model training and evaluation Hospital Mortality Prolonged Length of Stay Attribute: Sex Female Male Attribute: Race and Ethnicity Asian Black Hispanic Other White Attribute: Age Group [18-30) [30-45) [45-55) [55-65) [65-75) [75-90) 30-Day Readmission Performance Metric…”
BackgroundDo Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
Scheuerman, M. K., Hanna, A., & Denton, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476058 · Cited by 216
“…design frameworks which have explicitly embraced specific politics and values [8, 24, 65, 81] Particularly with its focus on real-world applications, the artifacts produced by machine learning researchers and practitioners — for example, in the form of datasets — are value-laden [51, 98, 107].”
Self-citationA sociotechnical view of algorithmic fairness
Dolata, M., Feuerriegel, S., & Schwabe, G. (2022) · Information Systems Journal · doi:10.1111/isj.12370 · Cited by 160
“does not work properly for them, this impacts on their dignity and belongs to the interactional justice dimension (Celis Bueno, 2020; Hanna et al., 2020; Robert et al., 2020).”
BackgroundSystemic racial disparities in funding rates at the National Science Foundation
Chen, C. Y., Kahanamoku, S. S., Tripati, A., Alegado, R. A., Morris, V. R., Andrade, K., & Hosbey, J. (2022) · eLife · doi:10.7554/elife.83071 · Cited by 127
"I wouldn't say offensive but...": Disability-Centered Perspectives on Large Language Models
Gadiraju, V., Kane, S., Dev, S., Taylor, A., Wang, D., Denton, R., & Brewer, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3593989 · Cited by 155
“Propelled by research on AI and algorithmic fairness for gender and race [37, 45], scholars have started to study the impacts of AI bias for people with disabilities [34, 74].”
Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415186 · Cited by 149
“Therefore, they are also culturally and historically specific [46].”
BackgroundDavis, J. L., Williams, A., & Yang, M. W. (2021) · Big Data & Society · doi:10.1177/20539517211044808 · Cited by 143
“These include proposals for critical race methodologies for algorithmic fairness (Hanna et al., 2020), critical race theories applied to human – computer interaction (Ogbonnaya-Ogburu et al., 2020), decolonial AI (Mohamed et al., 2020), decolonial computer science (Birhane and Guest), computing for…”
MethodDiversity, Equity, and Inclusion in Artificial Intelligence: An Evaluation of Guidelines
Cachat-Rosset, G., & Klarsfeld, A. (2023) · Applied Artificial Intelligence · doi:10.1080/08839514.2023.2176618 · Cited by 138
“Hanna et al. (2020, 2) also argue that AI fairness research does not sufficiently consider how the social group categories, operationalized in AI guidelines, are socially constructed, resulting in the “widespread use of racial categories as if they represent natural and objective differences…”
BackgroundA survey on bias in visual datasets
Fabbrizzi, S., Papadopoulos, S., Ntoutsi, E., & Kompatsiaris, I. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103552 · Cited by 137
“Furthermore, the use of fuzzy categories such as race or gender poses major challenge to algorithmic fairness from both the ontological and operational points of view [32, 42].”
AI ageism: a critical roadmap for studying age discrimination and exclusion in digitalized societies
Stypinska, J. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01553-5 · Cited by 133
“Thus, the vast majority of algorithmic fairness frameworks are stipulated with reference to particular social groups, often requiring a formal encoding of the groups into the dataset and/or algorithm (Hanna et al. 2020).”
BackgroundBennett, C. L., Gleason, C., Scheuerman, M. K., Bigham, J. P., Guo, A., & To, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445498 · Cited by 128
Rethinking Fairness: An Interdisciplinary Survey of Critiques of Hegemonic ML Fairness Approaches
Weinberg, L. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13196 · Cited by 127
“Hanna et al. (2020) also argue that algorithmic fairness research does not adequately account for the ways that social group categories, which AI fairness research operationalizes, are socially constructed, resulting in the "widespread use of racial categories as if they represent natural and…”
BackgroundStudying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
“Tracing the links to historical and ongoing asymmetries can be helpful to understand how data comes to be [27] and what kind of political work ML systems perform [45].”
BackgroundAlgorithmic Bias in EducationPreprint
Baker, R. S., & Hawn, A. (2021) · doi:10.35542/osf.io/pbmvz · Cited by 117
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111
“Work in this area has addressed race [13, 35], gender [34, 43, 72], and disability [11], calling into question what it means to even rely on these categories as a basis for assessing unfairness, and what harms are reproduced by relying on these infrastructures of categorization.”
Background“Fairness for unobserved characteristics: insights from technological impacts on queer communities”.
Tomasev, N., McKee, K. R., Kay, J., & Mohamed, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462540 · Cited by 70
“Additionally, the definition of racial and ethnic groups has fluctuated across time and place (Hanna et al. 2020).”
BackgroundHagendorff, T. (2022) · AI and Ethics · doi:10.1007/s43681-021-00122-8 · Cited by 103
“Hence, it may be a problem to formally encode and treat them as fixed instead of relational, social constructs or phenomena [43].”
BackgroundInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
Conservative AI and social inequality: conceptualizing alternatives to bias through social theory
Zajko, M. (2021) · AI & SOCIETY · doi:10.1007/s00146-021-01153-9 · Cited by 83
“This theoretical background is crucial to understanding how AI technologies are implicated in the ‘tech-to-prison pipeline’, and how certain forms of AI can only strengthen these structures and their oppressive effects (Coalition for Critical Technology 2020; Hanna et al. 2019).”
BackgroundWang, A., Ramaswamy, V. V., & Russakovsky, O. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533101 · Cited by 87
“8 The foundation of this problem is that categorizing people into discrete, socially constructed groups, while tenuous, is often necessary for machine learning systems to make sense of socially relevant distinctions [35, 56, 67].”
BackgroundRecommenders with a Mission: Assessing Diversity in News Recommendations
Vrijenhoek, S., Kaya, M., Metoui, N., Möller, J. E., Odijk, D., & Helberger, N. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446019 · Cited by 137
“[18] argue that race and ethnicity are strongly social constructs that should not be treated as objective differences between groups.”
AI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
Qadri, R., Shelby, R., Bennett, C. L., & Denton, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594016 · Cited by 85
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 83
“[29], we are actualizing a particular static conceptualization of observed race here.”
BackgroundDocumenting Computer Vision Datasets: An Invitation to Reflexive Data Practices
Miceli, M., Yang, T., Naudts, L., Schuessler, M., Serbanescu, D., & Hanna, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445880 · Cited by 82
Self-citationTowards Measuring Fairness in AI: The Casual Conversations Dataset
Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A., & Ferrer, C. C. (2022) · IEEE Transactions on Biometrics, Behavior, and Identity Science · doi:10.1109/tbiom.2021.3132237 · Cited by 79
“On the other hand, Hanna et al. [36] emphasizes that race is socially constructed and in algorithmic fairness methodologies ML practitioners should be mindful of the dimensions of race that are relevant for particular contexts.”
BackgroundGeiger, R. S., Cope, D., Ip, J., Lotosh, M., Shah, A., Weng, J., & Tang, R. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00144 · Cited by 141
“…al identify as “abstraction traps” ( Selbst et al., 2019 , p.60), such as failing to account for the particularities of different kinds and qualities of discrimination in a given social context — a critique Hanna et al ( 2020 ) make of fairness research that treats race as a single fixed attribute.”
BackgroundA Human-Centered Systematic Literature Review of Cyberbullying Detection Algorithms
Kim, S., Razi, A., Stringhini, G., Wisniewski, P. J., & De Choudhury, M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476066 · Cited by 79
“Scholars in the evolving and emergent area of humancentered machine learning have therefore argued that machine learning needs to stay grounded in human needs [22], models need to be built in inclusive ways that adequately represent the diverse experiences of different individuals and minimize biases [122], and that m…”
BackgroundREVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Wang, A., Liu, A., Zhang, R., Kleiman, A., Kim, L., Zhao, D., Shirai, I., Narayanan, A., & Russakovsky, O. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01625-5 · Cited by 77
“…dynamic and non-discrete concepts such as gender and using skin tone as a proxy for race, we reify certain conceptions of these concepts (Hanna et al., 2020; Jacobs and Wallach, 2021) that harm certain groups, e.g., non-binary individuals (Hamidi et al., 2018; Scheuerman et al., 2020).”
BackgroundMethodLam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 77
“Group-based fairness approaches tend to abstract away the complexities and hierarchies of identity (especially for categories like race that are socially constructed and contested) and tend to treat oppressed groups as interchangeable [29].”
BackgroundChoi, D., Lee, S., Kim, S.-I., Lee, K., Yoo, H. J., Lee, S., & Hong, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641989 · Cited by 78
“Propelled by research examining AI bias related to gender and race [9, 34, 42, 63, 72], recent years have seen growing attention to AI’s inclusivity toward identity-based minority groups, such as people with disabilities [38–40, 51, 96, 99, 101, 109] and LGBTQ communities [27, 82, 103].”
BackgroundA Survey of Race, Racism, and Anti-Racism in NLP
Field, A., Blodgett, S. L., Waseem, Z., & Tsvetkov, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.149 · Cited by 74
“…and can refer to a variety of different perspectives, including racial identity (how you self-identify), observed race (the race others perceive you to be), and reflected race (the race you believe others perceive you to be) (Roth, 2016; Hanna et al., 2020; Ogbonnaya-Ogburu et al., 2020).”
BackgroundInfluentialDesigning Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72
What's Fair About Individual Fairness?
Fleisher, W. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462621 · Cited by 141
Harrington, C., & Egede, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580719 · Cited by 108
Yee, K., Tantipongpipat, U., & Mishra, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479594 · Cited by 59
“The use of standardized racial and gender taxonomy here is not meant to essentialize or naturalize the construction of race and gender, or to imply that race is solely dependent on ancestry, but rather to study the impact of representational harm on historically marginalized populations [41, 61].”
BackgroundInfluentialFairness Through Robustness: Investigating Robustness Disparity in Deep Learning
Nanda, V., Dooley, S. W., Singla, S., Feizi, S., & Dickerson, J. P. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445910 · Cited by 119
“We know this to be reductive and serves to minimize the contextualization within which race and gender derive their meaning [6, 21].”
BackgroundEpistemic Injustice in Generative AI
Kay, J., Kasirzadeh, A., & Mohamed, S. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31671 · Cited by 76
Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58
“This can help differentiate, for example, whether biases in themodel derive from cultural norms embedded in the population that make up the training data or from under-representation of individuals with certain skin tones in the training data (or both) (Benthall and Haynes, 2019; Hanna et al., 2019).”
BackgroundThe Causal Fairness Field Guide: Perspectives From Social and Formal Sciences
Carey, A. N., & Wu, X. (2022) · AI and Ethics · doi:10.3389/fdata.2022.892837 · Cited by 135
“However, it is well studied that these categories are unstable, contingent, and rooted in racial inequality [79].”
BackgroundInfluentialFairCLIP: Harnessing Fairness in Vision-Language Learning
Luo, Y., Shi, M. Y., Khan, M. O., Afzal, M. M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01168 · Cited by 157
“In the field of image processing, biases exist in facial recognition [7, 11, 18, 78, 79] and pedestrian detection algorithms [9, 33, 38].”
BackgroundBad Seeds: Evaluating Lexical Methods for Bias Measurement
Antoniak, M., & Mimno, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.148 · Cited by 50
Razi, A., Alsoubai, A., Kim, S., Naher, N., Ali, S., Stringhini, G., De Choudhury, M., & Wisniewski, P. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503569 · Cited by 53
“porates human-centered design and transparency for the sake of explaining usages in real-life scenarios as well as any potential to cause harm [11, 14, 20].”
BackgroundRobots Enact Malignant Stereotypes
Hundt, A., Agnew, W. S., Zeng, V., Kacianka, S., & Gombolay, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533138 · Cited by 104
“[40] and Diversity and Inclusion Metrics [67] cover algorithmic fairness in the handling and sampling of human data.”
BackgroundMethodInfluentialRepresentativeness in Statistics, Politics, and Machine Learning
Chasalow, K., & Levy, K. E. C. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445872 · Cited by 89
“1 We see this effort as a complement to other work in FAccT that draws on fields like STS to interrogate taken-for-granted terms and categories [70, 76].”
MethodLaufer, B., Jain, S., Cooper, A. F., Kleinberg, J. M., & Heidari, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533107 · Cited by 88
“Notably, our results indicate that, separate from the well-documented fact that there are numerous, competing definitions of fairness [8, 19, 37, 41, 46, 49], there are also clear thematic differences in how FAccT papers examine fairness.”
ResultOne Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision
Khan, Z., & Fu, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445920 · Cited by 46
“In [18], Hanna, et.al call attention to the multidimensionality of race and the need to critically assess and justify the choice of racial categories and schemes.”
BackgroundMethodAndrus, M., & Villeneuve, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533226 · Cited by 51
“Dimensions of racial identity include self-identity (the race an individual self-identifies as), self-classification (the racial category an individual identifies with on an official form), observed race (the race others believe you to be), appearance-based (observed race based on readily observable characteristics),…”
BackgroundInfluentialTransparency of medical artificial intelligence systems
Kim, C., Gadgil, S. U., & Lee, S. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00363-w · Cited by 81
“in which demographic variables are defined and derived can influence which populations are made visible in analyses 50–54 .”
Envisioning Equitable Speech Technologies for Black Older Adults
Brewer, R. N., Harrington, C., & Heldreth, C. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594005 · Cited by 42
Racial Bias within Face Recognition: A Survey
Yücer, Ş., Tektas, F., Moubayed, N. A., & Breckon, T. P. (2024) · ACM Computing Surveys · doi:10.1145/3705295 · Cited by 81
“However, such definitions and classifications were adopted by the national census infrastructure across many jurisdictions [58].”
InfluentialCampedelli, G. M. (2021) · Journal of Computational Social Science · doi:10.1007/s42001-020-00082-9 · Cited by 39
“In parallel with the vivid debate on themes such as superintelligence and existential risk [39], research groups, policy-makers, and activists are pushing towards the definition of guidelines and the improvements of current practices to make AI safer and more ethical [40, 41, 42, 43].”
BackgroundUncurated Image-Text Datasets: Shedding Light on Demographic Bias
Garcia, N., Hirota, Y., Wu, Y., & Nakashima, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00672 · Cited by 43
“Moreover, the definition of some attributes, such as the ones related to race or ethnicity, is ambiguous and subjective [20].”
BackgroundReducing Uncertainty and Offering Comfort: Designing Technology for Coping with Interpersonal Racism
To, A., Carey, H., Kaufman, G. F., & Hammer, J. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445590 · Cited by 42
“, understanding racialized notions of algorithmic fairness [53], exploring how algorithms interpret and construct identity [88], etc.”
BackgroundExamining Identity as a Variable of Health Technology Research for Older Adults: A Systematic Review
Harrington, C., Martin-Hammond, A., & Bray, K. E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517621 · Cited by 38
“We have seen how omission of certain groups can be particularly harmful in other areas of HCI research such as facial recognition or algorithm bias [102, 207].”
BackgroundCoMPosT: Characterizing and Evaluating Caricature in LLM Simulations
Cheng, M., Piccardi, T., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.669 · Cited by 195
Ethical and methodological challenges in building morally informed AI systems
Hagendorff, T., & Danks, D. (2023) · AI and Ethics · doi:10.1007/s43681-022-00188-y · Cited by 36
“More recent work in critical race theory, critical algorithms studies, and related fields has argued that the multidimen-sionality of these concepts means that we need alternative ways to operationalize demographic categories [35].”
BackgroundThe Datafication of #MeToo: Whiteness, Racial Capitalism, and Anti-Violence Technologies
Henne, K., Shelby, R. M., & Harb, J. I. (2021) · Big Data & Society · doi:10.1177/20539517211055898 · Cited by 45
Gender and Racial Bias in Visual Question Answering Datasets
Hirota, Y., Nakashima, Y., & García, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533184 · Cited by 67
Excavating awareness and power in data science: A manifesto for trustworthy pervasive data research
Shilton, K., Moss, E., Gilbert, S., Bietz, M. J., Fiesler, C. L., Metcalf, J., Vitak, J., & Zimmer, M. T. (2021) · Big Data & Society · doi:10.1177/20539517211040759 · Cited by 50
“However, there are numerous helpful frameworks for thinking through issues of power adapted specifically for digital data research, including anti-essentialism (Neyland, 2016), feminism (D’Ignazio and Klein, 2020; Hoffman and Jonas, 2017), anti-racism (Benjamin, 2019; Hanna et al., 2020), anti-colonialism (Dourish and…”
MethodFairness in Recommender Systems
Ekstrand, M. D., Das, A., Burke, R., & Díaz, F. (2012) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_18 · Cited by 60
Rismani, S., Shelby, R., Smart, A., Jatho, E., Kroll, J., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 33
Self-citationIt's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness BenchmarksPreprint
Bao, M., Zhou, A., Zottola, S. A., Brubach, B., Desmarais, S. L., Horowitz, A., Lum, K., & Venkatasubramanian, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05498 · Cited by 136
“[39] further discusses challenges and inconsistencies associated with demographic data.”
BackgroundTaxonomizing and Measuring Representational Harms: A Look at Image Tagging
Katzman, J., Wang, A., Scheuerman, M., Blodgett, S. L., Laird, K., Wallach, H., & Barocas, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26670 · Cited by 32
Exploring Approaches to Data Literacy Through a Critical Race Theory Perspective
Johnson, B., Rydal Shapiro, B., DiSalvo, B., Rothschild, A., & DiSalvo, C. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445141 · Cited by 31
Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Peng, K., Mathur, A., & Narayanan, A. (2021) · arXiv · doi:10.48550/arxiv.2108.02922 · Cited by 31
“These concerns include the privacy of individuals included [45, 70], representational harms introduced by annotations [25, 44], effects of biases on downstream use [20, 21, 18], and use for ethically dubious purposes [45, 75, 66].”
BackgroundSahlgren, O. (2023) · Learning, Media and Technology · doi:10.1080/17439884.2021.1986065 · Cited by 30
Inductive Risk, Understanding, and Opaque Machine Learning Models
Sullivan, E. (2022) · Philosophy of Science · doi:10.1017/psa.2022.62 · Cited by 29
“The practice of race labeling involves social/ political considerations and has a history of marginalization and injustice (Browne 2015; Hanna et al. 2020; James and Burgos 2022).”
Backgroundvan Esch, P., Cui, Y. G., & Heilgenberg, K. (2024) · Australasian Marketing Journal (AMJ) · doi:10.1177/14413582241244504 · Cited by 39
“…in AI fairness research, scholars claim that prior work, unfortunately, overlooked the socially constructed nature of DEI in existing AI guidelines, thereby resulting in the disproportional dominance in treating differences in limited categories as an equal proxy of DEI (Hanna et al., 2020).”
When Fair Ranking Meets Uncertain Inference
Ghosh, A., Dutt, R., & Wilson, C. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462850 · Cited by 58
Competing narratives in AI ethics: a defense of sociotechnical pragmatism
Watson, D., Mökander, J., & Floridi, L. (2024) · AI & Society · doi:10.1007/s00146-024-02128-2 · Cited by 56
“The problem formulation itself—in terms of variables and averages, optimizing metrics for prespecified groups— fails to question how the categories that algorithmic fairness seeks to protect are themselves contested, socially constructed, and reductive (Hanna et al. 2020; Cantwell Smith 2019).”
Racial bias in AI-generated images
Yang, Y. (2025) · AI & Society · doi:10.1007/s00146-025-02282-1 · Cited by 50
“We should acknowledge that there are still risks of treating race as a visual characteristic, i.e., using visible information about subjects to determine race (Hanna et al. 2020; Khan and Fu 2021; Scheuerman et al. 2020).”
Mhasawade, V., & Chunara, R. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462587 · Cited by 43
“Related to this decomposition of effects, recent literature in critical race theory, sociology, epidemiology and algorithmic fairness have discussed similar concerns with the conceptualization of race as a fixed, individual-level attribute [8, 21, 49, 57].”
BackgroundMethodInfluentialKlumbytė, G., Draude, C., & Taylor, A. S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533207 · Cited by 46
“The FAccT community in particular has showcased research that brings together approaches, concepts and methods from social sciences and humanities together with ML research and design, or that explicitly starts from humanities and social sciences perspectives when analyzing implications of ML systems and proposing new…”
BackgroundCritical Race Theory: Exploring Its Application to Public Administration
Riccucci, N. M. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009122986 · Cited by 31
Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14886 · Cited by 25
“Therefore, they are also culturally and historically specific [46].”
BackgroundRobustness Disparities in Commercial Face DetectionPreprint
Dooley, S. W., Goldstein, T., & Dickerson, J. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12508 · Cited by 40
There is an elephant in the room: towards a critique on the use of fairness in biometrics
Valdivia, A., Corbera-Serrajòrdia, J., & Swianiewicz, A. (2022) · AI and Ethics · doi:10.1007/s43681-022-00249-2 · Cited by 37
“As other critical scholars pointed out: ‘[t]reating race as an attribute, rather than a structural, institutional, and relational phenomenon, can serve to minimise the structural aspects of algorithmic unfairness’ [49].”
BackgroundCase Study: Deontological Ethics in NLP
Prabhumoye, S., Boldt, B., Salakhutdinov, R., & Black, A. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.297 · Cited by 44
“A large body of work (Bolukbasi et al., 2016; Sun et al., 2019; Zhao et al., 2019, 2017; Sap et al., 2019; Hanna et al., 2020; Davidson et al., 2019) directs its efforts to mitigate bias in data, represen-”
BackgroundBeyond model interpretability: socio-structural explanations in machine learning
Smart, A., & Kasirzadeh, A. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02056-1 · Cited by 25
“However, geographic disparities affect job availability and commute times, disproportionately impacting women with childbearing responsibilities.”
Self-citationBeyond Skin Tone: A Multidimensional Measure of Apparent Skin Color
Thong, W. E., Joniak, P., & Xiang, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00452 · Cited by 49
“Still, the Fitzpatrick skin type has deviated from its original usage as up to fifty percent of dermatologists use the Fitzpatrick types to describe race and ethnicity [72], and doing so can be harmful as such simplification ignores their complexity [30].”
BackgroundFairness indicators for systematic assessments of visual feature extractors
Goyal, P., Soriano, A. R., Hazirbas, C., Sagun, L., & Usunier, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533074 · Cited by 24
“Although computer vision systems have recently yielded astonishing results, several societal issues have come to the surface as their use materializes in areas like face recognition, self-driving cars and other commercial applications [9, 19, 28, 56, 80].”
BackgroundNaoum, R. F., Szakadáti, T., & Balogh, G. (2026) · Management Review Quarterly · doi:10.1007/s11301-025-00580-y · Cited by 98
Perfect Density Models Cannot Guarantee Anomaly Detection
Lan, C. L., & Dinh, L. (2021) · Entropy · doi:10.3390/e23121690 · Cited by 58
“way: semantically close inputs can be considered distinct or identical depending on arbitrary choices of discretization/categorization [90], resulting in different probability values.”
Schelenz, L. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450614.3463293 · Cited by 37
“2020 [31] have argued that the widespread understanding of race as an individual descriptor rather than a system impedes fairness efforts in machine learning.”
Dirty Data Labeled Dirt Cheap: Epistemic Injustice in Machine Learning Systems
Hull, G. (2022) · Ethics and Information Technology · doi:10.1007/s10676-023-09712-y · Cited by 35
“Attempts to measure fairness and bias work particularly poorly for categories such as race, where the attribute in question is not fixed, but has meaning only in specific cultural contexts that can be obscured by objective-seeming proxies like census data (Hanna et al., 2020).”
BackgroundCan Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
Weerts, H., Pfisterer, F., Feurer, M., Eggensperger, K., Bergman, E., Awad, N. H., Vanschoren, J., Pechenizkiy, M., Bischl, B., & Hutter, F. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14747 · Cited by 53
“Moreover, commonly studied sensitive characteristics, such as race and gender, are social constructs that are complex to measure (Hanna et al., 2020; Jacobs & Wallach, 2021); this is further complicated when we consider intersectionality (Chen et al., 2019; Buolamwini & Gebru, 2018) and privacy…”
BackgroundEvaluating the Impact of Social Determinants on Health Prediction in the Intensive Care Unit
Yang, M. Y., Kwak, G. H., Pollard, T., Celi, L. A., & Ghassemi, M. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604719 · Cited by 33
Metadata
- Authors
- Hanna, A., Denton, R., Smart, A., & Smith-Loud, J.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 501-512
- OpenAlex
- W2992319600
- Semantic Scholar
- 6e7e095f46deb297713dcde05991faf635768d29
- Cited by
- 295
- References
- 123
- Citations by year
- 2026: 30 · 2025: 38 · 2024: 44 · 2023: 49 · 2022: 54 · 2021: 62 · 2020: 14
- FWCI
- 46.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Anthropology › Race, Genetics, and Society
- Social Sciences › Sociology and Political Science › Critical Race Theory in Education
- Keywords
- critical race theory, algorithmic fairness, racial classification, social construction of race, racial inequality, socio-technical systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.