Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Research in machine learning fairness has historically considered a single binary demographic attribute; however, the reality is of course far more complicated. In this work, we grapple with questions that arise along three stages of the machine learning pipeline when incorporating intersectionality as multiple demographic attributes: (1) which demographic attributes to include as dataset labels, (2) how to handle the progressively smaller size of subgroups during model training, and (3) how to move beyond existing evaluation metrics when benchmarking model fairness for more subgroups. For each question, we provide thorough empirical evaluation on tabular datasets derived from the US Census, and present constructive recommendations for the machine learning community. First, we advocate for supplementing domain knowledge with empirical validation when choosing which demographic attribute labels to train on, while always evaluating on the full set of demographic attributes. Second, we warn against using data imbalance techniques without considering their normative implications and suggest an alternative using the structure in the data. Third, we introduce new evaluation metrics which are more appropriate for the intersectional setting. Overall, we provide substantive suggestions on three necessary (albeit not sufficient!) considerations when incorporating intersectionality into machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Black Feminist Thought: Knowledge, Consciousness, and the Politics of Empowerment.
Brewer, R. M., & Collins, P. H. (1992) · Contemporary Sociology · doi:10.2307/2074808 · Cited by 19,445
The Complexity of Intersectionality
McCall, L. (2005) · Signs: Journal of Women in Culture and Society · doi:10.1086/426800 · Cited by 6,682 · cited 2×
“1 There is a long history of considering intersectional harms in fields outside of computer science [9, 22–24, 52, 80, 90, 92, 98, 112, 115], and an urgent need to do so in machine learning 1 There are complex nuances to this conceptualization that are out of scope of this work [98].”
BackgroundResultDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · cited 1×
“One field in which synthetic examples are being widely adopted, and where results are relatively accepted as indicative of real-world gains largely due to work on transfer learning (e.g., Sim2Real [66]), is reinforcement learning for robotics [82].”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 3×
“However, for the scope of our work since we consider the positive labels to be more desirable, we choose a metric analogous to equal opportunity [58], i.e., equalizing the true positive rate (TPR).”
BackgroundResultFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Gender: An Intersectionality Perspective
Shields, S. A. (2008) · Sex Roles · doi:10.1007/s11199-008-9501-8 · Cited by 1,509 · cited 1×
“1 There is a long history of considering intersectional harms in fields outside of computer science [9, 22–24, 52, 80, 90, 92, 98, 112, 115], and an urgent need to do so in machine learning 1 There are complex nuances to this conceptualization that are out of scope of this work [98].”
BackgroundStatistical Theories of Discrimination in Labor Markets
Aigner, D. J., & Cain, G. G. (1977) · ILR Review · doi:10.1177/001979397703000204 · Cited by 1,503
A Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
“The Combahee River Collective Statement” (1977)
Collective, C. R. (2017) · Available Means · doi:10.2307/j.ctt5hjqnj.50 · Cited by 1,020
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Algorithmic injustice: A relational ethics approach
Birhane, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100205 · Cited by 707 · cited 1×
“We also note that despite the language we employ, we do not suggest that fairness can be treated as a purely algorithmic problem that neglects the sociotechnical frame [12, 50, 51, 109].”
BackgroundAlgorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · cited 1×
“1 aimed to address is that underrepresentation can be a problem if the minority group is differently predictive from the majority group [85], but does not contain sufficient samples to train a robust model on.”
BackgroundA comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582
Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 469 · cited 8×
“To generalize the equal opportunity measure to more than two groups, we adopt a method similar to prior work [40, 77, 121].”
BackgroundMethodInfluentialRace as a Bundle of Sticks: Designs that Estimate Effects of Seemingly Immutable Characteristics
Sen, M., & Wasow, O. (2016) · Annual Review of Political Science · doi:10.1146/annurev-polisci-032015-010015 · Cited by 422
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · cited 1×
“8 The foundation of this problem is that categorizing people into discrete, socially constructed groups, while tenuous, is often necessary for machine learning systems to make sense of socially relevant distinctions [35, 56, 67].”
BackgroundDecision Theory for Discrimination-Aware Classification
Kamiran, F., Karim, A., & Zhang, X. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.45 · Cited by 300
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 1×
“8 The foundation of this problem is that categorizing people into discrete, socially constructed groups, while tenuous, is often necessary for machine learning systems to make sense of socially relevant distinctions [35, 56, 67].”
BackgroundScheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252
Lessons from archives: strategies for collecting sociocultural data in machine learning
Jo, E. S., & Gebru, T. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372829 · Cited by 240
An Intersectional Definition of Fairness
Foulds, J. R., Islam, R., Keya, K. N., & Pan, S. (2020) · 2020 IEEE 36th International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00203 · Cited by 233 · cited 9×
“To generalize the equal opportunity measure to more than two groups, we adopt a method similar to prior work [40, 77, 121].”
BackgroundMethodInfluentialDegenerate Feedback Loops in Recommender Systems
Jiang, R., Chiappa, S., Lattimore, T., György, A., & Kohli, P. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314288 · Cited by 232 · cited 1×
“…for problems with intersectional concerns may be more akin to that of simulations as testbeds and tools for understanding sociotechical systems such as in modeling recommender systems [17, 21, 45, 70, 89] and online information diffusion [44, 48, 119], rather than as a source for more data.”
BackgroundHow algorithmic popularity bias hinders or promotes quality
Ciampaglia, G. L., Nematzadeh, A., Menczer, F., & Flammini, A. (2018) · Scientific Reports · doi:10.1038/s41598-018-34203-2 · Cited by 204 · cited 1×
“that of simulations as testbeds and tools for understanding sociotechical systems such as in modeling recommender systems [17, 21, 45, 70, 89] and online information diffusion [44, 48, 119], rather than as a source for more data.”
MethodMultiaccuracy: Black-Box Post-Processing for Fairness in Classification
Kim, M. P., Ghorbani, A., & Zou, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314287 · Cited by 193 · cited 1×
“Morina et al. [96] propose a post-processing approach that generalizes that of Hardt et al. [58], and Kim et al. [79] similarly propose a post-processing approach as well as auditing procedure.”
ResultGuo, W., & Caliskan, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462536 · Cited by 180 · cited 1×
“They are commonly formulated in terms of the maximum difference (or ratio) of a performance metric either between one group to that of all [53, 77, 121], or between two groups [40, 46].”
BackgroundAlgorithmic realism: expanding the boundaries of algorithmic thought
Green, B., & Viljoen, S. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372840 · Cited by 161
Identifying and Correcting Label Bias in Machine Learning
Jiang, H., & Nachum, O. (2019) · arXiv · doi:10.48550/arxiv.1901.04966 · Cited by 152
Racial categories in machine learning
Benthall, S., & Haynes, B. D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287575 · Cited by 110
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72 · cited 1×
“Reporting disaggregated analyses for all subgroups would of course alleviate many of these problems, and should be done before deployment [8], but for iterating on model training can be unwieldy.”
BackgroundCreel, K., & Hellman, D. (2022) · Canadian Journal of Philosophy · doi:10.1017/can.2022.3 · Cited by 67
The Profiling Potential of Computer Vision and the Challenge of Computational Empiricism
Goldenfein, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287568 · Cited by 21
FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14
The Dangers of Counterfactual Causal Thinking about Detecting Racial Discrimination
Kohler-Hausmann, I. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3050650 · Cited by 9
Cited by
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“…244], an intersectional approach draws attention to how harms play out for people whose lives are shaped by interlocking forms of oppression [46, 229] – for aim(s)tocapturethemor-phological sexual differences between male and female faces by comparing their shape differences to a defined face…”
BackgroundPalmer, Z. B., & Oswal, S. K. (2025) · Journal of Business and Technical Communication · doi:10.1177/10506519241280644 · Cited by 22
Data Agency Theory: A Precise Theory of Justice for AI Applications
Ajmani, L., Stapleton, L., Houtti, M., & Chancellor, S. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658930 · Cited by 11
“Recent work in AI has focused heavily on expanding our evaluation metrics beyond quantitative ones, such as accuracy, and towards human-centered concepts, such as bias, fairness, and equality [14, 87].”
BackgroundThe Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics
Orlikowski, M., Röttger, P., Cimiano, P., & Hovy, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.88 · Cited by 9
“In contrast, social scientists have long adopted the idea of intersectionality (Crenshaw, 1989), which also informs research on fairness in machine learning (Wang et al., 2022).”
BackgroundCo-Producing AI: Toward an Augmented, Participatory Lifecycle
Mushkani, R., Berard, H., Ammar, T., Chatonnier, C., & Koseki, S. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i2.36674 · Cited by 5
Metadata
- Authors
- Wang, A., Ramaswamy, V. V., & Russakovsky, O.
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 336-349
- OpenAlex
- W4283166422
- Cited by
- 87
- References
- 37
- FWCI
- 18.23
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.