Diagnosing Gender Bias in Image Recognition Systems
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Image recognition systems offer the promise to learn from images at scale without requiring expert knowledge. However, past research suggests that machine learning systems often produce biased output. In this article, we evaluate potential gender biases of commercial image recognition platforms using photographs of U.S. members of Congress and a large number of Twitter images posted by these politicians. Our crowdsourced validation shows that commercial image recognition systems can produce labels that are correct and biased at the same time as they selectively report a subset of many possible true labels. We find that images of women received three times more annotations related to physical appearance. Moreover, women in images are recognized at substantially lower rates in comparison with men. We discuss how encoded biases such as these affect the visibility of women, reinforce harmful gender stereotypes, and limit the validity of the insights that can be gathered from such data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“A system is a black box either if its technical design is sufficiently complex that human users cannot interpret the meaning of the inner workings or if the details of the system’s design and construction are hidden from users, for example, as corporate trade secrets (Rudin 2019).”
BackgroundWEST, C., & ZIMMERMAN, D. H. (1987) · Gender & Society · doi:10.1177/0891243287001002002 · Cited by 8,954 · cited 1×
“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript by others is a critical dimension of gender and a part of the interactional process of “doing gender” (West and Zimmerman 1987).”
BackgroundFiske, S. T., Cuddy, A. J. C., Glick, P., & Xu, J. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.82.6.878 · Cited by 7,364 · cited 1×
“Widely held and enduring gender beliefs characterize women as less agentic, less worthy, and less competent than men (Conway, Pizzamiglio, and Mount 1996; Fiske et al. 2002; Lueptow, Garovich-Szabo, and Lueptow 2001; Spence and Buckner 2000).”
BackgroundThe Managed Heart: Commercialization of Human Feeling
Hochschild, A. R. (2012) · doi:10.1525/9780520951853 · Cited by 3,236
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Unpacking the Gender System: A Theoretical Perspective on Gender Beliefs and Social Relations
Ridgeway, C. L., & Correll, S. J. (2004) · Gender & Society · doi:10.1177/0891243204265269 · Cited by 1,823
The Gender Revolution: Uneven and Stalled
England, P. (2010) · Gender & Society · doi:10.1177/0891243210361475 · Cited by 1,687 · cited 1×
“…deal of social science research has investigated the puzzling endurance of these gender stereotypes over time (Cotter, Hermsen and Vanneman 2011; England 2010; Lueptow, Garovich-Szabo, and Lueptow 2001): beliefs that are continually reinscribed “in new social forms of social and economic…”
BackgroundFriedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203 · cited 1×
“At the same time, recent research has shown algorithmic classification systems to be mechanisms for the reproduction, and even amplification, of more general social biases (Friedman and Nissenbaum 1996; Noble 2018).”
BackgroundAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Cited by
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
“…given the combined set of labels that Google Cloud Vision (GCV), Amazon Rekognition and Microsoft returned for all the images, similar to the biases Schwemmer et al. (2020) found in GCV systems, we found our system also disproportionately attached labels to do with hair and appearance in general…”
BackgroundMethodResultInfluentialFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“[97] Carsten Schwemmer, Carly Knight, Emily D.”
BackgroundDiversity and language technology: how language modeling bias causes epistemic injustice
Helm, P., Bella, G., Koch, G., & Giunchiglia, F. (2024) · Ethics and Information Technology · doi:10.1007/s10676-023-09742-6 · Cited by 110
Investigating Gender and Racial Biases in DALL-E Mini Images
Cheong, M., Abedin, E., Ferreira, M., Reimann, R., Chalson, S., Robinson, P., Byrne, J., Ruppanner, L., Alfano, M., & Klein, C. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3649883 · Cited by 57
“For reproducibility and to ensure faithfulness to the extant Craiyon app, we used the source code from the oicial DALL-E mini GitHub repository [25] (https://github.com/borisdayma/dalle-mini/blob/main/tools/inference/inference_pipeline.ipynb).”
MethodTaxonomizing and Measuring Representational Harms: A Look at Image Tagging
Katzman, J., Wang, A., Scheuerman, M., Blodgett, S. L., Laird, K., Wallach, H., & Barocas, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26670 · Cited by 32
Metadata
- Authors
- Schwemmer, C., Knight, C., Bello-Pardo, E. D., Oklobdzija, S., Schoonvelde, M., & Lockhart, J. W.
- Venue
- Socius: Sociological Research for a Dynamic World
- Publisher
- SAGE Publications
- Published
- 2020-01
- Volume
- 6
- ISSN
- 2378-0231 (ISSN-L)
- OpenAlex
- W3103585759
- Cited by
- 90
- References
- 16
- FWCI
- 16.01
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Diagnosing Gender Bias in Image Recognition Systems. https://app.cochat.ai/papers/10.1177/2378023120967171
- The paper
- Schwemmer, C., Knight, C., Bello-Pardo, E. D., Oklobdzija, S., Schoonvelde, M., & Lockhart, J. W. (2020). Diagnosing Gender Bias in Image Recognition Systems. Socius: Sociological Research for a Dynamic World. https://doi.org/10.1177/2378023120967171
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.