Gender stereotype reinforcement: Measuring the gender bias conveyed by ranking algorithms
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Abstract Search Engines (SE) have been shown to perpetuate well-known gender stereotypes identified in psychology literature and to influence users accordingly. Similar biases were found encoded in Word Embeddings (WEs) learned from large online corpora. In this context, we propose the Gender Stereotype Reinforcement (GSR) measure, which quantifies the tendency of a SE to support gender stereotypes, leveraging gender-related information encoded in WEs. Through the critical lens of construct validity, we validate the proposed measure on synthetic and real collections. Subsequently, we use GSR to compare widely-used Information Retrieval ranking algorithms, including lexical, semantic, and neural models. We check if and how ranking algorithms based on WEs inherit the biases of the underlying embeddings. We also consider the most common debiasing approaches for WEs proposed in the literature and test their impact in terms of GSR and common performance measures. To the best of our knowledge, GSR is the first specifically tailored measure for IR, capable of quantifying representational harms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841 · cited 4×
“A widelyadopted evaluation measure, based on this user model, is the Discounted Cumulative Gain [43], which weighs documents according to a coefficient that decreases with rank in a logarithmic fashion.”
BackgroundMethodInfluentialBag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1607.01759 · Cited by 4,542
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Where do you know what you know? The representation of semantic knowledge in the human brain
Patterson, K., Nestor, P. J., & Rogers, T. T. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2277 · Cited by 2,706 · cited 2×
“This definition is based on a view of culture, social constructs and stereotypes as networks of concepts [32, 66] and implicit associations [3, 23, 35].”
BackgroundFilter Bubbles, Echo Chambers, and Online News Consumption
Flaxman, S., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw006 · Cited by 1,992
Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.69.6.1013 · Cited by 1,793 · cited 3×
“This definition is based on a view of culture, social constructs and stereotypes as networks of concepts [32, 66] and implicit associations [3, 23, 35].”
BackgroundGoogle Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips
Sparrow, B., Liu, J., & Wegner, D. M. (2011) · Science · doi:10.1126/science.1207745 · Cited by 1,762 · cited 1×
“Constant availability of information has shaped expectations and cognitive processes of SE users [77].”
BackgroundMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
Eagly, A. H., Nater, C., Miller, D. I., Kaufmann, M., & Sczesny, S. (2020) · American Psychologist · doi:10.1037/amp0000494 · Cited by 1,136
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · cited 7×
“Interestingly, Word Embeddings can be used to detect gender-related biases in the corpus they have been trained on [18, 29].”
BackgroundMethodInfluentialAn overview of online fake news: Characterization, detection, and discussion
Zhang, X., & Ghorbani, A. A. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.03.004 · Cited by 997 · cited 1×
“User interfaces also play a key role: responsible augmentation of search results may be important to convey information about fact checking [88] and controversy [90]; query auto-completion can lead to problematic results [58], while panels which summarize results for users within the SE may reduce click-through rates…”
BackgroundPatterns of gender development
Martin, C. L., & Ruble, D. N. (2010) · Annual Review of Psychology · doi:10.1146/annurev.psych.093008.100511 · Cited by 600 · cited 1×
“Women and men are particularly salient categories, recognizable since an early age, and available for stereotypical association with traits, behaviors and events [50].”
BackgroundWhat is a memory schema? A historical perspective on current neuroscience literature
Ghosh, V. E., & Gilboa, A. (2014) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.11.010 · Cited by 529
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · cited 7×
“Within the context of gender stereotypes in SE, the only user study we are aware of centers on image retrieval [46], while our proposed measure deals with textual data.”
BackgroundInfluentialThe CRANFIELD TESTS ON INDEX LANGUAGE DEVICES
CLEVERDON, C. (1967) · Aslib Proceedings · doi:10.1108/eb050097 · Cited by 509
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496
Gonen, H., & Goldberg, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1061 · Cited by 454 · cited 3×
“This family of debiasing techniques seem limited and imperfect [33], with gender information redundantly encoded along multiple dimensions, and thus hard to eradicate.”
BackgroundInfluentialMeasuring Fairness in Ranked Outputs
Yang, K., & Stoyanovich, J. (2017) · Proceedings of the 29th International Conference on Scientific and Statistical Database Management · doi:10.1145/3085504.3085526 · Cited by 378
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · cited 6×
“the phenomenon we want to study, from its subsequent operationalization, which details how the phenomenon can be measured from observed data [42, 52].”
BackgroundMethodInfluentialBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · cited 3×
“Additional problematic depictions of men and women have been identified in these WEs, including sexist analogies (such as woman− man ' midwife− doctor ' whore − coward [6]) and representation of jobs skewed with respect to gender, in ways that reflect current gender gaps in the US workforce [17, 29, 70].”
BackgroundEquity of Attention: Amortizing Individual Fairness in Rankings
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210063 · Cited by 337
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
Accurately Interpreting Clickthrough Data as Implicit Feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2017) · ACM SIGIR Forum · doi:10.1145/3130332.3130334 · Cited by 212 · cited 1×
“As generally known, users seldom dabble into result pages beyond the first one, and the likelihood of view decreases with document rank [24, 43, 44].”
BackgroundInvestigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · cited 3×
“[11], search results in resume SEs, which happen to be biased with respect to gender, may lead to a dual harm: an immediate one, for the providers of CVs, competing to appear in the current search, and a long-term one, for the perception and future decisions of recruiters.”
BackgroundMethodA Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2019) · arXiv · doi:10.48550/arxiv.1901.10002 · Cited by 82 · cited 2×
“This is not surprising; it is due to a combination of language specificity and historical bias [78] potentially present in news coverage, as discussed in Section 3.”
BackgroundFairness in representation: quantifying stereotyping as a representational harm
Abbasi, M., Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2019) · Proceedings of the 2019 SIAM International Conference on Data Mining · doi:10.1137/1.9781611975673.90 · Cited by 44 · cited 1×
“representational, likely to take place when individuals and groups are unable to self-determine their image, which may end up being stereotyped, inadequate or offensive [1, 58, 62].”
Background
Cited by
Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey
Fabris, A., Baranowska, N., Dennis, M. J., Graus, D., Hacker, P., Saldivar, J., Zuiderveen Borgesius, F., & Biega, A. J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3696457 · Cited by 117
Self-citationGrep-BiasIR: A dataset for investigating gender representation bias in information retrieval results
Krieg, K., Parada-Cabaleiro, E., Medicus, G., Lesota, O., Schedl, M., & Rekabsaz, N. (2023) · Proceedings of the 2023 Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578295 · Cited by 31
Metadata
- Authors
- Fabris, A., Purpura, A., Silvello, G., & Susto, G. A.
- Venue
- Information Processing & Management
- Publisher
- Elsevier BV
- Published
- 2020-11
- Volume
- 57
- Issue
- 6
- Pages
- 102377
- ISSN
- 0020-0271 (ISSN-L)
- 0306-4573
- OpenAlex
- W3081529310
- Cited by
- 52
- References
- 29
- FWCI
- 9.80
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.