Do neural ranking models intensify gender bias?
Open accessAbstract
Concerns regarding the footprint of societal biases in information retrieval (IR) systems have been raised in several previous studies. In this work, we examine various recent IR models from the perspective of the degree of gender bias in their retrieval results. To this end, we first provide a bias measurement framework which includes two metrics to quantify the degree of the unbalanced presence of gender-related concepts in a given IR model's ranking list. To examine IR models by means of the framework, we create a dataset of non-gendered queries, selected by human annotators. Applying these queries to the MS MARCO Passage retrieval collection, we then measure the gender bias of a BM25 model and several recent neural ranking models. The results show that while all models are strongly biased toward male, the neural models, and in particular the ones based on contextualized embedding models, significantly intensify gender bias. Our experiments also show an overall increase in the gender bias of neural models when they exploit transfer learning, namely when they use (already biased) pre-trained embeddings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Information retrieval models may return more documents with masculine-related concepts than feminine ones despite using non-gendered queries.
“Retrieved documents may exhibit similar exclusionary norms as machine translation models, returning more documents with masculinerelated concepts than feminine ones, despite using non-gendered queries (Rekabsaz and Schedl 2020).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Rekabsaz and Schedl 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Nogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615 · see citing works
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · see citing works
Baeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432 · cited 1× · see citing works
“The existence of various types of bias in IR, and the concerns regarding its implications on IR applications, have been pointed out in several previous studies [1, 2, 5, 11].”
ResultInvestigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- [2][2] Asia J Biega, Krishna P Gummadi, and Gerhard Weikum. 2018. Equity of attention: Amortizing individual fairness in rankings. In Proc. of SIGIR . -
- [3][3] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proc. of NeurIPS . -
- [6][6] Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proc. of WSDM . -
- [8][8] Yanai Elazar and Yoav Goldberg. 2018. Adversarial Removal of Demographic Attributes from Text Data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . -
- [9][9] Sebastian Hofstätter, Navid Rekabsaz, Carsten Eickhoff, and Allan Hanbury. 2019. On the Effect of Low-Frequency Terms on Neural-IR Models. In Proc. of SIGIR . -
- [10][10] Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2017. PACRR: A Position-Aware Neural IR Model for Relevance Matching. In Proc. of EMNLP . -
- [12][12] Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, M. Bilal Zafar, S. Ghosh, K. P Gummadi, and K. Karahalios. 2017. Quantifying search bias: Investigating sources of bias for political searches in social media. In Proc. of CSCW . -
- [13][13] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016). - · arXiv
- [15][15] Jahna Otterbacher, Jo Bates, and Paul Clough. 2017. Competent men and warm women: Gender stereotypes and backlash in image search results. In Proc. of CHI . -
- [16][16] Jahna Otterbacher, Alessandro Checco, Gianluca Demartini, and Paul Clough. 2018. Investigating user perception of gender bias in image search: the role of sexism. In Proc. of SIGIR . -
- [17][17] Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Proc. of AAAI . -
- [18][18] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proc. of EMNLP . -
- [21][21] Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proc. of SIGIR . -
- [22][22] Ke Yang and Julia Stoyanovich. 2017. Measuring fairness in ranked outputs. In Proc. of Conference on Scientific and Statistical Database Management . -
- [23][23] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, V. Ordonez, and K. Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proc. of NAACL-HTL .
Cited by
Sorted by citations
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Unlearning Protected User Attributes in Recommendations with Adversarial Training
Ganhör, C., Penz, D., Rekabsaz, N., Lesota, O., & Schedl, M. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531820 · Cited by 47 · see citing works
Self-citation
- Authors
- Rekabsaz, N., & Schedl, M.
- Affiliations
- Johannes Kepler University of Linz (AT)
- Venue
- Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2020-07-25
- Pages
- 2065-2068
- OpenAlex
- W3035680038
- Cited by
- 64
- References
- 5
- Citations by year
- 2026: 9 · 2025: 8 · 2024: 8 · 2023: 13 · 2022: 17 · 2021: 7 · 2020: 2
- FWCI
- 7.29
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- gender bias, neural ranking models, information retrieval, BM25, contextual embeddings, transfer learning, pre-trained embeddings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
