On measuring and mitigating biased inferences of word embeddings
Open accessAbstract
Word embeddings carry stereotypical connotations from the text they are trained on, which can lead to invalid inferences in downstream models that rely on them. We use this observation to design a mechanism for measuring stereotypes using the task of natural language inference. We demonstrate a reduction in invalid inferences via bias mitigation strategies on static word embeddings (GloVe). Further, we show that for gender bias, these techniques extend to contextualized embeddings when applied selectively only to the static components of contextualized embeddings (ELMo, BERT).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Natural language inference models may rely on misrepresentations or stereotypes to make invalid inferences.
“In predicting whether a premise entails or contradicts a hypothesis, a model may rely on misrepresentations or stereotypes to make invalid inferences.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dev et al. 2020
Bias NLI probes for bias through inference tasks.
“Bias NLI (Dev et al. 2020) alternatively probes for bias through inference tasks.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dev et al. 2020
Bias NLI instantiates the template 'the [subject] [verb] a/an [object]' with gender-occupation, nationality-polarity, and religion-polarity subject pairs.
“The dataset instantiates the template”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dev et al. 2020
Bias NLI presents sentence pairs that should not entail nor contradict each other, but may be predicted to imply one another if representational biases are present in a model.
“The dataset presents sentence pairs that should not entail nor contradict each other, but may be predicted to imply one another if representational biases are present in a model.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Dev et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1× · see citing works
“To discover biases, several lines of existing work (Bolukbasi et al. 2016; Caliskan, Bryson, and Narayanan 2017; Zhao et al. 2017; Dev and Phillips 2019) employ measurements intrinsic to the vector representations, which despite their utility, have two key problems.”
Method
Flagged first, then by citations
Research works printed but not linked: 17
- [2][2] Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP . -
- [3][3] Caliskan, A.; Bryson, J. J.; and Narayanan, A. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356(6334):183-186. -
- [4][4] Cheng, J.; Dong, L.; and Lapata, M. 2016. Long shortterm memory-networks for machine reading. In EMNLP . -
- [6][6] Dagan, I.; Glickman, O.; and Magnini, B. 2005. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges Workshop . Springer. -
- [7][7] Dev, S., and Phillips, J. 2019. Attenuating bias in word vectors. In AISTATS . -
- [8][8] Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT . -
- [10][10] Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018. Annotation artifacts in natural language inference data. In NAACL . -
- [11][11] Manzini, T.; Yao Chong, L.; Black, A. W.; and Tsvetkov, Y. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In NAACL . -
- [12][12] Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In NIPS . -
- [13][13] Parikh, A.; T¨ ackstr¨ om, O.; Das, D.; and Uszkoreit, J. 2016. A decomposable attention model for natural language inference. In EMNLP . -
- [14][14] Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP . -
- [16][16] Rudinger, R.; Naradowsky, J.; Leonard, B.; and Van Durme, B. 2018. Gender bias in coreference resolution. In NAACL , 8-14. -
- [17][17] Rudinger, R.; May, C.; and Van Durme, B. 2017. Social bias in Elicited Natural Language Inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing . -
- [18][18] Seo, M.; Kembhavi, A.; Farhadi, A.; and Hajishirzi, H. 2017. Bidirectional attention flow for machine comprehension. ICLR . -
- [19][19] Sweeney, C., and Najafian, M. 2019. A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings. In ACL . -
- [22][22] Zhao, J.; Zhou, Y.; Li, Z.; Wang, W.; and Chang, K.W. 2018. Learning gender-neutral word embeddings. In Proceedings of EMNLP 2018 , 4847-4853. Association for Computational Linguistics. -
- [23][23] Zhao, J.; Wang, T.; Yatskar, M.; Cotterell, R.; Ordonez, V.; and Chang, K.-W. 2019. Gender Bias in Contextualized Word Embeddings. In NAACL-HLT .
Cited by
Sorted by citations
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · see citing works
“…(2018); Zhang et al. (2018); Badjatiya et al. (2019); Bhargava and Forsyth (2019); Cao and Daumé (2019); Cho et al. (2019); Davidson et al. (2019); Dev et al. (2019); Garimella et al. (2019); Garg et al. (2019); Huang et al. (2019); James-Sorenson and AlvarezMelis (2019); Kaneko and Bollegala…”
BackgroundBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationSelf-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227 · see citing works
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
“They only care to show us the wheelchair”: disability representation in text-to-image AI models
Mack, K. A., Qadri, R., Denton, R., Kane, S. K., & Bennett, C. L. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642166 · Cited by 67 · see citing works
Explainable machine learning practices: opening another black box for reliable medical AI
Ratti, E., & Graves, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00141-z · Cited by 61 · see citing works
Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models
Ghanbarzadeh, S., Huang, Y., Palangi, H., Cruz Moreno, R., & Khanpour, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.336 · Cited by 17 · see citing works
Understanding Gender Bias in AI-Generated Product Descriptions
Kelly, M., Tahaei, M., Smyth, P., & Wilcox, L. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732169 · Cited by 4 · see citing works
- Authors
- Dev, S., Li, T., Phillips, J. M., & Srikumar, V.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 05
- Pages
- 7659-7666
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2969426522
- Cited by
- 14
- References
- 3
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 2 · 2022: 3 · 2021: 4 · 2020: 2
- FWCI
- 2.78
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1443046, 1350888, 1801446, 1816149, 1514520, 1822877, 1564287)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- word embeddings, stereotype bias, gender bias, natural language inference, bias mitigation, gloves, contextual embeddings, ELMo, BERT, static embeddings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
