Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle
Open accessThe DOI matches its Crossref record; 329 of the 329 references it cites were found in a registry or an index.
Abstract
Correctly resolving textual mentions of people fundamentally entails making inferences about those people. Such inferences raise the risk of systematic biases in coreference resolution systems, including biases that can harm binary and non-binary trans and cis stakeholders. To better understand such biases, we foreground nuanced conceptualizations of gender from sociology and sociolinguistics, and investigate where in the machine learning pipeline such biases can enter a coreference resolution system. We inspect many existing data sets for trans-exclusionary biases, and develop two new data sets for interrogating bias in both crowd annotations and in existing coreference resolution systems. Through these studies, conducted on English text, we confirm that without acknowledging and building systems that recognize the complexity of gender, we will build systems that fail for: quality of service, stereotyping, and over- or under-representation, especially for binary and non-binary trans users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“and processes of creating resources and technologies may further entrench such disparities [25, 38, 144].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [25]
Citations
Cites
Gender Trouble: Feminism and the Subversion of Identity
McIntosh, M. (1991) · Feminist Review · doi:10.1057/fr.1991.33 · Cited by 29,379 · cited 3×
“Following (Butler 1989), in our annotation we considered drag performance as a form of genderfluidity; as such, we annotate the performer and the drag persona as coreferent with each other (as well as the relevant pronouns), akin to how we believed a reasonable model for handling stage names (e.g.,…”
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“In the case of machine translation, one can ask whether standard evaluation metrics like Bleu (Papineni et al. 2002) are sensitive to misgendering.”
BackgroundInfluentialWEST, C., & ZIMMERMAN, D. H. (1987) · Gender & Society · doi:10.1177/0891243287001002002 · Cited by 8,954
Gender trouble: feminism and the subversion of identity
(1990) · Choice Reviews Online · doi:10.5860/choice.28-1264 · Cited by 7,707
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
(2018) · Cited by 3,510
Lakoff, R. (1973) · Language in Society · doi:10.1017/s0047404500000051 · Cited by 3,138
(2005) · Cited by 3,125
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
The World Atlas of Language Structures Online
Dryer, M. S., Haspelmath, M., & Forkel, R. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7135714 · Cited by 2,542
Research works printed but not linked: 18
- Andrews, 2017, The singular, gender-neutral ‘they’ added to the Associated Press Stylebook
- Bagga, 1998, Algorithms for scoring coreference chains, 563
- Barocas, 2017, The problem with bias: Allocative versus representational harms in machine learning
- Bender, 2019, A typology of ethical risks in language technology with an eye towards where transparent documentation can help, The Future of Artificial Intelligence: Language, Ethics, Technology
- Blodgett, 2020, Language (technology) is power: The need to be explicit about NLP harms
- Burton, 1883, Kama Sutra, Translation
- Conrod, 2018, Changes in singular they
- Conrod, 2018, What does it mean to agree? Coreference with singular they
- Corbett, 2013, Number of genders
- Craig, 1994, Classifier languages, The Encyclopedia of Language and Linguistics, 2, 565
- Dryer, 2013, Expression of pronominal subjects
- HaNasi, 189, Mishnah bikkurim
- Mitchell, 2005, Ace 2004 multilingual training corpus, Linguistic Data Consortium, Philadelphia, 1, 1
- Ochs, 1992, Indexing gender, Rethinking Context: Language as an Interactive Phenomenon, 11, 335
- Orita, 2015, Why discourse affects speakers’ choice of referring expressions, 10.3115/v1/P15-1158 · DOI
- Silverstein, 1979, Language structure and linguistic ideology, The Elements: A Parasession on Linguistic Units and Levels, 193
- Stryker, 2008, Transgender History
- Weischedel, 2011, OntoNotes: A Large Training Corpus for Enhanced Processing
Cited by
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies
Dev, S., Monajatipoor, M., Ovalle, A., Subramonian, A., Phillips, J., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.150 · Cited by 123
Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis
Mohammad, S. M. (2022) · Computational Linguistics · doi:10.1162/coli_a_00433 · Cited by 107
“See Cao and Daumé (2021) for a useful discussion on linguistic vs. social gender and also for a great example to create more inclusive data for research.”
BackgroundLee, J., Wang, J., Brown, E., Chu, L., Rodríguez, S., & Froehlich, J. E. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642230 · Cited by 166
“Below, we describe GazePointAR’s design and implementation, starting with a taxonomy of pronoun usage drawn from linguistics literature.”
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Li, X. L., Kuncoro, A., Hoffmann, J., d’Autume, C. D. M., Blunsom, P., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.812 · Cited by 133
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
“Unsurprisingly, gendered and racial disparities have been documented in a number of language technologies [37, 79, 141], and processes of creating resources and technologies may further entrench such disparities [25, 38, 144].”
BackgroundOvalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., & Gupta, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594078 · Cited by 43
“They may be unable to consistently recall them to a subject in text [14].”
BackgroundSteed, R., Panda, S., Kobren, A., & Wick, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.247 · Cited by 42
“…of fine-tuned LLMs, including discriminatory medical diagnoses (Zhang et al., 2020), over-reliance on binary gender for coreference resolution (Cao and Daumé, 2021), the re-inforcement of traditional gender roles in part-of-speech tagging (Garimella et al., 2019), toxic text generation…”
BackgroundFairness in Language Models Beyond English: Gaps and Challenges
Ramesh, K., Sitaram, S., & Choudhury, M. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.157 · Cited by 36
GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models
Yin, D., Bansal, H., Monajatipoor, M., Li, L. H., & Chang, K.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.132 · Cited by 123
Queiroga, E. M., Machado, M. F. B., Rodés, V., Primo, T. T., & Cechinel, C. (2022) · Information · doi:10.3390/info13090401 · Cited by 54
A domain-specific language for describing machine learning datasets
Giner-Miguelez, J., Gómez, A., & Cabot, J. (2023) · Journal of Computer Languages · doi:10.1016/j.cola.2023.101209 · Cited by 45
“For example, creators can dis-recommend specific applications due to the potential social impact of the data, as [5] does regarding gender research.”
BackgroundMethodAn Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models
Xie, Z., & Lukasiewicz, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.876 · Cited by 58
Younis, H. A., Ruhaiyem, N. I. R., Badr, A. A., Abdul-Hassan, A. K., Alfadli, I. M., Binjumah, W. M., Altuwaijri, E. A., & Nasser, M. (2023) · Processes · doi:10.3390/pr11051488 · Cited by 21
Choose Your Lenses: Flaws in Gender Bias Evaluation
Orgad, H., & Belinkov, Y. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.17 · Cited by 38
Gender Bias in Natural Language Processing and Computer Vision: A Comparative Survey
Bartl, M., Mandal, A., Leavy, S., & Little, S. (2025) · ACM Computing Surveys · doi:10.1145/3700438 · Cited by 48
“A few papers have started to close that gap by, e.g., including neutral gender pronouns in co-reference resolution challenge datasets [27, 100], or the handling of non-binary gender expressions within large language models [38, 58].”
InfluentialTranscending the “Male Code”: Implicit Masculine Biases in NLP Contexts
Seaborn, K., Chandra, S., & Fabre, É. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581017 · Cited by 29
“Cao and Daumé [20] raised awareness of the distinction between social and linguistic genders, warning those of us working in NLP, if not especially on English data sets, to avoid drawing conclusions of subject gender based on gendered terms.”
BackgroundInfluentialHoward, P., Madasu, A., Le, T., Lujan-Moreno, G. A., Bhiwandiwalla, A., & Lal, V. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01138 · Cited by 85
The tail wagging the dog: Dataset construction biases of social bias benchmarks
Selvam, N., Dev, S., Khashabi, D., Khot, T., & Chang, K.-W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.118 · Cited by 11
Sitaram, S., Choudhury, M., Patra, B., Chaudhary, V., Ahuja, K., & Bali, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-tutorials.3 · Cited by 10
“Furthermore, there is little work done on the interaction among the biases in corpora from different languages, influence of grammatical gender (Cao and Daumé, 2021) and other syntactic and semantic factors on measurement and mitigation of biases, and socio-cultural aspects of biases (Sam-basivan…”
BackgroundIntersectional biases in narratives produced by open-ended prompting of generative language models
Shieh, E., Vassel, F.-M., Sugimoto, C. R., & Monroe‐White, T. (2026) · Nature Communications · doi:10.1038/s41467-025-68004-9 · Cited by 32
“To extract textual identity proxies at scale, we fi ne-tune a coreference resolution model (ChatGPT 3.5) using 150 hand-labeled examples to address underperformance in the pretrained LMs on underrepresented groups (e.g., non-binary) 140 .”
InfluentialTransphobia Is in the Eye of the Prompter: Trans-Centered Perspectives on Large Language Models
Scheuerman, M. K., Weathington, K., Petterson, A., Doyle, D. T., Das, D., DeVito, M. A., & Brubaker, J. R. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3743676 · Cited by 14
“While numerous scholars have analyzed and critiqued the trans-exclusive approaches to gender in NLP more broadly (e.g., [48, 70, 88, 125, 204, 238, 262]), it is less clear how these biases manifest in action in the most popular chatbots.”
Optimization’s Neglected Normative Commitments
Laufer, B., Gilbert, T. K., & Nissenbaum, H. F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593976 · Cited by 18
“[42, 111]) including as a framework for evaluating normative assumptions and potential harms [5, 20, 69, 71, 112].”
BackgroundMethodHavens, L., Alex, B., Bach, B., & Terras, M. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.4 · Cited by 19
“Though less explicit in the names of our taxonomy’s labels, we also looked to the descriptions of gender and gender bias from Cao and Daumé III (2021), who point out the limited gender information available in language.”
BackgroundMethodDiscovering Differences in the Representation of People using Contextualized Semantic Axes
Lucy, L., Tadimeti, D., & Bamman, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.228 · Cited by 21
Gender bias in natural language processing
Basta, C. (2022) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-379361 · Cited by 7
“I'm” Lost in Translation: Pronoun Missteps in Crowdsourced Data Sets
Seaborn, K., & Kim, Y. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585667 · Cited by 9
“A wealth of research on NLP and machine translation has pointed to gender biases and especially masculine biases [10, 12, 20, 32].”
BackgroundHearing Ambiguity: Exploring Beyond-Gender Impressions of Artificial Ambiguous Voices
De, M., Seaborn, K., Obaid, M., & Torre, I. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736622 · Cited by 6
“At the same time, we must be careful not to relegate human social identities, especially marginalised ones, to machines, nor equate “gender ambiguity” in voice phenomena with complex human social identities [9, 21, 52, 64].”
Using ChatGPT to Generate Gendered Language
Soundararajan, S., Jeyaraj, M. N., & Delany, S. J. (2023) · Irish Conference on Artificial Intelligence and Cognitive Science (AICS) · doi:10.1109/aics60730.2023.10470830 · Cited by 12
Gender-Inclusive Grammatical Error Correction through Augmentation
Lund, G., Omelianchuk, K., & Samokhin, I. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.13 · Cited by 9
Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?
Gautam, V., Bingert, E., Zhu, D., Lauscher, A., & Klakow, D. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00719 · Cited by 22
“=> yes”
Understanding and Mitigating Gender Bias in Information Retrieval Systems
Seyedsalehi, S., Bigdeli, A., Arabzadeh, N., AlMousawi, B., Marshall, Z., Zihayat, M., & Bagheri, E. (2025) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000103 · Cited by 5
Sobhani, N., Sengupta, K., Microsoft, D., & Delany, S. J. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_119 · Cited by 12
“Also, work by Cao and Daumé III (2021) proposes methods for improving gender inclusivity throughout the Machine Learning lifecycle, including data collection, model training, and evaluation.”
MethodTransforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns
van Boven, G., Du, Y., & Nguyen, D. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659049 · Cited by 5
“However, the majority of work in natural language processing (NLP) considers gender as binary and immutable [8, 14], thereby excluding transgender individuals, who do not identify with the gender they were assigned at birth.”
BackgroundMethodInfluentialDiversity and gender equity in networks of knowledge production and dissemination
Haque, S. A. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20474752 · Cited by 1
AI‐Based Surveillance Systems for Effective Attendance Management
Garg, P. S., Sharma, S., Singh, A. K., & Kumar, N. (2024) · Wiley eBooks · doi:10.1002/9781394200733.ch4 · Cited by 4
Deep Learning-Based Coreference Resolution for Bridge Report Analytics
Chen, Q., & El-Gohary, N. (2024) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784485262.026
Striving for open-source and equitable speech-to-speech translation
Alumäe, T., & Koenecke, A. (2025) · Nature · doi:10.1038/d41586-024-04095-6
English Verb Morphological Adaptive Clustering Based on Semantic Feature Mining
Yin, J., & Yu, N. (2025) · International Journal of High Speed Electronics and Systems · doi:10.1142/s0129156425404413
Enhanced Accuracy in Gender Classification Using Anthropometric Measurements and Machine Learning
Al-Asadi, M. A., Sahan, Z. A., Bhushan, B., & Al-Azzawi, M. S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-90893-4_82
Qiu, X., Cheng, L., Hao, T., & Tan, X. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3630623 · Cited by 2
“For example, combining model averaging with sparse updates or low-rank approximations can mitigate catastrophic forgetting and reduce computational costs [39].”
Ethics and Bias in Natural Language Processing
Talat, Z., & Blodgett, S. L. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.01226-6 · Cited by 1
Digital tools in evaluation: Addressing ethical risks and policy gaps in South Africa
Matlala, L. S. (2025) · Evaluation · doi:10.1177/13563890251378471
Acharya: A Multimodal BERT & CNN and LLM Agent Architecture in Job Matching for Ethical Candidate
Sekar, R. R., Rao, P. V., Ruthik, P., Sumanth, P., & Reddy, B. M. K. (2025) · International Conference on Innovative Data Communication Technologies and Application (ICIDCA) · doi:10.1109/icidca66325.2025.11280422
“%) when producing text for marginalized groups, demonstrating systematic bias within pretrained LMs , Lucy and Bamman[14] (2021) introduced Gender Inclusive Coreference (GIC) testing to measure bias towards gender in coreference resolution.”
Fairness at Risk: Where Bias Emerges in Machine Learning
Albuquerque, O. D. P., Fantinato, M., & Peres, S. M. (2026) · Expert Systems · doi:10.1111/exsy.70265 · Cited by 1
“Moreover, different languages exhibit these biases in unique ways, such as the gendered noun classifications in languages like Spanish, French and Portuguese, which can perpetuate specific gender assumptions (Cao and Daumé 2021).”
InfluentialDo men and women use emojis differently? A corpus-based investigation
Zaki, M., & Mohamed, E. (2026) · Research in Corpus Linguistics · doi:10.32714/ricl.14.01.01
Metadata
- Authors
- Cao, Y. T., & Daumé, H.
- Affiliations
- University of Maryland, College Park (US)
- Microsoft Research, University of Maryland. me@hal3.name
- Corresponding author
- Cao, Yang Trista
- Daumé, Hal
- Venue
- Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2021-11
- Volume
- 47
- Issue
- 3
- Pages
- 615-661
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W3178522238
- Cited by
- 47
- References
- 319
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 7 · 2023: 14 · 2022: 12 · 2021: 1
- FWCI
- 10.64
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Gender Studies › Gender Studies in Language
- Keywords
- coreference resolution, gender bias, trans and non-binary people, stereotyping, machine learning pipelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.