Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing
Open accessAbstract
Several linguistic studies have shown the prevalence of various lexical and grammatical patterns in texts authored by a person of a particular gender, but models for part-of-speech tagging and dependency parsing have still not adapted to account for these differences.To address this, we annotate the Wall Street Journal part of the Penn Treebank with the gender information of the articles' authors, and build taggers and parsers trained on this data that show performance differences in text written by men and women.Further analyses reveal numerous part-of-speech tags and syntactic relations whose prediction performances benefit from the prevalence of a specific gender in the training data.The results underscore the importance of accounting for gendered differences in syntactic tasks, and outline future venues for developing more accurate taggers and parsers.We release our data to the research community.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Major differences between men and women are observed in dialogue structure, word reading, and children's linguistic tasks.
“major differences are observed in dialogue structure75, word reading76, and even in children’s linguistic tasks77.”
From Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare (2020) · cites this paper as [75]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Software Framework for Topic Modelling with Large Corpora
Rek, R. R. U., & Sojka, P. (2010) · doi:10.13140/2.1.2393.1847 · Cited by 3,766 · see citing works
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1075096.1075150 · Cited by 3,040 · see citing works
A Fast and Accurate Dependency Parser using Neural Networks
Chen, D., & Manning, C. D. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1082 · Cited by 1,810 · see citing works
Eckert, P., & McConnell‐Ginet, S. (2013) · Cambridge University Press eBooks · doi:10.1017/cbo9781139245883 · Cited by 1,454 · see citing works
A Maximum Entropy Model for Part-Of-Speech Tagging
Ratnaparkhi, A. (1996) · Cited by 1,268 · see citing works
Universal Dependencies v1: A Multilingual Treebank Collection
Nivre, J., de Marneffe, M.-C., Ginter, F., Goldberg, Y., Hajič, J., Manning, C. D., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., et al. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2hnj7od8afy8 · Cited by 1,031 · see citing works
Enriching the knowledge sources used in a maximum entropy part-of-speech tagger
Toutanova, K., & Manning, C. D. (2000) · doi:10.3115/1117794.1117802 · Cited by 947 · see citing works
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 4× · see citing works
“…by accounting for demographics, and can even improve classification performance (Volkova et al., 2013; Hovy, 2015; Bolukbasi et al., 2016; Benton et al., 2017; Zhao et al., 2017; Lynn et al., 2017), there is still little understanding on the amount and sources of bias in most training sets.”
BackgroundInfluentialSome Advances in Transformation-Based Part of Speech TaggingPreprint
Brill, E. (1994) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9406010 · Cited by 497 · see citing works
Globally Normalized Transition-Based Neural Networks
Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., & Collins, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1231 · Cited by 494 · see citing works
The social impact of natural language processing
Hovy, D., & Spruit, S. L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2096 · Cited by 329 · see citing works
Self-citationDeterministic dependency parsing of English text
Nivre, J., & Scholz, M. (2004) · International Conference on Computational Linguistics · doi:10.3115/1220355.1220365 · Cited by 222 · see citing works
Memory-Based Dependency Parsing
Nivre, J., Hall, J., & Nilsson, J. (2004) · Cited by 204 · see citing works
When Sex and Syntax Go Hand in Hand: Gender Agreement in Language Production
Vigliocco, G., & Franck, J. (1999) · Journal of Memory and Language · doi:10.1006/jmla.1998.2624 · Cited by 199 · see citing works
Exploring Demographic Language Variations to Improve Multilingual Sentiment Analysis in Social Media
Volkova, S., Wilson, T., & Yarowsky, D. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1187 · Cited by 191 · see citing works
Demographic Factors Improve Classification Performance
Hovy, D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1073 · Cited by 148 · see citing works
Self-citationTagging Performance Correlates with Author Age
Hovy, D., & Søgaard, A. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-2079 · Cited by 106 · see citing works
Self-citationWhat’s in a Name: A Study of Names, Gender Inference, and Gender Behavior in Facebook
Tang, C., Ross, K. W., Saxena, N., & Chen, R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20244-5_33 · Cited by 99 · see citing works
Cross-lingual syntactic variation over age and gender
Johannsen, A., Hovy, D., & Søgaard, A. (2015) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k15-1011 · Cited by 85 · see citing works
Self-citationMulti-Task Learning for Mental Health using Social Media TextPreprint
Benton, A., Mitchell, M., & Hovy, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.03538 · Cited by 84 · see citing works
Self-citationChallenges of studying and processing dialects in social media
Jørgensen, A., Hovy, D., & Søgaard, A. (2015) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w15-4302 · Cited by 62 · see citing works
Self-citationQuantifying and Reducing Stereotypes in Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. T. (2016) · OpenBU (Boston University) · doi:10.48550/arxiv.1606.06121 · Cited by 57 · see citing works
Human Centered NLP with User-Factor Adaptation
Lynn, V., Son, Y., Kulkarni, V., Balasubramanian, N., & Schwartz, H. A. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1119 · Cited by 47 · see citing works
Gender Differences in English Syntax
Mondorf, B. (2002) · Journal of English Linguistics · doi:10.1177/007242030002005 · Cited by 24 · see citing works
Improving Data Driven Dependency Parsing using Clausal Information
Gadde, P. H., Jindal, K., Husain, S., Sharma, D. M., & Sangal, R. (2010) · Cited by 17 · see citing works
Gender Differences in Syntactic Development Among English Speaking Adolescents
Cornett, H. E. (2014) · Inquiries Journal · Cited by 10 · see citing works
Dialog Structure Through the Lens of Gender, Gender Environment, and Power
Prabhakaran, V., & Rambow, O. (2017) · Dialogue & Discourse · doi:10.5087/dad.2017.202 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Hannah E. Cornett. 2014. Gender differences in syn- tactic development among english speaking adoles- cents. Inquiries Journal/Student Pulse, 6(3):1-6.
- Dan Klein and Christopher D. Manning. 2003. Accu- rate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Compu- tational Linguistics (Volume 1), ACL 2003, pages 423-430, Sapporo, Japan.
- Karo Moilanen and Stephen Pulman. 2007. Senti- ment composition. In Proceedings of the Confer- ence on Recent Advances in Natural Language Pro- cessing, volume 7 of RANLP 2007, pages 378-382, Borovets, Bulgaria.
- Kristina Toutanova, Dan Klein, Christopher D. Man- ning, and Yoram Singer. 2003. Feature-rich part-of- speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computa- tional Linguistics on Human Language Technology- Volume 1, NAACL-HLT 2003, pages 173-180, Ed- monton, Canada.
Cited by
Sorted by citations
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · see citing works
“…al. (2018); Badjatiya et al. (2019); Bhargava and Forsyth (2019); Cao and Daumé (2019); Cho et al. (2019); Davidson et al. (2019); Dev et al. (2019); Garimella et al. (2019); Garg et al. (2019); Huang et al. (2019); James-Sorenson and AlvarezMelis (2019); Kaneko and Bollegala (2019); Liu et al.…”
BackgroundInfluentialSex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · see citing works
Five sources of bias in natural language processing
Hovy, D., & Prabhumoye, S. (2021) · Language and Linguistics Compass · doi:10.1111/lnc3.12432 · Cited by 298 · see citing works
Self-citationGender Bias in Machine Translation
Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00401 · Cited by 189 · see citing works
HONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95 · see citing works
Self-citationCzarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72 · see citing works
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 74 · see citing works
The Gender Gap Tracker: Using Natural Language Processing to measure gender bias in media
Asr, F. T., Mazraeh, M., Lopes, A., Gautam, V., Gonzales, J., Rao, P. P., & Taboada, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0245533 · Cited by 40 · see citing works
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 40 · see citing works
Gender bias in transformers: A comprehensive review of detection and mitigation strategies
Nemani, P., Joel, Y. D., Vijay, P., & Liza, F. F. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100047 · Cited by 33 · see citing works
Garimella, A., Amarnath, A., KUMAR, K. T. K., Yalla, A. P., Natarajan, A., Chhaya, N., & Srinivasan, B. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.397 · Cited by 33 · see citing works
Self-citationNLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 45 · see citing works
Responsible natural language processing: A principlist framework for social benefits
Behera, R. K., Bala, P. K., Rana, N. P., & Irani, Z. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122306 · Cited by 26 · see citing works
Minot, J. R., Cheney, N., Maier, M., Elbers, D. C., Danforth, C. M., & Dodds, P. S. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3524887 · Cited by 23 · see citing works
Comparative Evaluation of Label-Agnostic Selection Bias in Multilingual Hate Speech Datasets
Ousidhoum, N. D., Song, Y., & Yeung, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.199 · Cited by 25 · see citing works
DramatVis Personae: Visual Text Analytics for Identifying Social Biases in Creative Writing
Hoque, M. N., Ghai, B., & Elmqvist, N. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533526 · Cited by 18 · see citing works
Towards Socially Responsible AI: Cognitive Bias-Aware Multi-Objective Learning
Sen, P., & Ganguly, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5654 · Cited by 18 · see citing works
Purschke, C. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.536086 · Cited by 13 · see citing works
Investigating Failures of Automatic Translationin the Case of Unambiguous Gender
Renduchintala, A., & Williams, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.243 · Cited by 11 · see citing works
Havens, L., Terras, M., Bach, B., & Alex, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05911 · Cited by 11 · see citing works
Investigating Failures of Automatic Translation in the Case of Unambiguous GenderPreprint
Renduchintala, A., & Williams, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.07838 · Cited by 28 · see citing works
Minot, J. R., Cheney, N., Maier, M. E., Elbers, D. C., Danforth, C. M., & Dodds, P. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05841 · Cited by 9 · see citing works
An Overview of Fairness in Data – Illuminating the Bias in Data Pipeline
Kumar, B. S., Chandrabose, A., & Chakravarthi, B. R. (2021) · Cited by 6 · see citing works
SocioProbe: What, When, and Where Language Models Learn about Sociodemographics
Lauscher, A., Bianchi, F., Bowman, S. R., & Hovy, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.539 · Cited by 6 · see citing works
Self-citationWhat do Bias Measures MeasurePreprint
Dev, S., Sheng, E., Zhao, J., Sun, J., Hou, Y., Sanseverino, M., Kim, J., Peng, N., & Chang, K.-W. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. L., Reinecke, K., & Sap, M. (2023) · arXiv · doi:10.48550/arxiv.2306.01943 · Cited by 31 · see citing works
“Even tasks that are considered purely linguistic have seen design biases: in parsing and tagging, performance differences exist between texts written by people of different genders (Garimella et al., 2019), ages (Hovy and Søgaard, 2015), and races (Johannsen et al., 2015; Jørgensen et al., 2015).”
BackgroundSlovene and Croatian word embeddings in terms of gender occupational analogies
Ulčar, M., Supej, A., Robnik‐Šikonja, M., & Pollak, S. (2021) · Slovenščina 2 0 empirične aplikativne in interdisciplinarne raziskave · doi:10.4312/slo2.0.2021.1.26-59 · Cited by 4 · see citing works
A Study on Bias Detection and Classification in Natural Language ProcessingPreprint
Evans, A. S., Moniz, H., & Luisa, C. (2023) · Research Square · doi:10.21203/rs.3.rs-3351695/v1 · Cited by 4 · see citing works
Addressing Gender Bias in Generative Large Language ModelsPreprint
Zhou, H., Inkpen, D. Z., & Kantarcı, B. (2024) · Research Square · doi:10.21203/rs.3.rs-4670889/v1 · Cited by 2 · see citing works
Evaluating Debiasing Techniques for Intersectional Biases
Subramanian, S., Han, X., Baldwin, T., Cohn, T., & Frermann, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.193 · Cited by 40 · see citing works
Fairness-aware Class Imbalanced Learning
Subramanian, S., Rahimi, A., Baldwin, T., Cohn, T., & Frermann, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.155 · Cited by 24 · see citing works
da Silva, E. H., Pardo, T. A. S., & Roman, N. T. (2023) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2023.233848 · Cited by 1 · see citing works
How Well Do You Know Your Audience? Toward Socially-aware Question Generation
Stewart, I., & Mihalcea, R. F. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.27 · Cited by 1 · see citing works
Self-citationReview of Dunn (2022): Natural Language Processing for Corpus Linguistics
Schmück, H. (2023) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.00057.sch · Cited by 1 · see citing works
Demographic Factors as Domains for Adaptation in Linguistic Preprocessing
Morini, S. (2019) · CUNY Academic Works (City University of New York)
Learning PyTorch Through A Neural Dependency Parsing Exercise
Jurgens, D. (2021) · Workshop on Teaching NLP (TeachingNLP) · doi:10.18653/v1/2021.teachingnlp-1.10
How Well Do You Know Your Audience? Reader-aware Question Generation.Preprint
Stewart, I., & Mihalcea, R. F. (2021) · arXiv (Cornell University)
Self-citationPGST: A Persian gender style transfer method
Khanmohammadi, R., & Mirroshandel, S. A. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000426 · Cited by 1 · see citing works
Mitigating Gender Bias in Hindi to English Neural Machine Translation
Tyagi, S., & Tyagi, A. (2025) · Algorithms for intelligent systems · doi:10.1007/978-981-97-7946-8_8
- Authors
- Garimella, A., Banea, C., Hovy, D., & Mihalcea, R.
- Affiliations
- University of Michigan (US)
- Bocconi University (IT)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3493-3498
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2952638532
- Cited by
- 41
- References
- 31
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 8 · 2022: 6 · 2021: 18 · 2020: 6 · 2019: 1
- FWCI
- 2.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion, cc-by)
- INFM-OAR (INFN Catania) (submittedVersion, other-oa)
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Funders
- National Science Foundation (1815291)
- John Templeton Foundation (61156)
- Michigan Institute for Data Science, University of Michigan
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- gender bias, part-of-speech tagging, dependency parsing, gendered language, author gender, Penn Treebank, Wall Street Journal, syntactic relations, training data bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
