Social Biases in NLP Models as Barriers for Persons with Disabilities
Open accessAbstract
Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate undesirable social biases from the data on which they are trained. In this paper, we present evidence of such undesirable biases towards mentions of disability in two different English language models: toxicity prediction and sentiment analysis. Next, we demonstrate that the neural embeddings that are the critical first step in most NLP pipelines similarly contain undesirable biases towards mentions of disability. We end by highlighting topical biases in the discourse about disability which may contribute to the observed model biases; for instance, gun violence, homelessness, and drug addiction are over-represented in texts discussing mental illness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
'Clean' text has been misrepresented as being 'unbiased' against any community, a notion that has been strongly contested.
“'Clean' text additionally has been misrepresented as being 'unbiased' against any community-a notion that has been strongly contested [22, 73, 130].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [73]
Section 3, “Biases in Text Classification Models” (page unavailable), reports that non-toxic comments mentioning disability can be flagged as toxic. Section 7, “Discussion and Conclusion” (page unavailable), says language norms differ across groups and disability-related terminology is contested. These passages challenge an assumption of neutrality, but the paper does not document a claim that ‘clean’ text was represented as unbiased against every community. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 43 · Cited by 274
Research works printed but not linked: 35
- Su Lin Blodgett, Solon Barocas, Hal III Daume, and Hanna Wallach. 2020. Language (technology) is power: The need to be explicit about NLP harms. In Proceedings of the Annual Meeting of the Associ- ation for Computational Lingustics (ACL).
- Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer As Woman is to Homemaker? Debiasing Word Embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems.
- Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference.
- Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceed- ings of the Annual Meeting of the Association for Computational Lingustics (ACL).
- Michael Castelle. 2018. The linguistic ideologies of deep abusive language classification. In Proceed- ings of the 2nd Workshop on Abusive Language On- line (ALW2), Brussels, Belgium. ACL. · DOI
- Eshwar Chandrasekharan, Chaitrali Gandhi, Matthew Wortley Mustelier, and Eric Gilbert. 2019. Crossmod: A cross-community learning- based system to assist reddit moderators. Proc. ACM Hum.-Comput. Interact., 3(CSCW). · DOI · Paper page
- Alexander D'Amour, Hansa Srinivasan, James At- wood, Pallavi Baljekar, D Sculley, and Yoni Halpern. 2020. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In Pro- ceedings of the 2020 Conference on Fairness, Ac- countability, and Transparency, pages 525-534.
- Thomas Davidson, Debasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy. Association for Com- putational Linguistics. · DOI · Paper page
- Maria De-Arteaga, Alexey Romanov, Hanna Wal- lach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kentha- padi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (NAACL-HLT).
- Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigat- ing unintended bias in text classification. In Pro- ceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society.
- Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd inno- vations in theoretical computer science conference.
- Danielle Ensign, Sorelle A Friedler, Scott Neville, Car- los Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway feedback loops in predictive polic- ing. In Conference of Fairness, Accountability, and Transparency.
- Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2017. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115.
- Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfac- tual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19. Association for Computing Machinery. · DOI
- Dirk Hovy and Shannon L Spruit. 2016. The social im- pact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
- David Jurgens, Libby Hemphill, and Eshwar Chan- drasekharan. 2019. A just and comprehensive strat- egy for using NLP to address online abuse. In Pro- ceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, Florence, Italy. Association for Computational Linguistics. · DOI
- Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contex- tualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Associ- ation for Computational Linguistics. · DOI · Paper page
- Lydia T. Liu, Sarah Dean, Esther Rolf, Max Sim- chowitz, and Moritz Hardt. 2019. Delayed impact of fair machine learning. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Confer- ences on Artificial Intelligence Organization. · DOI
- Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as cau- casian is to police: Detecting and removing multi- class bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Asso- ciation for Computational Linguistics. · DOI
- Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William Isaac. 2020. Participatory problem formulation for fairer ma- chine learning through community based system dy- namics approach. In ICLR Workshop on Machine Learning in Real Life (ML-IRL).
- Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measur- ing social biases in sentence encoders. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers).
- Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. 2019. The social cost of strategic classifica- tion. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 230-239.
- Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, account- ability, and transparency, pages 220-229.
- Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin'words: Lexical feature selec- tion and evaluation for identifying the content of po- litical conflict. Political Analysis, 16(4):372-403.
- Mohamed M Mostafa. 2013. More than words: So- cial networks' text mining for consumer brand sentiments. Expert Systems with Applications, 40(10):4241-4251.
- Deirdre K Mulligan, Joshua A Kroll, Nitin Kohli, and Richmond Y Wong. 2019. This thing called fairness: Disciplinary confusion realizing a value in technol- ogy. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1-36.
- Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. NYU Press.
- Brian A. Nosek, Frederick L. Smyth, Jeffrey J. Hansen, Thierry Devos, Nicole M. Lindner, Kate A. Ranganath, Colin Tucker Smith, Kristina R. Ol- son, Dolly Chugh, Anthony G. Greenwald, and Mahzarin R. Banaji. 2007. Pervasiveness and corre- lates of implicit attitudes and stereotypes. European Review of Social Psychology, 18(1):36-88. · DOI · Paper page
- Vinodkumar Prabhakaran, Ben Hutchinson, and Mar- garet Mitchell. 2019. Perturbation sensitivity analy- sis to detect unintended model biases. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), Hong Kong, China. As- sociation for Computational Linguistics. · DOI · Paper page
- Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
- Katrina Scior. 2011. Public awareness, attitudes and beliefs regarding intellectual disability: A system- atic review. Research in developmental disabilities, 32(6):2164-2182.
- Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical sys- tems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 59-68.
- Andreas Veglis. 2014. Moderation techniques for so- cial media content. In International Conference on Social Computing and Social Media. Springer.
- Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detec- tion. In Proceedings of the Third Workshop on Abu- sive Language Online, pages 80-93, Florence, Italy. Association for Computational Linguistics. · DOI
- Authors
- Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 5491-5501
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 2005.00813
- OpenAlex
- W3035296331
- Cited by
- 274
- References
- 43
- Citations by year
- 2026: 21 · 2025: 41 · 2024: 44 · 2023: 66 · 2022: 44 · 2021: 48 · 2020: 10
- FWCI
- 43.56
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › Safety Research › Disability Rights and Representation
- Keywords
- social bias, disability, natural language processing models, toxicity prediction, sentiment analysis, neural embeddings, bias in language models, mental illness discourse, inclusive technology, disability representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
