Unintended machine learning biases as social barriers for persons with disabilitiess
Abstract
Persons with disabilities face many barriers to full participation in society, and the rapid advancement of technology has the potential to create ever more. Building equitable and inclusive technologies for people with disabilities demands paying attention to more than accessibility, but also to how social attitudes towards disability are represented within technology. Representations perpetuated by machine learning (ML) models often inadvertently encode undesirable social biases from the data on which they are trained. This can result, for example, in text classification models producing very different predictions for I am a person with mental illness , and I am a tall person . In this paper, we present evidence of such biases in existing ML models, and in data used for model development. First, we demonstrate that a machine-learned model to moderate conversations classifies texts which mention disability as more "toxic". Similarly, a machine-learned sentiment analysis model rates texts which mention disability as more negative. Second, we demonstrate that neural text representation models that are critical to many ML applications can also contain undesirable biases towards mentions of disabilities. Third, we show that the data used to develop such models reflects topical biases in social discourse which may explain such biases in the models - for instance, gun violence, homelessness, and drug addiction are over-represented in discussions about mental illness.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Natural language processing (NLP) models associate mental illness with negative terms like gun violence, homelessness, and drug addiction.
“For instance, researchers found that natural language processing (NLP) models associate mental illness with negative terms like gun violence, homelessness, and drug addiction (Hutchinson et al., 2020).”
From Exploring the role of generative AI in higher education: Semi-structured interviews with students with disabilities (2025) · cites this paper as Hutchinson et al., 2020
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · see citing works
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · see citing works
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · see citing works
Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008) · Political Analysis · doi:10.1093/pan/mpn018 · Cited by 542 · see citing works
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279 · see citing works
Pierrès, O., Darvishy, A., & Christen, M. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13134-8 · Cited by 48 · see citing works
“For instance, researchers found that natural language processing (NLP) models associate mental illness with negative terms like gun violence, homelessness, and drug addiction (Hutchinson et al., 2020).”
Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation
Thieme, A., Faia Marques, R., Grayson, M., Balachandar, S., Tyler Cassidy, C., Choksi, M. Z., Longden, C., Huda, R. S., Ileve Kalovwe, N., Mallon, C., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790768 · Cited by 1 · see citing works
“Generative AI outputs tend to reproduce normative views and mirror societal biases for marginalized groups [15, 21, 36, 41, 42, 45, 53, 66, 72, 91], including PwD [10, 12, 39, 48, 54, 62, 77, 113].”
- Authors
- Hutchinson, B., Prabhakaran, V., Denton, R., Webster, K., Zhong, Y., & Denuyl, S.
- Affiliations
- Google (United States) (US)
- Venue
- ACM SIGACCESS Accessibility and Computing
- Published
- 2020-03-02
- Issue
- 125
- Pages
- 1-1
- ISSN
- 1558-1187 (ISSN-L)
- 1558-2337
- OpenAlex
- W2979746687
- Cited by
- 49
- References
- 8
- Citations by year
- 2026: 3 · 2025: 13 · 2024: 9 · 2023: 8 · 2022: 6 · 2021: 7 · 2020: 3
- FWCI
- 8.18
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGACCESS Accessibility and Computing (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Safety Research › Disability Rights and Representation
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- disability bias, machine learning bias, social attitudes toward disability, inclusive technology, accessibility, sentiment analysis, text representation models, training data bias, mental illness discourse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
