CoChat
Sign Up
PreprintCited by 14Sentiment Analysis and Opinion MiningEthics and Social Impacts of AIHate Speech and Cyberbullying Detection

Identification of Bias Against People with Disabilities in Sentiment Analysis and Toxicity Detection Models

Open access
Venkit, Pranav Narayanan, Wilson, Shomir·2021·arXiv
DOI 10.48550/arxiv.2111.13259arXiv 2111.13259
Open this paper in CoChat — free accountOpen PDF

Abstract

Sociodemographic biases are a common problem for natural language processing, affecting the fairness and integrity of its applications. Within sentiment analysis, these biases may undermine sentiment predictions for texts that mention personal attributes that unbiased human readers would consider neutral. Such discrimination can have great consequences in the applications of sentiment analysis both in the public and private sectors. For example, incorrect inferences in applications like online abuse and opinion analysis in social media platforms can lead to unwanted ramifications, such as wrongful censoring, towards certain populations. In this paper, we address the discrimination against people with disabilities, PWD, done by sentiment analysis and toxicity classification models. We provide an examination of sentiment and toxicity analysis models to understand in detail how they discriminate PWD. We present the Bias Identification Test in Sentiments (BITS), a corpus of 1,126 sentences designed to probe sentiment analysis models for biases in disability. We use this corpus to demonstrate statistically significant biases in four widely used sentiment analysis tools (TextBlob, VADER, Google Cloud Natural Language API and DistilBERT) and two toxicity analysis models trained to predict toxic comments on Jigsaw challenges (Toxic comment classification and Unintended Bias in Toxic comments). The results show that all exhibit strong negative biases on sentences that mention disability. We publicly release BITS Corpus for others to identify potential biases against disability in any sentiment analysis tools and also to update the corpus to be used as a test for other sociodemographic variables as well.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2111.13259

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A study by Pennsylvania State University on 13 AI language models revealed that the models consistently created negative associations with disability-related terms, highlighting explicit and implicit biases in the training data.

    “For example, a study by Pennsylvania State University on 13 AI language models revealed that the models consistently created negative associations with disability-related terms, highlighting explicit and implicit biases in the training data (Venkit & Wilson, 2021).”

    From Evolving intersections: AI, disability, and academic integrity in higher education (2025) · cites this paper as Venkit & Wilson, 2021

    The authors’ affiliation is Pennsylvania State University (p. 1), and the Abstract (p. 1) reports strong negative bias toward sentences mentioning disability. But it identifies four sentiment-analysis tools and two toxicity models, not 13 language models. The Discussion and Conclusion (p. 10) links the findings to non-inclusive training, but the paper does not establish both explicit and implicit biases in the training data as claimed. (LLM, conf=0.91)

    Experimental — accuracy not yet measured

  • Linguistic models have been shown to encode disability-related terms with negative valence, reinforcing the bias problem.

    “Linguistic models have been shown to encode disabilityrelated terms with negative valence (Venkit & Wilson, 2021), reinforcing the bias problem (McMurtrie, 2023; Rocky Mountain ADA Center, 2023).”

    From The Use of Generative AI by Students with Disabilities in UK Higher Education: A Mixed-Methods Survey of 125 Students (2025) · cites this paper as Venkit & Wilson, 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 37 · Cited by 14

Research works printed but not linked: 23

  1. [3][3] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science , 356(6334):183-186, 2017. -
  2. [4][4] Ekaba Bisong. Google automl: Cloud natural language processing. In Building Machine Learning and Deep Learning Models on Google Cloud Platform , pages 599-612. Springer, 2019. -
  3. [5][5] Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. Contextualizing hate speech classifiers with post-hoc explanation. arXiv preprint arXiv:2005.02439 , 2020. - · arXiv
  4. [6][6] Jerome Bickenbach. The world report on disability. Disability & Society , 26(5):655-658, 2011. -
  5. [8][8] Mike Oliver. The politics of disability. Critical social policy , 4(11):21-32, 1984. -
  6. [9][9] Bo Chen and Donna Marie McNamara. Disability discrimination, medical rationing and covid-19. Asian bioethics review , 12(4):511-518, 2020. -
  7. [11][11] David Noever. Machine learning suites for online toxicity detection. arXiv preprint arXiv:1810.01869 , 2018. - · arXiv
  8. [13][13] John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. Toxicity detection: Does context really matter? Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020. -
  9. [15][15] Bo Pang and Lillian Lee. Opinion mining and sentiment analysis. Comput. Linguist , 35(2):311-312, 2009. -
  10. [16][16] Fabio Del Vigna12, Andrea Cimino23, Felice Dell'Orletta, Marinella Petrocchi, and Maurizio Tesconi. Hate me, hate me not: Hate speech detection on facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17) , pages 86-95, 2017. -
  11. [17][17] CHE Gilbert and Erric Hutto. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth International Conference on Weblogs and Social Media (ICWSM-14). Available at (20/04/16) http://comp. social. gatech. edu/papers/icwsm14. vader. hutto. pdf , volume 81, page 82, 2014. -
  12. [21][21] Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, and Rocco Oliveto. Sentiment analysis for software engineering: How far can we go? In Proceedings of the 40th International Conference on Software Engineering , pages 94-104, 2018. -
  13. [28][28] Stephen J Macdonald, Catherine Donovan, and John Clayton. The disability bias: understanding the context of hate in comparison with other minority populations. Disability & Society , 32(4):483-499, 2017. -
  14. [33][33] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337 , 2019. - · arXiv
  15. [34][34] Walaa Medhat, Ahmed Hassan, and Hoda Korashy. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal , 5(4):1093-1113, 2014. -
  16. [35][35] Filipe N Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, and Fabrício Benevenuto. Sentibench-a benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science , 5(1):1-29, 2016. -
  17. [37][37] Mike Thelwall. Gender bias in sentiment analysis. Online Information Review , 2018. -
  18. [42][42] Nathan R Kuncel, David M Klieger, and Deniz S Ones. In hiring, algorithms beat instinct. Harvard business review , 92(5):p32-32, 2014. -
  19. [44][44] Alan Rubel. The black box society: The secret algorithms that control money and information, by frank pasquale. cambridge: Harvard university press, 2015. 320 pp. isbn 978-0674368279. Business Ethics Quarterly , 26(4):568-571, 2016. -
  20. [48][48] Giacomo Vivanti. Ask the editor: What is the most appropriate way to talk about individuals with a diagnosis of autism? Journal of autism and developmental disorders , 50(2):691-693, 2020. -
  21. [49][49] Monique Botha, Jacqueline Hanlon, and Gemma Louise Williams. Does language matter? identity-first versus person-first language use in autism research: A response to vivanti. Journal of Autism and Developmental Disorders , pages 1-9, 2021. -
  22. [50][50] Paul Ekman. Facial expression and emotion. American psychologist , 48(4):384, 1993. -
  23. [51][51] Prabhaker Mishra, Uttam Singh, Chandra M Pandey, Priyadarshni Mishra, and Gaurav Pandey. Application of student's t-test, analysis of variance, and covariance. Annals of cardiac anaesthesia , 22(4):407, 2019. -

Authors
Venkit, P. N., & Wilson, S.
Affiliations
Pennsylvania State University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2111.13259
arXiv
2111.13259
OpenAlex
W3217764315
Cited by
14
References
37
Citations by year
2025: 3 · 2024: 4 · 2023: 7
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
sentiment analysis, toxicity detection, sociodemographic bias, TextBlob, VADER, DistilBERT, online abuse

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review