Artificial Intelligence in mental health and the biases of language based models
Open accessThe DOI matches its Crossref record; 70 of the 72 references it cites were found in a registry or an index.
Abstract
Background: The rapid integration of Artificial Intelligence (AI) into the healthcare field has occurred with little communication between computer scientists and doctors. The impact of AI on health outcomes and inequalities calls for health professionals and data scientists to make a collaborative effort to ensure historic health disparities are not encoded into the future. We present a study that evaluates bias in existing Natural Language Processing (NLP) models used in psychiatry and discuss how these biases may widen health inequalities. Our approach systematically evaluates each stage of model development to explore how biases arise from a clinical, data science and linguistic perspective. Design/Methods: A literature review of the uses of NLP in mental health was carried out across multiple disciplinary databases with defined Mesh terms and keywords. Our primary analysis evaluated biases within ‘GloVe’ and ‘Word2Vec’ word embeddings. Euclidean distances were measured to assess relationships between psychiatric terms and demographic labels, and vector similarity functions were used to solve analogy questions relating to mental health. Results: Our primary analysis of mental health terminology in GloVe and Word2Vec embeddings demonstrated significant biases with respect to religion, race, gender, nationality, sexuality and age. Our literature review returned 52 papers, of which none addressed all the areas of possible bias that we identify in model development. In addition, only one article existed on more than one research database, demonstrating the isolation of research within disciplinary silos and inhibiting cross-disciplinary collaboration or communication. Conclusion: Our findings are relevant to professionals who wish to minimize the health inequalities that may arise as a result of AI and data-driven algorithms. We offer primary research identifying biases within these technologies and provide recommendations for avoiding these harms in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs are prone to biased outputs due to the nature of their training data, which often reflects societal inequalities and fails to uniformly represent diverse demographic groups.21”
From Role of large language models in mental health research: an international survey of researchers' practices and perspectives (2025) · cites this paper as [21]
Experimental — accuracy not yet measuredIn the citing paper
“AI biases in medical research have generated considerable concern, since biases related to race, ethnicity, gender, sexuality, age, nationality, and socioeconomic status in health-related datasets can perpetuate health disparities by supporting biased hypotheses, models, theories, and policies [177, 198, 211].”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [211]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Unequal treatment: confronting racial and ethnic disparities in health care
(2003) · Choice Reviews Online · doi:10.5860/choice.40-5843 · Cited by 7,043
Capps, L. (2003) · JAMA · doi:10.1001/jama.290.18.2487-b · Cited by 4,526
Lakoff, R. (1973) · Language in Society · doi:10.1017/s0047404500000051 · Cited by 3,138
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,071
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Psychological Aspects of Natural Language Use: Our Words, Our Selves
Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145041 · Cited by 2,624 · cited 2×
“In the psychiatric literature there exists extensive research that describes how depression closely relates to increased use of first person singular pronouns, however we are yet to see a tool that can accurately differentiate between word particles that result from mental pathology, and those that originate from indi…”
Background(2016) · Cited by 2,334
Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Vyas, D. A., Eisenstein, L. G., & Jones, D. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmms2004740 · Cited by 1,731
Research works printed but not linked: 2
- Bolukbasi T, Chang K-W, Zou J, Saligrama V, Kalai A. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. ArXiv:160706520 [Cs, Stat] 2016.
- Learning Word Vectors for Sentiment Analysis | Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies—Volume 1. https://dl.acm.org/doi/abs/10.5555/2002472.2002491. Accessed 2 Mar. 2020. · doi:10.5555/2002472.2002491
Cited by
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
“…facilitate documentation of patient reports (e.g., a progress report) 3,4 , improving diagnostic accuracy 5 , and assisting in various clinical care 6,7 , while at the same time concerning the hallucinations and fabrications 7,8 , bias and stereotype 9 , and risks of patient privacy and ethics 10 .”
Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263
Artificial intelligence in positive mental health: a narrative review
Thakkar, A., Gupta, A., & De Sousa, A. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1280235 · Cited by 251
The Opportunities and Risks of Large Language Models in Mental Health
Lawrence, H. R., Schneider, R. A., Rubin, S. B., Matarić, M. J., McDuff, D. J., & Jones Bell, M. (2024) · JMIR Mental Health · doi:10.2196/59479 · Cited by 235
“…supervision and evaluation can, therefore, lead to problematic generation of biased content and disparate model performance for different groups [45,55-57] (of note, however, there is some evidence that clinicians perceive less bias in LLM-generated responses [58] relative to clinician-generated…”
BackgroundExploring the Ethical Challenges of Conversational AI in Mental Health Care: Scoping Review
Rahsepar Meadi, M., Sillekens, T., Metselaar, S., van Balkom, A., Bernstein, J., & Batelaan, N. (2025) · JMIR Mental Health · doi:10.2196/60432 · Cited by 211
Metadata
- Authors
- Straw, I., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Straw, Isabel
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-12-17
- Volume
- 15
- Issue
- 12
- Pages
- e0240376
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7745984
- OpenAlex
- W3112631310
- Cited by
- 164
- References
- 72
- Citations by year
- 2026: 40 · 2025: 46 · 2024: 37 · 2023: 23 · 2022: 12 · 2021: 6
- FWCI
- 13.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (2418758)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- algorithmic bias, natural language processing, mental health, word embeddings, gloves, Word2Vec, health disparities, psychiatric terminology, demographic bias, interdisciplinary collaboration
- MeSH terms
- Intersectoral Collaboration, Data Science, Humans, Linguistics, Mental Health, Natural Language Processing, Psychiatry, Bias, Health Status Disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial Intelligence in mental health and the biases of language based models. https://app.cochat.ai/papers/10.1371/journal.pone.0240376
- The paper
- Straw, I., & Callison-Burch, C. (2020). Artificial Intelligence in mental health and the biases of language based models. PLOS ONE. https://doi.org/10.1371/journal.pone.0240376
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.