Artificial Intelligence in mental health and the biases of language based models
Open accessThe DOI matches its Crossref record; 70 of the 72 references it cites were found in a registry or an index.
Abstract
Background: The rapid integration of Artificial Intelligence (AI) into the healthcare field has occurred with little communication between computer scientists and doctors. The impact of AI on health outcomes and inequalities calls for health professionals and data scientists to make a collaborative effort to ensure historic health disparities are not encoded into the future. We present a study that evaluates bias in existing Natural Language Processing (NLP) models used in psychiatry and discuss how these biases may widen health inequalities. Our approach systematically evaluates each stage of model development to explore how biases arise from a clinical, data science and linguistic perspective. Design/Methods: A literature review of the uses of NLP in mental health was carried out across multiple disciplinary databases with defined Mesh terms and keywords. Our primary analysis evaluated biases within ‘GloVe’ and ‘Word2Vec’ word embeddings. Euclidean distances were measured to assess relationships between psychiatric terms and demographic labels, and vector similarity functions were used to solve analogy questions relating to mental health. Results: Our primary analysis of mental health terminology in GloVe and Word2Vec embeddings demonstrated significant biases with respect to religion, race, gender, nationality, sexuality and age. Our literature review returned 52 papers, of which none addressed all the areas of possible bias that we identify in model development. In addition, only one article existed on more than one research database, demonstrating the isolation of research within disciplinary silos and inhibiting cross-disciplinary collaboration or communication. Conclusion: Our findings are relevant to professionals who wish to minimize the health inequalities that may arise as a result of AI and data-driven algorithms. We offer primary research identifying biases within these technologies and provide recommendations for avoiding these harms in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs are prone to biased outputs due to the nature of their training data, which often reflects societal inequalities and fails to uniformly represent diverse demographic groups.21”
From Role of large language models in mental health research: an international survey of researchers' practices and perspectives (2025) · cites this paper as [21]
Experimental — accuracy not yet measuredIn the citing paper
“AI biases in medical research have generated considerable concern, since biases related to race, ethnicity, gender, sexuality, age, nationality, and socioeconomic status in health-related datasets can perpetuate health disparities by supporting biased hypotheses, models, theories, and policies [177, 198, 211].”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [211]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Unequal treatment: confronting racial and ethnic disparities in health care
(2003) · Choice Reviews Online · doi:10.5860/choice.40-5843 · Cited by 7,043
Capps, L. (2003) · JAMA · doi:10.1001/jama.290.18.2487-b · Cited by 4,526
Lakoff, R. (1973) · Language in Society · doi:10.1017/s0047404500000051 · Cited by 3,138
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,071
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Psychological Aspects of Natural Language Use: Our Words, Our Selves
Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145041 · Cited by 2,624 · cited 2×
“In the psychiatric literature there exists extensive research that describes how depression closely relates to increased use of first person singular pronouns, however we are yet to see a tool that can accurately differentiate between word particles that result from mental pathology, and those that originate from indi…”
Background(2016) · Cited by 2,334
Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Vyas, D. A., Eisenstein, L. G., & Jones, D. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmms2004740 · Cited by 1,731
Research works printed but not linked: 2
- Bolukbasi T, Chang K-W, Zou J, Saligrama V, Kalai A. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. ArXiv:160706520 [Cs, Stat] 2016.
- Learning Word Vectors for Sentiment Analysis | Proceedings of the 49th Annual Meeting of the Associa- tion for Computational Linguistics: Human Language Technologies-Volume 1. https://dl.acm.org/doi/ abs/10.5555/2002472.2002491. Accessed 2 Mar. 2020. · doi:10.5555/2002472.2002491
Cited by
Items 101–165 of 165
Dev, V., Consedine, N. S., Boggiss, A. L., & Serlachius, A. S. (2026) · Diabetic Medicine · doi:10.1111/dme.70299 · Cited by 2
Wang, R., Sun, X., Sun, Y., & Wang, J. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1794568 · Cited by 4
Jun, G. (2024) · Mobile Networks and Applications · doi:10.1007/s11036-024-02385-x · Cited by 2
Lange, M., Koliousis, A., Fayez, F., Gogarty, E., & Twumasi, R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315768 · Cited by 4
“GloVe is also used to investigate bias in NLP models; this is commonly done using graphical representations of words [91, 105, 134, 137–140].”
InfluentialBuckeridge, D. L. (2022) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-09108-7_13 · Cited by 2
Hasheminasab, S. A., Jamil, F., Afzal, M. U., Khan, A. H., Ilyas, S., Noor, A., Abbas, S. M., Cheema, H. N., Shabbir, M. U., Hameed, I., et al. (2024) · medRxiv · doi:10.1101/2024.06.05.24308365 · Cited by 7
“[15-17] In this paper, we studied the strengths and limitations of LLMs in a real-world, global South clinical setting.”
BackgroundApplying analytics to sociodemographic disparities in mental health
Baird, A. M., & Xia, Y. (2025) · Nature Mental Health · doi:10.1038/s44220-024-00359-2 · Cited by 4
Fairness Analysis in AI Algorithms in Healthcare: A Study on Post-Processing Approaches
Martini, V. G., & Berton, L. (2024) · doi:10.5753/eniac.2024.244467 · Cited by 1
McDermott, B., & Eaton, S. E. (2025) · Advances in business ethics research · doi:10.1007/978-3-031-77665-6_6 · Cited by 1
Matos, A. G. M. (2022) · EthAIca · doi:10.56294/ai202220 · Cited by 8
Gonzalez, L., & Dollarhide, L. (2025) · International Journal of Systemic Therapy · doi:10.1080/2692398x.2025.2549605 · Cited by 2
“Training AI tools with narrow and non-representative datasets can inadvertently provide clients with biased outputs that can directly affect mental healthcare (Straw et al., 2020; Timmons et al., 2023).”
Dritsona, A., Koutroumpa, M.-E., Sergentanis, T. Ν., & Τσίτσικα, Ά. (2025) · Advances in experimental medicine and biology · doi:10.1007/978-3-032-03394-9_45 · Cited by 4
Ibitoye, A., Oladimeji, O. O., Olaleye, I. O., & Emuoyibofarhe, O. N. (2026) · Data Science and Management · doi:10.1016/j.dsm.2025.12.002 · Cited by 3
Can AI developers avoid bias in public health applications?
Harms, R. J., Ankeny, R. A., Carter, L., Mankad, A., & Scully, J. L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1752729 · Cited by 1
“Responsibility is placed on developers to be aware of potential sources of bias and to be reflexive about their own biases (18, 19).”
Hasheminasab, S. A., Jamil, F., Afzal, M. U., Khan, A. H., Ilyas, S., Noor, A., Touseef, A., Abbas, S. M., Cheema, H. N., Shabbir, M. U., et al. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03366-8 · Cited by 5
Harms, R. J., Ankeny, R. A., Carter, L., Mankad, A., & Scully, J. L. (2026) · AI and Ethics · doi:10.1007/s43681-026-01088-1 · Cited by 3
“Accord - ing to Straw and Callison-Burch, developers should practice reflexivity when designing algorithms [ 47 ].”
Large Language Models for Clinical Narrative Processing: Methods, Applications, and Challenges
Livieratos, A., Lin, J., Chasani, P., Gaga, M., Fousekis, F. S., Gogos, C. A., Akinosoglou, K., Katsanos, K. H., & Gamalo, M. (2026) · Methods and Protocols · doi:10.3390/mps9030069 · Cited by 2
“Equally important to the successes of LLMs in healthcare are the challenges and safeguards needed for their responsible integration into clinical practice [15,16,78].”
InfluentialSuicide Prevention Technologies and Social Media Platforms: Legal, Social and Ethical ImplicationsPreprint
Stephenson, A., Cohen, G. A., Beller, Y., & Greenbaum, D. S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.4194066 · Cited by 2
Straw, I., Rees, G. E., & Nachev, P. (2023) · Research Square · doi:10.21203/rs.3.rs-2948898/v1 · Cited by 1
Self-citationEquity Readiness Index for Digital Health OrganizationsPreprint
Wagon, L., Nißen, M., & Kowatsch, T. (2025) · medRxiv · doi:10.1101/2025.11.14.25340230 · Cited by 1
Palakodety, S., KhudaBukhsh, A. R., & Jayachandran, G. (2021) · SpringerBriefs in computer science · doi:10.1007/978-981-16-5625-5_3
Kaur, G. (2023) · The British Journal of Psychiatry · doi:10.1192/bjp.2022.130
Using artificial intelligence in medical research: Challenge or opportunity?
Wang, X., & Li, J. (2023) · Asian Journal of Surgery · doi:10.1016/j.asjsur.2023.05.102
Hallucinated and spoken linguistic patterns as markers of psychiatric disorders
Hernández, H. C. (2024) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.872402816
Qin, Y. (2023) · IEEE International Conference on Electrical, Automation and Computer Engineering (ICEACE) · doi:10.1109/iceace60673.2023.10442889
“For a long time, simulated intelligence has been effectively executed in the field of applied sciences, however it is as of now tracking down its direction in a variety of disciplines, including the fields of social and behavioral sciences [32].”
BackgroundData access, data bias, data equity
Shung, D. L., & Laine, L. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95068-8.00002-9
Rubeis, G. (2024) · The International library of ethics, law and technology · doi:10.1007/978-3-031-55744-6_6
The Role of Generative AI-Assisted Literature Reviews in Transforming Academic Research
Vihari, N. S., & Kaur, A. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1798-3.ch006 · Cited by 2
Patient Attitudes Toward Artificial Intelligence in Cancer Care: A Scoping ReviewPreprint
Hilbers, D., Nekain, N., Bates, A. T., & Nuñez, J.-J. (2025) · medRxiv · doi:10.1101/2025.03.15.25324029
Keum, B. T., Pittman, D. M., Flores, L. Y., & Miller, M. J. (2025) · The Counseling Psychologist · doi:10.1177/00110000251350300
“…of these platforms often overlooks issues of social justice, equity, and regulatory oversight as they document issues of algorithmic and data-related biases that contribute to disparities in mental health services (Hopkins et al., 2024; Straw & Callison-Burch, 2020; Timmons et al., 2023).”
AI Technology and Bias in Detecting Psychiatric Disorders
Flowers, K. A. (2025) · Undergraduate Journal of Public Health · doi:10.3998/ujph.7619
Data Representativeness Through a Sociocultural Lens in AI Modeling for Mental Health
Karve, S., & Joeg, P. (2026) · Journal of Electrical Electronics Engineering · doi:10.33140/jeee.05.05.14
Transforming Mental Health Care through Clinical Decision Support System Surface Technology
Poonguzhali, C., Neethika, A., Yalagi, P. C. K., Praveen, R., Isaac, J. S., & Srivel, R. (2025) · International Conference on Communications and Signal Processing (ICCSP) · doi:10.1109/iccsp64183.2025.11089466
Khanmohammadi, R., Ghassemi, M. M., Verdecchia, K., Ghanem, A. I., Luo, B., Chetty, I. J., Bagher‐Ebadian, H., Siddiqui, F., Elshaikh, M. A., Movsas, B., et al. (2025) · BJR|Artificial Intelligence · doi:10.1093/bjrai/ubaf010 · Cited by 9
Responsible Artificial Intelligence for Healthcare: Trends, Challenges, and Opportunities
Baker, S., & Ahn, E. (2025) · Springer eBooks · doi:10.1007/978-3-031-88024-7_1
Foundations of Artificial Intelligence
Morelli, V. (2025) · Primary Care Clinics in Office Practice · doi:10.1016/j.pop.2025.07.001
Artificial Intelligence and Student Mental Health: Risks and Opportunities
Akat, M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-6375-2.ch002 · Cited by 2
Zahra, F. T., Sohail, A., Li, Z., & Sharaf, M. A. F. (2025) · Computational and Applied Mathematics · doi:10.1007/s40314-025-03519-x
Ravi, S. S., Bhagat, S., Paramasivam, S., & Yohannan, S. S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-6831-3.ch008
Artificial Intelligence in Emotional Recognition in Animals
Manoharan, G. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5483-5.ch003
Effectiveness of Chatbots in Mental Health Screening and Assessment: A Systematic Review (Preprint)Preprint
Morales, I., Marti-Estevez, I., Doval, S., Gutiérrez, J., Fausor, R., Arribas-García, S., Álvarez-Mon, M., & Quintero, J. (2026) · doi:10.2196/preprints.93672
Salley, C., Yoo, D., Chatell, M., Loos, S., Hall, S., & Wang, L. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00022
“Ex-panding to a more cross-disciplinary team will also help address the aforementioned biases by bringing in a broader range of perspectives and expertise [57].”
How Do LLMs Generate Emotional Metaphors? a Vector-Based Cognitive Investigation
Mao, A., Zhang, W., Mao, K., Peng, K., Bai, H., & Tong, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7144-4_10
Transforming Mental Health Care Through AI Technology
Yashwant, R. P. B., & Wong, E. H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/407379
Shirvani, M. S., Crowley, J., Peng, C., Liu, J., Chao, T. W.-C., Martinez, S., Brandt, L., Kim, I.-J., & Yoon, D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790986 · Cited by 4
“Cultural sensitivity is another key consideration for responsible design: SCs trained on limited linguistic or cultural norms may alienate users from underrepresented backgrounds [19, 109].”
Ferro, E., Castano-Villegas, N., Cárdenas, M., Gómez, A., Torres-Delgado, C., Calderon, L. O., Barrientos, K. M., & Zea, J. (2026) · medRxiv · doi:10.64898/2026.05.07.26352679
Rubeis, G. (2026) · Springer eBooks · doi:10.1007/978-3-032-14301-3_6
Azhar, M. H., Riaz, N., Azeem, W., Dewi, D. A., Amjad, A., & Arman, M. (2026) · Information · doi:10.3390/info17050496 · Cited by 3
EXPLORING THE MAGNITUDE OF ARTIFICIAL INTELLIGENCE ON FAMILY COMMUNICATION
Akinloye, D. N. (2026) · Journal International Review of Research Studies · doi:10.66104/7k305450 · Cited by 7
“These findings are different from the findings of Straw and Callison-Burch (2020) who reviewed and analyzed 52 studies and found gender bias in clinical data, data science, and linguistic perspective which is reflected in the AI algorithm called algorithm bias.”
PhysB: a framework for designing conversational agents to support physical activity in adolescents
Moore, R., Al-Tamimi, A.-K., & Lazuras, L. (2026) · Journal of Health Population and Nutrition · doi:10.1186/s41043-026-01254-5
“There are also several potential drawbacks to using LLMs in the design of CAs for adolescents [65, 67].”
Care in Pieces: Unpacking Experiences of Using LLMs for Self-management of Endometriosis and PCOS
Lucchini, A., Rojas, C. L., Bozzon, A., & Colombo, S. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3812813
“These conversational systems risk hallucinating information [2] which may spread misinformation and bias [36] for already complex conditions negatively impacting the health of users.”
Artificial Intelligence and Mental Health Discourse: Methodological Reflections for Nursing Practice
Wu, C.-J. J., & Hung, P. C. K. (2026) · Issues in Mental Health Nursing · doi:10.1080/01612840.2026.2665756
“This concern is especially relevant in mental health research, where language is highly context-dependent, shaped by stigma, cultural norms, and the subjective nature of lived experience (Straw & Callison-Burch, 2020).”
HarmoniqMind: AI-Powered Mental Equilibrium and Relationship Optimizer
Thilagam, T., Ram, S. K., & Karn, S. (2026) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn70047.2026.11594214
“Straw and Callison-Burch [16] identified demographic biases embedded within language-based models applied in psychiatric contexts, highlighting risks of unfair predictions.”
You must not fool yourself: Feynman, neurodiversity, and honest AI in digital mental health
RUTTENBERG, D. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1819411 · Cited by 2
“The neurodiversity paradigm holds that neurological variation — including autism, attention-deficit/hyperactivity disorder (ADHD), dyslexia, and related conditions — represents natural and valuable human diversity rather than deviation from a pathological norm (23, 24).”
InfluentialEffectiveness of Chatbots in Mental Health Screening and Assessment: Systematic Review
Morales, I., Marti-Estevez, I., Doval, S., Gutiérrez, J., Fausor, R., Arribas-García, S., Álvarez-Mon, M. Á., & Quintero, J. (2026) · JMIR Mental Health · doi:10.2196/93672
“Qualitative analyses reveal that race-explicit or race-implied patient information frequently results in inferior treatment recommendations from LLMs [7], and natural language processing models exhibit measurable biases related to religion, race, gender, nationality, sexuality, and age [31,32].”
Roy, A. K., Barbhuyan, M. F. K., Nath, S., Aanjankumar, S., Dhanaraj, R. K., & Poonkuntran, S. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44744-0.00007-2
Kamalraj, R., Akkamahadevi, C., Jena, D., & Yadav, A. (2026) · doi:10.1002/9781394392872.ch10
Ding, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-10720911/v1
Alhalawany, R. M., Khatatbeh, Y. M., & Jawkhab, A. A. (2026) · Healthcare · doi:10.3390/healthcare14172721
Ethics, computation, and institutional linkages in AI and mental-health research
Boudourides, M., & Männikkö, N. (2026) · Frontiers in Human Dynamics · doi:10.3389/fhumd.2026.1922040
Cho, C., & Kim, Y. (2026) · Psychiatry Research · doi:10.1016/j.psychres.2026.117528
Fermin, J. N. E. (2026) · The Canadian Journal of Psychiatry · doi:10.1177/07067437261496644
Balancing Innovation and Humanity: Advanced Practice Nurses Harnessing AI in Mental Health Care
Henry-Okafor, Q., Schroer, M. E., & Russell, R. G. (2026) · Journal of the American Psychiatric Nurses Association · doi:10.1177/10783903261499687
Effiong, E. N., & Gajere, J. B. (2026) · Research Square · doi:10.21203/rs.3.rs-11016638/v1
The opportunities and risks of large language models in mental health
Lawrence, H. R., Schneider, R. A., Rubin, S. B., Mataric, M. J., McDuff, D. J., & Bell, M. J. (2024) · arXiv · doi:10.48550/arxiv.2403.14814
“…supervision and evaluation can, therefore, lead to problematic generation of biased content and disparate model performance for different groups [45,55-57] (of note, however, there is some evidence that clinicians perceive less bias in LLM-generated responses [58] relative to clinician-generated…”
Background
Metadata
- Authors
- Straw, I., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Straw, Isabel
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-12-17
- Volume
- 15
- Issue
- 12
- Pages
- e0240376
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7745984
- OpenAlex
- W3112631310
- Cited by
- 164
- References
- 72
- Citations by year
- 2026: 40 · 2025: 46 · 2024: 37 · 2023: 23 · 2022: 12 · 2021: 6
- FWCI
- 13.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (2418758)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- algorithmic bias, natural language processing, mental health, word embeddings, gloves, Word2Vec, health disparities, psychiatric terminology, demographic bias, interdisciplinary collaboration
- MeSH terms
- Intersectoral Collaboration, Data Science, Humans, Linguistics, Mental Health, Natural Language Processing, Psychiatry, Bias, Health Status Disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.