Centering health equity in large language model deployment
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Early buzz around ChatGPT's [1] proficiency at healthcare tasks [2][3][4] has turned into rapid deployment of large language model (LLM) technology across healthcare.Multiple healthcare systems including UC San Diego Health, Stanford Health Care, and University of Wisconsin Health are already piloting the use of GPT-4 to draft responses to patient messages [5].Others, including The University of Kansas Health System, are piloting similar LLMs to support clinical documentation [6].Early conversations about LLMs in healthcare tended towards extremes, either lauding their breathtaking potential or lamenting the threat they pose to human-centric care.More recently, nuanced narratives regarding the ethical implications of LLMs have started to emerge [7,8].Still missing from the conversation, though, is a grounded discussion of the pragmatic equity implications of LLM deployment in real-world contexts.In this perspective, we explore various health equity risks generated by the practical use of LLMs in healthcare and offer risk mitigation recommendations.We define health equity using Dr. Braveman's definition: the "absence of differences in health between the most advantaged group in a given category and all others" [9].Equity risks originating in the development of LLMs are widely discussed, with substantial dialogue dedicated to intrinsic equity and general ethical considerations of how, why, and with what data and algorithmic design choices these models should be developed [10,11].Equity risks in AI model deployment are less commonly discussed.These emergent risks arise from pragmatic use by end users (patients, clinicians, and healthcare systems) in real-world settings.It is critical to understand emergent equity risks of LLMs to avoid replicating inequities that resulted from the use of other technologies in medicine [12].For patients, LLMs have a remarkable ability to generate human-like responses and summarize complex or hard-to-find information [2,4], providing automated care with potential to alleviate disparities in access, understandability, and personalization.However, if minoritized populations who have experienced barriers to accessing traditional care are disproportionately more likely to seek care from a free chatbot rather than a licensed provider this could result in a uniquely biased care experience-with higher data privacy risks [13], biased outputs, misinformation, or personally offensive content from models trained on biased data [14][15][16].This could perpetuate negative care experiences, medical disenfranchisement and distrust.For clinicians, LLM-supported tools offer efficiency gains that may help address issues of burnout and dissatisfaction.For example, DocsGPT [17] automates high-burden, low-cognition tasks such as composing prior authorization letters.However, the choice to use LLMs versus not based on patient language, health literacy, or insurance coverage could widen existing
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“Prompting enhances the performance of general-purpose models on healthcare tasks [18–20] and can change model accuracy, safety, and bias [18].”
BackgroundBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1×
“…that adopt widespread direct-to-patient LLM strategies that remove clinicians entirely from patient exchanges also risk impersonal, inaccurate, or biased [14–16] communications that would further ostracize minoritized patients from their providers and the experience of care delivery in general.”
BackgroundHealth disparities and health equity: Concepts and measurement
Braveman, P. (2006) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.27.021405.102103 · Cited by 1,570
Harrer, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104512 · Cited by 480
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343
Cited by
A toolbox for surfacing health equity harms and biases in large language models
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03258-2 · Cited by 113
Wan, P., Huang, Z., Tang, W., Nie, Y., Pei, D., Deng, S., Chen, J., Zhou, Y., Duan, H., Chen, Q., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03148-7 · Cited by 97
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63
Self-citationMixed methods assessment of the influence of demographics on medical advice of ChatGPT
Andreadis, K., Newman, D. R., Twan, C., Shunk, A., Mann, D. M., & Stevens, E. R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae086 · Cited by 20
Self-citationEvaluating and Mitigating Discrimination in Language Model Decisions
Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., & Ganguli, D. (2023) · arXiv · doi:10.48550/arxiv.2312.03689 · Cited by 11
“…may choose to limit the use of language models for such decisions, it remains important to proactively anticipate and mitigate such potential risks as early as possible, especially given that initial deployments in high-stakes settings have already begun (Wodzak, 2022; Singh et al., 2023).”
Background
Metadata
- Authors
- Singh, N., Lawrence, K., Richardson, S., & Mann, D. M.
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-10-24
- Volume
- 2
- Issue
- 10
- Pages
- e0000367
- ISSN
- 2767-3170 (ISSN-L)
- OpenAlex
- W4387902603
- Cited by
- 25
- References
- 14
- FWCI
- 0.86
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.