Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation
Open accessAbstract
Large language models like GPT-3.5-turbo and GPT-4 hold promise for healthcare professionals, but they may inadvertently inherit biases during their training, potentially affecting their utility in medical applications. Despite few attempts in the past, the precise impact and extent of these biases remain uncertain. Through both qualitative and quantitative analyses, we find that these models tend to project higher costs and longer hospitalizations for White populations and exhibit optimistic views in challenging medical scenarios with much higher survival rates. These biases, which mirror real-world healthcare disparities, are evident in the generation of patient backgrounds, the association of specific diseases with certain races, and disparities in treatment recommendations, etc. Our findings underscore the critical need for future research to address and mitigate biases in language models, especially in critical healthcare applications, to ensure fair and accurate outcomes for all patients.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models are less likely to recommend advanced imaging for patients from underrepresented racial groups.
“LLMs being less likely to recommend advanced imaging for patients from underrepresented racial groups [6].”
From Evaluating and addressing demographic disparities in medical large language models: a systematic review (2025) · cites this paper as [6]
The Main section reports that generated medical reports favored White patients with superior treatments, including surgery for a White patient where a Black patient received conservative care. It does not report that LLMs were less likely to recommend advanced imaging for patients in other racial groups; searches for imaging, MRI, and CT did not identify such a finding. (LLM, conf=0.85)
Experimental — accuracy not yet measuredCommercially available large language models can perpetuate race-based medical misconceptions.
“Recent research has shown that commercially available LLMs can perpetuate race-based medical misconceptions [3, 5, 6].”
From Evaluating and addressing demographic disparities in medical large language models: a systematic review (2025) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 10
- OpenAI. GPT-4 Technical Report. Preprint at http://arxiv.org/abs/2303.08774 (2023). · arXiv
- Jin, Q., Wang, Z., Floudas, C. S., Sun, J. & Lu, Z. Matching Patients to Clinical Trials with Large Language Models. Preprint at https://doi.org/10.48550/arXiv.2307.15051 (2023). · DOI · Paper page
- Tian, S. et al. Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health. Preprint at https://doi.org/10.48550/arXiv.2306.10070 (2023). · DOI
- Zhuo, T. Y., Huang, Y., Chen, C. & Xing, Z. Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity. Preprint at https://doi.org/10.48550/arXiv.2301.12867 (2023). · DOI · Paper page
- Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How Does LLM Safety Training Fail? Preprint at http://arxiv.org/abs/2307.02483 (2023). · arXiv
- Hanna, J. J., Wakene, A. D., Lehmann, C. U. & Medford, R. J. Assessing Racial and Ethnic Bias in Text Generation for Healthcare-Related Tasks by ChatGPT1. medRxiv 2023.08.28.23294730 (2023) doi:10.1101/2023.08.28.23294730. · DOI · Paper page
- Sun, Z. et al. Evaluating GPT-4 on Impressions Generation in Radiology Reports. Radiology 307, e231259 (2023).
- Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on Medical Challenge Problems. Preprint at https://doi.org/10.48550/arXiv.2303.13375 (2023). · DOI · Paper page
- Dickman, S. L. et al. Trends in Health Care Use Among Black and White Persons in the US, 1963-2019. JAMA Netw. Open 5, e2217383 (2022).
- Zhao, Z., Jin, Q., Chen, F., Peng, T. & Yu, S. A large-scale dataset of patient summaries for retrieval-based clinical decision support systems. Sci. Data 10, 909 (2023).
Cited by
Sorted by citations
Omar, M., Sorin, V., Agbareia, R., Apakama, D. U., Soroush, A., Sakhuja, A., Freeman, R., Horowitz, C. R., Richardson, L. D., Nadkarni, G. N., et al. (2025) · International Journal for Equity in Health · doi:10.1186/s12939-025-02419-0 · Cited by 110 · see citing works
Fairness in AI-driven oncology: investigating racial and gender biases in large language models
Agrawal, A. (2024) · Cureus · doi:10.7759/cureus.69541 · Cited by 10 · see citing works
- Authors
- Yang, Y., Liu, X., Jin, Q., Huang, F., & Lu, Z.
- Affiliations
- National Institutes of Health (US)
- University of Maryland, College Park (US)
- Corresponding author
- Lu, Zhiyong
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Volume
- 4
- Issue
- 1
- Pages
- 176-176
- ISSN
- 2730-664X (ISSN-L)
- arXiv
- 2401.13867
- PMC
- PMC11387737
- OpenAlex
- W4402400689
- Cited by
- 94
- References
- 0
- Citations by year
- 2026: 35 · 2025: 53 · 2024: 6
- FWCI
- 38.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Communications Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by-nc-sa)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Health and Human Services
- National Institutes of Health
- U.S. National Library of Medicine
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- racial bias, large language models, medical report generation, health care disparities, algorithmic fairness, hospitalization prediction, healthcare cost prediction, mortality prediction, GPT-4, GPT-3.5-turbo, treatment guidelines, bias mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
