Fairness in AI-driven oncology: investigating racial and gender biases in large language models
Open accessAbstract
INTRODUCTION: Large language model (LLM) chatbots have many applications in medical settings. However, these tools can potentially perpetuate racial and gender biases through their responses, worsening disparities in healthcare. With the ongoing discussion of LLM chatbots in oncology and the widespread goal of addressing cancer disparities, this study focuses on biases propagated by LLM chatbots in oncology. METHODS: Chat Generative Pre-trained Transformer (Chat GPT; OpenAI, San Francisco, CA, USA) was asked to determine what occupation a generic description of "assesses cancer patients" would correspond to for different demographics. Chat GPT, Gemini (Alphabet Inc., Mountain View, CA, USA), and Bing Chat (Microsoft Corp., Redmond, WA, USA) were prompted to provide oncologist recommendations in the top U.S. cities and demographic information (race, gender) of recommendations was compared against national distributions. Chat GPT was also asked to generate a job description for oncologists with different demographic backgrounds. Finally, Chat GPT, Gemini, and Bing Chat were asked to generate hypothetical cancer patients with race, smoking, and drinking histories. RESULTS: LLM chatbots are about two times more likely to predict Blacks and Native Americans as oncology nurses than oncologists, compared to Asians (p < 0.01 and < 0.001, respectively). Similarly, they are also significantly more likely to predict females than males as oncology nurses (p < 0.001). Chat GPT's real-world oncologist recommendations overrepresent Asians by almost double and underrepresent Blacks by double and Hispanics by seven times. Chatbots also generate different job descriptions based on demographics, including cultural competency and advocacy and excluding treatment administration for underrepresented backgrounds. AI-generated cancer cases are not fully representative of real-world demographic distributions and encode stereotypes on substance abuse, such as Hispanics having a greater proportion of smokers than Whites by about 20% in Chat GPT breast cancer cases. CONCLUSION: To our knowledge, this is the first study of its kind to investigate racial and gender biases of such a diverse set of AI chatbots, and that too, within oncology. The methodology presented in this study provides a framework for targeted bias evaluation of LLMs in various fields across medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs and vision-language models trained on datasets that underrepresent specific demographic groups exhibit racial and gender biases in oncological recommendations.
“LLMs and vision-language models trained on datasets that underrepresent specific demographic groups have been shown to exhibit racial and gender biases in oncological recommendations”
From A generative AI multi-agent framework with integrated XAI governance for cancer diagnostics: from multi-omics interpretation to lifestyle risk stratification (2026) · cites this paper as Agrawal, 2024
The Results and Discussion report racial and gender biases in LLM-generated oncology job descriptions and oncologist recommendations. The study did not evaluate vision-language models or biases in cancer treatment recommendations; its Discussion identifies clinical plan generation as an area for future study. It suggests that training-data representation may contribute to bias but does not establish that the tested models were trained on datasets underrepresenting specific groups. (LLM, conf=0.90)
Experimental — accuracy not yet measuredLLM-based oncological tools exhibit racial and gender biases documented in the literature.
“LLM-based oncological tools exhibit racial and gender biases documented in the literature (Agrawal, 2024)”
From A generative AI multi-agent framework with integrated XAI governance for cancer diagnostics: from multi-omics interpretation to lifestyle risk stratification (2026) · cites this paper as Agrawal, 2024
LLM-based oncological tools exhibit racial and gender biases that could widen, rather than narrow, existing cancer care disparities.
“LLM-based oncological tools exhibit racial and gender biases that could widen, rather than narrow, existing cancer care disparities”
From A generative AI multi-agent framework with integrated XAI governance for cancer diagnostics: from multi-omics interpretation to lifestyle risk stratification (2026) · cites this paper as Agrawal, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1× · see citing works
“Importantly, existing literature has highlighted that algorithms overemphasize the representation of Blacks and Hispanics in generated disease cases [5,25].”
BackgroundChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628 · see citing works
Zack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00225-x · Cited by 510 · see citing works
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
Patel, M. I., Lopez, A. M., Blackstock, W., Reeder-Hayes, K., Moushey, E. A., Phillips, J., & Tap, W. (2020) · Journal of Clinical Oncology · doi:10.1200/jco.20.00642 · Cited by 338 · see citing works
ChatGPT: ethical concerns and challenges in academics and research
Guleria, A., Krishan, K., Sharma, V., & Kanchan, T. (2023) · The Journal of Infection in Developing Countries · doi:10.3855/jidc.18738 · Cited by 154 · see citing works
An overview of cancer health disparities: new approaches and insights and why they matter
Minas, T. Z., Bailey‐Whyte, M., Ajao, A., & Ambs, S. (2020) · Carcinogenesis · doi:10.1093/carcin/bgaa121 · Cited by 117 · see citing works
Perceived Discrimination and Substance Use among Latino Adolescents
Okamoto, J. M. (2009) · American Journal of Health Behavior · doi:10.5993/ajhb.33.6.9 · Cited by 107 · see citing works
Unmasking and quantifying racial bias of large language models in medical report generation
Yang, Y., Liu, X., Jin, Q., Huang, F., & Lu, Z. (2024) · Communications Medicine · doi:10.1038/s43856-024-00601-z · Cited by 105 · see citing works
Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation
Yang, Y., Liu, X., Jin, Q., Huang, F., & Lu, Z. (2024) · arXiv · doi:10.48550/arxiv.2401.13867 · Cited by 94 · see citing works
Ito, N., Kadomatsu, S., Fujisawa, M., Fukaguchi, K., Ishizawa, R., Kanda, N., Kasugai, D., Nakajima, M., Goto, T., & Tsugawa, Y. (2023) · JMIR Medical Education · doi:10.2196/47532 · Cited by 63 · see citing works
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63 · see citing works
Development of AI-generated medical responses using the ChatGPT for cancer patients
Lee, J. W., Yoo, I.-S., Kim, J., Kim, W. T., Jeon, H. J., Yoo, H.-S., Shin, J. G., Kim, G.-H., Hwang, S., Park, S., et al. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108302 · Cited by 39 · see citing works
Bias and Inaccuracy in AI Chatbot Ophthalmologist Recommendations
Oca, M. C., Meller, L. L. T., Wilson, K., Parikh, A. O., McCoy, A., Chang, J. Y., Sudharshan, R., Gupta, S., & Zhang-Nunes, S. (2023) · Cureus · doi:10.7759/cureus.45911 · Cited by 35 · see citing works
Mixed methods assessment of the influence of demographics on medical advice of ChatGPT
Andreadis, K., Newman, D. R., Twan, C., Shunk, A., Mann, D. M., & Stevens, E. R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae086 · Cited by 20 · see citing works
Sabado-Liwag, M., Zamora, M., & El‐Toukhy, S. (2022) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2022.04.009 · Cited by 14 · see citing works
Li, Y., Gao, W., Luan, Z., Zhou, Z., & Li, J. (2023) · Cureus · doi:10.7759/cureus.48670 · Cited by 11 · see citing works
Accuracy and Bias in Artificial Intelligence Chatbot Recommendations for Oculoplastic Surgeons
Parikh, A. O., Oca, M. C., Conger, J. R., McCoy, A., Chang, J. Y., & Zhang-Nunes, S. (2024) · Cureus · doi:10.7759/cureus.57611 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [23]Role of the oncology nurse Holland-Frei Cancer Medicine Rieger PT Yarbro CH Hamilton BC Decker 2003 https://www.ncbi.nlm.nih.gov/books/NBK13570/
- [24]The role of the oncologist Holland-Frei Cancer Medicine Moryl N Carver AC Foley KM Hamilton BC Decker 2003 https://www.ncbi.nlm.nih.gov/books/NBK13072/
Cited by
Sorted by citations
Barki, C., Chouchen, M., Sediri, A., Ceylan, H. İ., Rahmouni, H. B., Muntean, R. I., El-Seedi, H. R., Bragazzi, N. L., & Dergaa, I. (2026) · Frontiers in Systems Biology · doi:10.3389/fsysb.2026.1892418 · Cited by 1 · see citing works
“LLM-based oncological tools exhibit racial and gender biases documented in the literature (Agrawal, 2024), and lifestyle data from epidemiological studies systematically overrepresents high-income Western populations (Ktena et al., 2024; Marchesi et al., 2025).”
- Authors
- Agrawal, A.
- Affiliations
- The University of Texas at Austin (US)
- Corresponding author
- Agrawal, Anjali
- Venue
- Cureus
- Published
- 2024-09-16
- Volume
- 16
- Issue
- 9
- Pages
- e69541-e69541
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC11482645
- OpenAlex
- W4402786880
- Cited by
- 10
- References
- 18
- Citations by year
- 2026: 5 · 2025: 5
- FWCI
- 4.13
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Gender Studies › Diversity and Career in Medicine
- Keywords
- large language models, AI chatbots, racial bias, gender bias, oncology, cancer disparities, ChatGPT, Gemini, Bing Chat, stereotyping, health care disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
