CoChat
Sign Up
Cited by 60Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)

Preventing harm from non-conscious bias in medical generative AI

Open access
Hastings, Janna·2024·The Lancet Digital Health
DOI 10.1016/s2589-7500(23)00246-7
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models such as OpenAI's GPT-4 have the potential to transform medicine1Clusmann J Kolbinger FR Muti HS et al.The future landscape of large language models in medicine.Commun Med (Lond). 2023; 3: 141Crossref PubMed Google Scholar by enabling automation of a range of tasks, including writing discharge summaries,2Patel SB Lam K ChatGPT: the future of discharge summaries?.Lancet Digit Health. 2023; 5: e107-e108Summary Full Text Full Text PDF PubMed Scopus (159) Google Scholar answering patient questions,3Ayers JW Poliak A Dredze M et al.Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.JAMA Intern Med. 2023; 183: 589-596Crossref PubMed Scopus (156) Google Scholar and supporting clinical treatment planning.4Dennstädt F Hastings J Putora PM et al.Exploring the capabilities of large language models such as ChatGPT in radiation oncology.Adv Radiat Oncol. 2023; (published online Nov 4.)https://doi.org/10.1016/j.adro.2023.101400Summary Full Text Full Text PDF Google Scholar These models are so useful that their adoption has been immediate, and efforts are already well underway to integrate them with ubiquitous clinical information systems. However, the unchecked use of the technology has the potential to cause harm. An ambitious study by Travis Zack and Eric Lehman and colleagues5Zack T Lehman E Suzgun M et al.Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.Lancet Digit Health. 2024; 6: e12-e22Summary Full Text Full Text PDF Google Scholar in The Lancet Digital Health comprehensively shows that GPT-4 exhibits racial and gender bias across clinically relevant tasks, including the generation of cases for medical education, support for differential diagnostic reasoning, medical plan recommendation, and subjective assessments of patients. For each of these tasks, GPT-4 was found to exaggerate known disease prevalence differences between groups, over-represent stereotypes including problematic representations of minority groups, and amplify harmful societal biases. These findings are seriously concerning and are in line with previous research about bias in large-scale generative artificial intelligence models more broadly. However, the study falls short of providing actionable recommendations on how the technology can safely be incorporated into clinical workflows given these findings. Strategies to mitigate bias in language models are a very active research area, commonly divided into approaches that augment or alter the model training data, strategies that enhance or provide additional model training, strategies that alter the model execution, and strategies that apply corrections post-processing.6Gallegos IO Rossi RA Barrow J et al.Bias and fairness in large language models: a survey.arXiv. 2023; (published online Sept 2.) (preprint).https://doi.org/10.48550/arXiv.2309.00770Google Scholar Supplementing such models with additional training through data augmentation or additional reinforcement learning from human feedback might help to some extent, but are unlikely to be completely successful as it is largely outside of the potential of the underlying technology to avoid bias completely. In addition, as Zack and colleagues5Zack T Lehman E Suzgun M et al.Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.Lancet Digit Health. 2024; 6: e12-e22Summary Full Text Full Text PDF Google Scholar note, hospital information systems adopting a closed model such as GPT-4 have limited potential to redevelop the model itself, and even though this is in principle possible with open-source models, it is unlikely that individual hospitals have sufficient resources to resolve all the underlying problems. Thus, it follows that biased models will be deployed. Attempts to debias the results through instruction are also likely to fail: language models have no possibility for self-awareness or self-reflection, and no consistent model of the world to which their outputs can be expected to correspond. Thus, as Zack and colleagues5Zack T Lehman E Suzgun M et al.Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.Lancet Digit Health. 2024; 6: e12-e22Summary Full Text Full Text PDF Google Scholar observed in their study, explicitly instructing the model to perform equitably or to avoid bias will probably not have the desired outcome, and might result in the model overcorrecting to a worse bias. Worryingly, even deployment with a clinician in the loop will not fully address the problem—biased model behaviour might lead the clinician to become more biased, rather than the desired outcome of the clinician reducing model bias.7Vicente L Matute H Humans inherit artificial intelligence biases.Sci Rep. 2023; 1315737Crossref Scopus (0) Google Scholar How, then, can these models be safe

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/s2589-7500(23)00246-7

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Hastings, J.
Affiliations
University of Zurich (CH)
Corresponding author
Hastings, Janna
ORCID
Hastings, Janna
Venue
The Lancet Digital Health
Publisher
Elsevier BV
Published
2024-01
Volume
6
Issue
1
Pages
e2-e3
ISSN
2589-7500 (ISSN-L)
DOI
10.1016/s2589-7500(23)00246-7
OpenAlex
W4389898530
Cited by
60
References
9
Citations by year
2026: 7 · 2025: 24 · 2024: 28 · 2023: 1
FWCI
2.43
Type
journal article
Language
en
Open access
Licence
Locations
  • The Lancet Digital Health (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, GPT-4, algorithmic bias, racial bias, gender bias, health care disparities, bias mitigation, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review