CoChat
Sign Up
Cited by 117Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)

The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.

A toolbox for surfacing health equity harms and biases in large language models

Open access
Pfohl, Stephen R., Cole-Lewis, Heather, Sayres, Rory, Neal, Darlene, Asiedu, Mercy, Dieng, Awa, Tomasev, Nenad, Rashid, Qazi Mamunur, Azizi, Shekoofeh, Rostamzadeh, Negar, McCoy, Liam G., Celi, Leo Anthony, Liu, Yun, Schaekermann, Mike, Walton, Alanna, Parrish, Alicia, Nagpal, Chirag, Singh, Preeti, Dewitt, Akeiylah, Mansfield, Philip, Prakash, Sushant, Heller, Katherine, Karthikesalingam, Alan, Semturs, Christopher, Barral, Joelle, Corrado, Greg, Matias, Yossi, Smith-Loud, Jamila, Horn, Ivor, Singhal, Karan·2024·Nature Medicine
DOI 10.1038/s41591-024-03258-2PMC PMC11645264
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and our dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed by narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an artificial intelligence (AI) system promotes equitable health outcomes, we hope that it can be leveraged and built upon toward a shared goal of LLMs that promote accessible and equitable healthcare.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03258-2

Cites

78 of 78 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of our main contributions. We employ an iterative, participatory approach to design human assessment rubrics for surfacing health equity harms and biases; introduce EquityMedQA, a collection of seven newly released adversarial medical question-answering datasets enriched for equity-related content that substantially expands upon the volume and breadth of previously studied adversarial data for medical question answering; and perform a large scale empirical study of health equity-related biases in LLMs.
Fig. 1 Overview of our main contributions. We employ an iterative, participatory approach to design human assessment rubrics for surfacing health equity harms and biases; introduce EquityMedQA, a collection of seven newly released adversarial medical question-answering datasets enriched for equity-related content that substantially expands upon the volume and breadth of previously studied adversarial data for medical question answering; and perform a large scale empirical study of health equity-related biases in LLMs.CC BY · PubMed Central
Fig. 2 Results of independent evaluation of bias in Med-PaLM 2 answers. We report the rate at which raters reported minor or severe bias in Med-PaLM 2 answers for physician and health equity expert raters for each dataset and dimension of bias. The numbers of answers rated for each dataset are reported in Table 2 and the Methods. Statistics for multiply rated datasets (Mixed MMQA–OMAQ and Omiye et al.) were computed with pooling over replicates with the level of replication indicated in parentheses. Data are reported as proportions with 95% CIs.
Fig. 2 Results of independent evaluation of bias in Med-PaLM 2 answers. We report the rate at which raters reported minor or severe bias in Med-PaLM 2 answers for physician and health equity expert raters for each dataset and dimension of bias. The numbers of answers rated for each dataset are reported in Table 2 and the Methods. Statistics for multiply rated datasets (Mixed MMQA–OMAQ and Omiye et al.) were computed with pooling over replicates with the level of replication indicated in parentheses. Data are reported as proportions with 95% CIs.CC BY · PubMed Central
Fig. 3 Results of pairwise evaluation of Med-PaLM 2 answers compared to Med-PaLM and physician answers. We report the rates at which raters reported a lesser degree of bias in Med-PaLM 2 answers versus comparator answers across datasets, rater types and dimensions of bias. The numbers of answers rated for each dataset are reported in Table 2 and the Methods. The comparator is Med-PaLM in all cases except for the case of physician-written answers to HealthSearchQA questions. Data are reported as proportions with 95% CIs.
Fig. 3 Results of pairwise evaluation of Med-PaLM 2 answers compared to Med-PaLM and physician answers. We report the rates at which raters reported a lesser degree of bias in Med-PaLM 2 answers versus comparator answers across datasets, rater types and dimensions of bias. The numbers of answers rated for each dataset are reported in Table 2 and the Methods. The comparator is Med-PaLM in all cases except for the case of physician-written answers to HealthSearchQA questions. Data are reported as proportions with 95% CIs.CC BY · PubMed Central
Fig. 4 Results of counterfactual and independent evaluation on counterfactual datasets. In the top four rows, we report the rates at which raters reported bias in counterfactual pairs using the proposed counterfactual rubric as well as the rates at which they reported bias in one, one or more or both of the answers using the independent evaluation rubric for the CC-Manual (n = 102 pairs, triple replication) and the CC-LLM datasets (n = 200 pairs). For comparison, the bottom row reports independent evaluation results aggregated across all unpaired questions for the CC-Manual (n = 42) and CC-LLM (n = 100) datasets. Data are reported as proportions with 95% CIs.
Fig. 4 Results of counterfactual and independent evaluation on counterfactual datasets. In the top four rows, we report the rates at which raters reported bias in counterfactual pairs using the proposed counterfactual rubric as well as the rates at which they reported bias in one, one or more or both of the answers using the independent evaluation rubric for the CC-Manual (n = 102 pairs, triple replication) and the CC-LLM datasets (n = 200 pairs). For comparison, the bottom row reports independent evaluation results aggregated across all unpaired questions for the CC-Manual (n = 42) and CC-LLM (n = 100) datasets. Data are reported as proportions with 95% CIs.CC BY · PubMed Central
Extended Data Fig. 1 Summary of pairwise comparisons of counterfactual answers. Results are pooled over the CC-Manual (n = 102 pairs, triple replicaIon) and CC-LLM (n = 200 pairs) datasets. Top, the counts of the number of pairs of Med-PaLM 2 answers to counterfactual questions reported for each category of answer similarity, stratified by rater group and whether the ideal answers to the counterfactual quesIons were reported to be (left) the same or (right) different. Bottom, the rate that counterfactual pairs are reported as containing bias for cases where the ideal answers were reported to be (left) the same and (right) different, stratified by rater group and the reported category of answer similarity. Data are reported as proporIons with 95% confidence intervals.
Extended Data Fig. 1 Summary of pairwise comparisons of counterfactual answers. Results are pooled over the CC-Manual (n = 102 pairs, triple replicaIon) and CC-LLM (n = 200 pairs) datasets. Top, the counts of the number of pairs of Med-PaLM 2 answers to counterfactual questions reported for each category of answer similarity, stratified by rater group and whether the ideal answers to the counterfactual quesIons were reported to be (left) the same or (right) different. Bottom, the rate that counterfactual pairs are reported as containing bias for cases where the ideal answers were reported to be (left) the same and (right) different, stratified by rater group and the reported category of answer similarity. Data are reported as proporIons with 95% confidence intervals.CC BY · PubMed Central
Extended Data Fig. 2 Evaluation of answers to the questions from Omiye et al.23. In (A) we show the number of raters, out of five, that reported the presence of bias and its dimensions across the nine questions for the independent rubric. In (B), we show the results of pairwise evaluation of Med-PaLM and Med-PaLM 2 answers to the questions for each question and rater.
Extended Data Fig. 2 Evaluation of answers to the questions from Omiye et al.23. In (A) we show the number of raters, out of five, that reported the presence of bias and its dimensions across the nine questions for the independent rubric. In (B), we show the results of pairwise evaluation of Med-PaLM and Med-PaLM 2 answers to the questions for each question and rater.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 66 references

Research works printed but not linked: 4

  1. Randolph, J. J. Free-marginal multirater kappa (multirater Kfree): an alternative to Fleiss’ fixed-marginal multirater kappa. In Joensuu Learning and Instruction Symposium (2005).
  2. Wang, D. et al. All that agrees is not gold: evaluating ground truth labels and dialogue content for safety. Preprint at Google Research https://research.google/pubs/all-that-agrees-is-not-gold-evaluating-ground-truth-labels-and-dialogue-content-for-safety (2023).
  3. Johnson, T. P. Handbook of Health Survey Methods (Wiley Online Library, 2015).
  4. Kusner, M. J., Loftus, J., Russell, C. & Silva, R. Counterfactual fairness. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) (Curran Associates, 2017).

Cited by

Sorted by citations

See all 117

Related

Metadata

Authors
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., McCoy, L. G., Celi, L. A., Liu, Y., Schaekermann, M., Walton, A., Parrish, A., Nagpal, C., Singh, P., Dewitt, A., Mansfield, P., Prakash, S., Heller, K., Karthikesalingam, A., Semturs, C., Barral, J., Corrado, G., Matias, Y., Smith-Loud, J., Horn, I., & Singhal, K.
Affiliations
  • Google (United States) (US)
  • University of Alberta (CA)
  • Beth Israel Deaconess Medical Center (US)
Corresponding author
  • Pfohl, Stephen R.
  • Cole-Lewis, Heather
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-09-23
Volume
30
Issue
12
Pages
3590-3600
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-024-03258-2
PMC
PMC11645264
OpenAlex
W4402748869
Cited by
113
References
66
Citations by year
2026: 57 · 2025: 49 · 2024: 7
FWCI
46.83
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • This study was funded by Google LLC and/or subsidiary thereof (Google).
  • Google
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, health equity, algorithmic bias, health disparities, bias detection, human evaluation of AI, adversarial queries, participatory evaluation, health care equity
MeSH terms
Health Equity, Artificial Intelligence, Humans, Language, Bias, Healthcare Disparities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review