The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
A toolbox for surfacing health equity harms and biases in large language models
Open accessAbstract
Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and our dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed by narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an artificial intelligence (AI) system promotes equitable health outcomes, we hope that it can be leveraged and built upon toward a shared goal of LLMs that promote accessible and equitable healthcare.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another subset of data in HealthBench is from physician red teaming of large language models in healthrelated settings, designed to identify queries where models have weaknesses or do not respond appropriately (Pfohl et al., 2024).”
From HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025) · cites this paper as Pfohl et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior research has highlighted the risk of LLMs perpetuating societal biases, potentially leading to harm across patient groups.45, 46, 47, 48”
From irAE-GPT: leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets (2026) · cites this paper as [47]
The cited source says what the claim attributes to it.
In the citing paper
“EquityMedQA introduces seven adversarial datasets and human evaluation rubrics to measure disparities across race, gender, and geography, revealing subtle inequities in LLM responses (Pfohl et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Pfohl et al., 2024
In the citing paper
“LLMs have demonstrated limited performance in clinical scenarios demanding flexible reasoning or information-seeking strategies23,24.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.64134 · Cited by 10,460
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,968
Structural racism and health inequities in the USA: evidence and interventions
Bailey, Z. D., Krieger, N., Agénor, M., Graves, J., Linos, N., & Bassett, M. T. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30569-x · Cited by 5,203
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citationBetter Bootstrap Confidence Intervals
Efron, B. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478410 · Cited by 3,462
Flagged first, then by citations
Research works printed but not linked: 4
- Randolph, J. J. Free-marginal multirater kappa (multirater Kfree): an alternative to Fleiss’ fixed-marginal multirater kappa. In Joensuu Learning and Instruction Symposium (2005).
- Wang, D. et al. All that agrees is not gold: evaluating ground truth labels and dialogue content for safety. Preprint at Google Research https://research.google/pubs/all-that-agrees-is-not-gold-evaluating-ground-truth-labels-and-dialogue-content-for-safety (2023).
- Johnson, T. P. Handbook of Health Survey Methods (Wiley Online Library, 2015).
- Kusner, M. J., Loftus, J., Russell, C. & Silva, R. Counterfactual fairness. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) (Curran Associates, 2017).
Cited by
Sorted by citations
Items 101–117 of 117
Zhang, J., Li, Y., Feng, X., Zhang, L., Huang, Z., & Chen, C. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818853
Abroms, L. C., Huntriss, R., Baqer, M., Carcone, A. I., Updegraff, J. A., Brennan, C., & Lahiri, S. (2026) · doi:10.2196/preprints.109169
Schulz, C., Adämmer, P., Fechner, E., Kassberg, E., Laux, I., Pollatscheck, E., Porsch, K., Romamti, L., Tymchuk, I., & Gräske, J. (2026) · BMC Health Services Research · doi:10.1186/s12913-026-15305-4
“Tools for systematically identifying bias and potential equity harms in large language model (LLM) outputs have been proposed, underscoring the relevance of discrimination-sen-sitive evaluation [13]. with the response of a physiotherapist to a real question from an online health forum.”
Gandhi, S., & Balas, M. (2026) · Journal of Personalized Medicine · doi:10.3390/jpm16090448
Raza, S., & Goel, R. (2026) · Vox Sanguinis · doi:10.1111/vox.70365
Wilson, E., Jong, F. D., & Peters, L. (2026) · Annals of Pharmacy Practice and Pharmacotherapy · doi:10.51847/xt3dqdt8kp
Xue, Z., Fu, K., Zhang, Y., Wu, B., & Wu, J. (2026) · Journal of Medical Internet Research · doi:10.2196/97852
“Empirical studies demonstrate significant degradation in performance, safety guardrails, and cultural competence when these models are queried in low-resource languages or dialects [9].”
Sex and gender bias in large language models: an old problem at a new scale
Barbati, C., Casigliani, V., Rizzo, C., & Odone, A. (2026) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2026-102380
“22 Participatory evaluation using adversarial red-teaming can further identify bias patterns that may be missed by conventional performance evaluation, 20 while structured stakeholder involvement can incorporate different perspectives into the definition and evaluation of potential harms.”
Systemic privacy risks of personal data exposure through conversational large language model agents
Yahia, A. B., Kadir, I., Harrak, E. F. E., Chisembe, S., Abdallaoui, A., Hmaidi, A. E., & Dehbi, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02431-5
“To help uncover these harms, along with measures to reduce them, tools such as health equity auditing [58] and red-teaming [59] are vital.”
Bressan, M., Meijer, E. N., Hinić, V., Ellendorff, T., Goldman, J.-P., Kew, T., Nolte, O., Schneider, G., Egli, A., Ahlström, M., et al. (2026) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2026.09.035
Goel, R., Raza, S., Russell, W. A., Seheult, J. N., Pham, T. D., D’Alessandro, A., Dubey, R., Montemayor, C., Daniel, D. B., Numark, C., et al. (2026) · Transfusion · doi:10.1111/trf.70418
Mitigating Domain Shift and Dataset Bias in Deep Learning Models for Cancer Diagnostics
Ameyaw, S. A., & Ameyaw, J. A. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02353-9
A scoping review of bias in large AI models for medicine and healthcare
Xiao, Y., Chen, Z., Huang, J.-T., Chen, W., Liu, Y., Li, K., Mousavi, M. R., Dobson, R., & Zhang, J. M. (2026) · Communications Health · doi:10.1038/s44528-026-00041-2
“Pfohl et al. 24 used two counterfactual datasets (CC-Manual and CC-LLM) to compare bias rates in Med-PaLM models as judged by physicians and health equity experts.”
InfluentialSelf-citationVariation in Medical Ethics Recommendations Across Large Language Models: Case-Based Audit Study
Mitsuyama, Y., Watanabe, K., Walston, S. L., Takita, H., Saito, K., Ishimaru, S., & Ueda, D. (2026) · JMIR AI · doi:10.2196/102713
“These cases provide structured templates for future counterfactual studies using prespecified matched pairs that differ only in 1 sensitive attribute and corresponding paired analyses [25].”
Li, Z., Wenjie, Z., Jiang, L., Wei, X., Zhou, J., Yang, W., Zhong, B., & Hu, Q. (2026) · Healthcare · doi:10.3390/healthcare14203549
Large language models show product name convergence in drug recommendations
Sato, J., Walston, S. L., Takita, H., Mitsuyama, Y., Saito, K., Wu, J., & Ueda, D. (2026) · Nature Health · doi:10.1038/s44360-026-00205-y
Cornelisse, E., Kousa, D., Menz, B. D., Murphy-Reuter, B., Shahnam, A., Chin-Yee, B. H., Calabro, R., Li, L. X., Ramsey, I., Abuhelwa, A. Y., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/100531 · Cited by 1
“These include dimensions such as consistency and robustness [8,14], equity-related harms, fairness or biases [93-96], deployment metrics (such as cost and latency) [8], regulatory compliance [96], trust [14,94], or user satisfaction [14,96].”
Metadata
- Authors
- Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., McCoy, L. G., Celi, L. A., Liu, Y., Schaekermann, M., Walton, A., Parrish, A., Nagpal, C., Singh, P., Dewitt, A., Mansfield, P., Prakash, S., Heller, K., Karthikesalingam, A., Semturs, C., Barral, J., Corrado, G., Matias, Y., Smith-Loud, J., Horn, I., & Singhal, K.
- Affiliations
- Google (United States) (US)
- University of Alberta (CA)
- Beth Israel Deaconess Medical Center (US)
- Corresponding author
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- ORCID
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Sayres, Rory
- Asiedu, Mercy
- Dieng, Awa
- Rashid, Qazi Mamunur
- Azizi, Shekoofeh
- Rostamzadeh, Negar
- McCoy, Liam G.
- Celi, Leo Anthony
- Liu, Yun
- Schaekermann, Mike
- Parrish, Alicia
- Nagpal, Chirag
- Singh, Preeti
- Dewitt, Akeiylah
- Mansfield, Philip
- Prakash, Sushant
- Heller, Katherine
- Karthikesalingam, Alan
- Semturs, Christopher
- Barral, Joelle
- Corrado, Greg
- Matias, Yossi
- Smith-Loud, Jamila
- Horn, Ivor
- Singhal, Karan
- Venue
- Nature Medicine
- Published
- 2024-09-23
- Volume
- 30
- Issue
- 12
- Pages
- 3590-3600
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11645264
- OpenAlex
- W4402748869
- Cited by
- 113
- References
- 66
- Citations by year
- 2026: 57 · 2025: 49 · 2024: 7
- FWCI
- 46.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- This study was funded by Google LLC and/or subsidiary thereof (Google).
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, health equity, algorithmic bias, health disparities, bias detection, human evaluation of AI, adversarial queries, participatory evaluation, health care equity
- MeSH terms
- Health Equity, Artificial Intelligence, Humans, Language, Bias, Healthcare Disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
