A toolbox for surfacing health equity harms and biases in large language models
Open accessAbstract
Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and our dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed by narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an artificial intelligence (AI) system promotes equitable health outcomes, we hope that it can be leveraged and built upon toward a shared goal of LLMs that promote accessible and equitable healthcare.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Another subset of HealthBench data is from physician red teaming of large language models in health-related settings, designed to identify queries where models have weaknesses or do not respond appropriately.
“Another subset of data in HealthBench is from physician red teaming of large language models in healthrelated settings, designed to identify queries where models have weaknesses or do not respond appropriately (Pfohl et al., 2024).”
From HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025) · cites this paper as Pfohl et al., 2024
The Methods (EquityMedQA; no page available) describe human-written medical questions designed to target observed model failures, using responses that physicians had flagged as biased. The Results (EquityMedQA; no page available) call this a manual red-teaming dataset. Our copy does not establish that these questions became a subset of HealthBench, and it does not specify that physicians wrote the red-teaming questions. (LLM, conf=0.84)
Experimental — accuracy not yet measuredPrior research has highlighted the risk of LLMs perpetuating societal biases, potentially leading to harm across patient groups.
“Prior research has highlighted the risk of LLMs perpetuating societal biases, potentially leading to harm across patient groups.45, 46, 47, 48”
From irAE-GPT: leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets (2026) · cites this paper as [47]
EquityMedQA introduces seven adversarial datasets and human evaluation rubrics to measure disparities across race, gender, and geography, revealing subtle inequities in LLM responses.
“EquityMedQA introduces seven adversarial datasets and human evaluation rubrics to measure disparities across race, gender, and geography, revealing subtle inequities in LLM responses (Pfohl et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Pfohl et al., 2024
LLMs have demonstrated limited performance in clinical scenarios demanding flexible reasoning or information-seeking strategies.
“LLMs have demonstrated limited performance in clinical scenarios demanding flexible reasoning or information-seeking strategies23,24.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.64134 · Cited by 10,460 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,673 · see citing works
Structural racism and health inequities in the USA: evidence and interventions
Bailey, Z. D., Krieger, N., Agénor, M., Graves, J., Linos, N., & Bassett, M. T. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30569-x · Cited by 5,203 · see citing works
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695 · see citing works
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Self-citationBetter Bootstrap Confidence Intervals
Efron, B. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478410 · Cited by 3,462 · see citing works
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Self-citationRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 1,372 · see citing works
Self-citationThe future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · see citing works
Understanding how discrimination can affect health
Williams, D. R., Lawrence, J. A., Davis, B. A., & Vu, C. (2019) · Health Services Research · doi:10.1111/1475-6773.13222 · Cited by 853 · see citing works
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
Ethical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645 · see citing works
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550 · see citing works
Estimating the Reliability, Systematic Error and Random Error of Interval Data
Krippendorff, K. (1970) · Educational and Psychological Measurement · doi:10.1177/001316447003000105 · Cited by 511 · see citing works
Zack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00225-x · Cited by 510 · see citing works
Self-citationTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483 · see citing works
Self-citationTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · see citing works
Self-citationHarrer, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104512 · Cited by 480 · see citing works
Field, C., & Welsh, A. H. (2007) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00593.x · Cited by 466 · see citing works
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · see citing works
Algorithmic Colonization of Africa
Birhane, A. (2020) · SCRIPTed: A Journal of Law, Technology & Society · doi:10.2966/scrip.170220.389 · Cited by 349 · see citing works
Considerations for addressing bias in artificial intelligence for health equity
Abràmoff, M. D., Tarver, M. E., Loyo-Berrios, N., Trujillo, S., Char, D., Obermeyer, Z., Eydelman, M. B., Foundational Principles of Ophthalmic Imaging and Algorithmic Interpretation Working Group of the Collaborative Community for Ophthalmic Imaging Foundation, W., & Maisel, W. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00913-9 · Cited by 349 · see citing works
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Self-citationPower to the People? Opportunities and Challenges for Participatory AI
Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022) · Equity and Access in Algorithms Mechanisms and Optimization · doi:10.1145/3551624.3555290 · Cited by 330 · see citing works
The value of standards for health datasets in artificial intelligence-based applications
Arora, A., Alderman, J. E., Palmer, J., Ganapathi, S., Laws, E., McCradden, M. D., Oakden‐Rayner, L., Pfohl, S. R., Ghassemi, M., McKay, F., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02608-w · Cited by 312 · see citing works
Self-citationTowards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · see citing works
Self-citationTowards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
Self-citationProblem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · see citing works
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279 · see citing works
Self-citationLiu, X., Glocker, B., McCradden, M. M., Ghassemi, M., Denniston, A. K., & Oakden-Rayner, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00003-6 · Cited by 240 · see citing works
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · see citing works
Counterfactual Fairness in Text Classification through Robustness
Garg, S., Perot, V., Limtiaco, N., Taly, A., Ed, H., & Beutel, A. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3317950 · Cited by 200 · see citing works
Red-Teaming for Generative AI: Silver Bullet or Security Theater?Preprint
Feffer, M., Sinha, A., Deng, W., Lipton, Z. C., & Heidari, H. (2024) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.48550/arxiv.2401.15897 · Cited by 186 · see citing works
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · see citing works
Self-citationAn empirical characterization of fair machine learning for clinical risk prediction
Pfohl, S. R., Foryciarz, A., & Shah, N. H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103621 · Cited by 164 · see citing works
Self-citationStereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.81 · Cited by 147 · see citing works
Considering Biased Data as Informative Artifacts in AI-Assisted Health Care
Ferryman, K. S., Mackintosh, M. E., & Ghassemi, M. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2214964 · Cited by 142 · see citing works
Health inequities and the inappropriate use of race in nephrology
Eneanya, N. D., Boulware, L. E., Tsai, J., Bruce, M. A., Ford, C. L., Harris, C., Morales, L. S., Ryan, M. J., Reese, P. P., Thorpe, R. J., et al. (2022) · Nature Reviews Nephrology · doi:10.1038/s41581-021-00501-8 · Cited by 140 · see citing works
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · see citing works
How Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution?
Quarfoot, D., & Levine, R. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1141708 · Cited by 121 · see citing works
Perturbation Sensitivity Analysis to Detect Unintended Model Biases
Prabhakaran, V., Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1578 · Cited by 115 · see citing works
Targeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 115 · see citing works
Med-Flamingo: a Multimodal Medical Few-shot LearnerPreprint
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Zakka, C., Dalmia, Y., Reis, E. P., Rajpurkar, P., & Leskovec, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.15189 · Cited by 111 · see citing works
Cary, M. P., Zink, A., Wei, S., Olson, A., Yan, M., Senior, R., Bessias, S., Gadhoumi, K., Jean-Pierre, G., Wang, D., et al. (2023) · Health Affairs · doi:10.1377/hlthaff.2023.00553 · Cited by 94 · see citing works
Comparing Bayesian Models of Annotation
Paun, S., Carpenter, B., Chamberlain, J., Hovy, D., Kruschwitz, U., & Poesio, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00040 · Cited by 86 · see citing works
Red-Teaming for Generative AI: Silver Bullet or Security Theater?
Feffer, M., Sinha, A., Deng, W. H., Lipton, Z. C., & Heidari, H. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31647 · Cited by 75 · see citing works
Sociotechnical Safety Evaluation of Generative AI Systems
Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11986 · Cited by 49 · see citing works
Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers
Abacha, A. B., Mrabet, Y., Sharp, M. J., Goodwin, T. R., Shooshan, S. E., & Demner‐Fushman, D. (2019) · Studies in health technology and informatics · doi:10.3233/shti190176 · Cited by 44 · see citing works
Large Language Models are Few-Shot Health LearnersPreprint
Liu, X., McDuff, D., Kovacs, G., Galatzer‐Levy, I. R., Sunshine, J. E., Zhan, J., Poh, M., Liao, S., Achille, P. D., & Patel, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15525 · Cited by 41 · see citing works
LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J., Reif, E., Kallarackal, K., Chang, M., Terry, M. A., & Dixon, L. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650755 · Cited by 36 · see citing works
LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J., Reif, E., Kallarackal, K., Chang, M., Terry, M., & Dixon, L. (2024) · arXiv · doi:10.48550/arxiv.2402.10524 · Cited by 34 · see citing works
Documenting Data Production Processes: A Participatory Approach for Data Work
Miceli, M., Yang, T., Alvarado Garcia, A., Posada, J., Wang, S. M., Pohl, M., & Hanna, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555623 · Cited by 28 · see citing works
Centering health equity in large language model deployment
Singh, N. P., Lawrence, K. S., Richardson, S. I., & Mann, D. M. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000367 · Cited by 28 · see citing works
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm ReductionPreprint
Shelby, R. M., Rismani, S., Henne, K., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.05791 · Cited by 22 · see citing works
Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability
Wong, K., Paritosh, P. K., & Aroyo, L. M. (2021) · doi:10.18653/v1/2021.acl-long.548 · Cited by 15 · see citing works
Documenting Data Production Processes: A Participatory Approach for Data WorkPreprint
Miceli, M., Yang, T., Garcia, A. A., Posada, J., Wang, S. M., Pohl, M., & Hanna, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.04958 · Cited by 7 · see citing works
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives
Prabhakaran, V., Homan, C. M., Aroyo, L. M., Davani, A. M., Parrish, A., Taylor, A. S., Díaz, M., Wang, D., & Serapio‐García, G. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.190 · Cited by 5 · see citing works
Self-citationSmith-Loud, J., Smart, A., Neal, D., Ebinama, A., Corbett, E., Nicholas, P., Rashid, Q. M., Peckham, A., Murphy-Gray, S., Morris, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.08177 · Cited by 4 · see citing works
Self-citationThe Case for Globalizing Fairness: A Mixed Methods Study on Colonialism, AI, and Health in AfricaPreprint
Asiedu, M., Dieng, A., Haykel, I., Rostamzadeh, N., Pfohl, S., Nagpal, C., Nagawa, M., Oppong, A., Koyejo, S., & Heller, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03357 · Cited by 4 · see citing works
Self-citationA Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models
Pfohl, S., Cole-Lewis, H. J., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26133973 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 7
- [41]Randolph, J. J. Free-marginal multirater kappa (multirater Kfree): an alternative to Fleiss’ fixed-marginal multirater kappa. In Joensuu Learning and Instruction Symposium (2005).
- [45]Homan, C. M. et al. Intersectionality in AI safety: using multilevel models to understand diverse perceptions of safety in conversational AI. In Proc. of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024 (eds Abercrombie, G. et al.) 131–141 (2024).
- [48]Wang, D. et al. All that agrees is not gold: evaluating ground truth labels and dialogue content for safety. Preprint at Google Research https://research.google/pubs/all-that-agrees-is-not-gold-evaluating-ground-truth-labels-and-dialogue-content-for-safety (2023).
- [51]Johnson, T. P. Handbook of Health Survey Methods (Wiley Online Library, 2015).
- [52]Harkness, J. A. et al. In Comparative Survey Methodology Ch. 1, 1–16 (John Wiley & Sons, 2010).
- [64]Kusner, M. J., Loftus, J., Russell, C. & Silva, R. Counterfactual fairness. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) (Curran Associates, 2017).
- [68]Ben Abacha, A., Agichtein, E., Pinter, Y. & Demner-Fushman, D. Overview of the medical question answering task at TREC 2017 LiveQA. In Text Retrieval Conference 2017 (2017).
Cited by
Sorted by citations
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Self-citationSociodemographic biases in medical decision making by large language models
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03626-6 · Cited by 173 · see citing works
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83 · see citing works
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Participation in the age of foundation models
Suresh, H., Tseng, E., Young, M., Gray, M. L., Pierson, E., & Levy, K. E. C. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658992 · Cited by 56 · see citing works
Bouguettaya, A., Stuart, E. M., & Aboujaoude, E. N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01746-4 · Cited by 53 · see citing works
Zhou, J., Zhang, J., Wan, R., Cui, X., Liu, Q., Guo, H., Shi, X., Fu, B., Meng, J., Yue, B., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06916-2 · Cited by 55 · see citing works
Savage, C., Kanhere, A. U., Parekh, V. S., Langlotz, C. P., Joshi, A., Huang, H., & Doo, F. X. (2025) · Radiology · doi:10.1148/radiol.241073 · Cited by 57 · see citing works
Collaborative, T. C. (2025) · BMJ · doi:10.1136/bmj-2024-083305 · Cited by 50 · see citing works
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 45 · see citing works
Generative artificial intelligence–mediated confirmation bias in health information seeking
Lopez‐Lopez, E., Abels, C. M., Holford, D., Herzog, S. M., & Lewandowsky, S. (2025) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15413 · Cited by 38 · see citing works
Unlocking new frontiers in epilepsy through AI: From seizure prediction to personalized medicine
AbuAlrob, M. A., Itbaisha, A., & Mesraoua, B. (2025) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2025.110327 · Cited by 32 · see citing works
Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Grabb, D., Lamparth, M., & Vasan, N. (2024) · doi:10.1101/2024.04.07.24305462 · Cited by 40 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
“Another subset of data in HealthBench is from physician red teaming of large language models in health-related settings, designed to identify queries where models have weaknesses or do not respond appropriately (Pfohl et al., 2024).”
Self-citationEvaluating gender bias in large language models in long-term care
Rickman, S. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03118-0 · Cited by 27 · see citing works
Golden, A., & Aboujaoude, E. N. (2024) · JMIR Formative Research · doi:10.2196/62963 · Cited by 21 · see citing works
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 34 · see citing works
Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models
Srinivasan, S., Ai, X., Zou, M., Zou, K., Kim, H., Lo, T. W. S., Pushpanathan, K., Yang, G. D., Goh, J. H. L., Kong, Y., et al. (2025) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2025.2413 · Cited by 21 · see citing works
Best Practices for the Safe Use of Large Language Models and Other Generative AI in Radiology
Yi, P. H., Haver, H. L., Jeudy, J. J., Kim, W., Kitamura, F., Oluyemi, E. T., Smith, A. D., Moy, L. P., & Parekh, V. S. (2025) · Radiology · doi:10.1148/radiol.241516 · Cited by 20 · see citing works
Bejan, C. A., Wang, M., Venkateswaran, S., Bergmann, E. A., Hiles, L., Xu, Y., Chandler, G. S., Brondfield, S., Silverstein, J., Wright, F., et al. (2026) · eBioMedicine · doi:10.1016/j.ebiom.2026.106227 · Cited by 16 · see citing works
The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs
Gourabathina, A., Gerych, W., Pan, E., & Ghassemi, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732121 · Cited by 14 · see citing works
Cao, H., Hao, C., Zhang, T., Zheng, X., Gao, Z., Wu, J., Gan, L., Liu, Y., Zeng, X., & Wang, W. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1605908 · Cited by 13 · see citing works
Self-citationBhimani, M., Miller, A., Agnew, J. D., Ausin, M. S., Raglow-Defranco, M., Mangat, H. S., Voisard, M., Taylor, M., Bierman-Lytle, S., Parikh, V., et al. (2025) · medRxiv · doi:10.1101/2025.03.17.25324157 · Cited by 13 · see citing works
Yang, Y., Jin, Q., Zhu, Q., Wang, Z., Álvarez, F. E., Wan, N., Hou, B., & Lu, Z. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094851 · Cited by 15 · see citing works
Safety principles for medical summarization using generative AI
Obika, B. D., Kelly, C. J., Ding, N., Farrance, C., Krause, J., Mittal, P., Cheung, D., Cole-Lewis, H. J., Elish, M. C., Karthikesalingam, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03313-y · Cited by 10 · see citing works
Self-citationLi, Y., Li, Z., Li, J., Liu, L., Liu, Y., Zhu, B., Shi, K., Lu, Y., Li, Y., Zeng, X., et al. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105961 · Cited by 10 · see citing works
Self-citationKim, M.-G., Hwang, G., Chang, J., Chang, S., Roh, H. W., & Park, R. W. (2025) · Journal of Medical Internet Research · doi:10.2196/69857 · Cited by 9 · see citing works
Abbott, E. E., Rehman, T., Rosania, A. R., Lum, D. L., Taylor, T. B., Kirk, A., Taylor, R. A., Baker, E. F., Rabin, E., Padela, A. I., et al. (2025) · Journal of the American College of Emergency Physicians Open · doi:10.1016/j.acepjo.2025.100311 · Cited by 12 · see citing works
Methodological opportunities in genomic data analysis to advance health equity
Lehmann, B., Bräuninger, L., Cho, Y., Falck, F., Jayadeva, S., Katell, M. A., Nguyen, T., Perini, A., Tallman, S., Mackintosh, M. E., et al. (2025) · Nature Reviews Genetics · doi:10.1038/s41576-025-00839-w · Cited by 7 · see citing works
Democratizing large language model-based graph data augmentation via latent knowledge graphs
Feng, Y., Chan, T. H., Yin, G., & Yu, L. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107777 · Cited by 7 · see citing works
Identifying Bias at Scale in Clinical Notes Using Large Language Models
Apakama, D. U., Nguyen, K.-A.-N., Hyppolite, D., Soffer, S., Mudrik, A., Ling, E. J., Moses, A., Temnycky, I., Glasser, A., Anderson, R., et al. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100296 · Cited by 13 · see citing works
A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health
Zhou, J., Chen, A. Z., Shah, D., Schwab‐Reese, L. M., & Choudhury, M. D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757544 · Cited by 7 · see citing works
Generative Psychometrics—An Emerging Frontier in Mental Health Measurement
Galatzer‐Levy, I. R., Tomašev, N., Chung, S., & Williams, G. (2025) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2025.3258 · Cited by 7 · see citing works
Memon, S., Brand, A., Taylor, B., Michael, A., & Smithson, R. (2026) · BMC Health Services Research · doi:10.1186/s12913-025-13954-5 · Cited by 7 · see citing works
Socio-demographic gaps in pain management guided by large language models
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Glicksberg, B. S., Hurd, Y. L., Apakama, D. U., Charney, A. W., Reich, D. L., Nadkarni, G. N., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00017-6 · Cited by 7 · see citing works
A scalable framework for evaluating health language models
Mallinar, N., Heydari, A. A., Liu, X., Faranesh, A. Z., Winslow, B. D., Hammerquist, N., Graef, B., Speed, C., Malhotra, M., Patel, S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02492-x · Cited by 14 · see citing works
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 14 · see citing works
Self-citationGautam, S., Chandra, M., De, A., Chakravorti, T., Malik, G., & De Choudhury, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06849 · Cited by 7 · see citing works
Evaluating the Effectiveness of Large Language Models in Converting Clinical Data to FHIR Format
Delaunay, J., Girbes, D., & Cusidó, J. (2025) · Applied Sciences · doi:10.3390/app15063379 · Cited by 6 · see citing works
Clinical applications of large language models in knee osteoarthritis: a systematic review
Ma, Z., Liu, Y., Zhang, Z., Chen, R., Fan, H., Cao, X., & Ni, L. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1670824 · Cited by 8 · see citing works
Self-citationInnovating global regulatory frameworks for generative AI in medical devices is an urgent priority
Ong, J. C. L., Ning, Y., Liu, M., Ma, Y., Zhao, L., Singh, K., Chang, R. T., Vogel, S., Lim, J. C. W., Tan, I. S. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02552-2 · Cited by 7 · see citing works
Yang, E., Garcia, T., Williams, H., Kumar, B., Ramé, M., Rivera, E., Ma, Y., Amar, J., Catalani, C., & Jia, Y. (2025) · JMIR Formative Research · doi:10.2196/75421 · Cited by 5 · see citing works
Transforming the self: Individual-level changes arising from collaboration with generative AI
Nandagopal, S. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100232 · Cited by 9 · see citing works
Redefining Bias Audits for Generative AI in Health Care
Chen, I. Y., & Alsentzer, E. (2025) · NEJM AI · doi:10.1056/aip2500015 · Cited by 5 · see citing works
Naidu, S. U., Li, H., Moon, J. T., Kim, R., Patel, E., Bercu, Z. L., Newsome, J. M., Gichoya, J. W., & Trivedi, H. (2026) · Academic Radiology · doi:10.1016/j.acra.2025.12.008 · Cited by 3 · see citing works
Ruelle, T., Grinda, T., Mastro, L. D., Lacroix‐Triki, M., Pistilli, B., & Gessain, G. (2026) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2025.100662 · Cited by 5 · see citing works
A framework for clinical validation of generative artificial intelligence therapeutics
Galatzer‐Levy, I. R., Tomašev, N., Rodríguez, C. I., & Torous, J. B. (2026) · World Psychiatry · doi:10.1002/wps.70067 · Cited by 4 · see citing works
Cavalera, C., Frisone, F., Rossi, C., Oasi, O., Pagnini, F., Riva, G., & Antichi, L. (2026) · Current Psychiatry Reports · doi:10.1007/s11920-026-01690-4 · Cited by 5 · see citing works
Delaunay, J., & Cusidó, J. (2024) · Applied Sciences · doi:10.3390/app15010061 · Cited by 4 · see citing works
Bias Patterns in the Application of LLMs for Clinical Decision Support
Poulain, R., Adiba, F. I., Fayyaz, H., & Beheshti, R. (2026) · Delaware Journal of Public Health · doi:10.32481/djph.2026.03.10 · Cited by 32 · see citing works
DxDirector: an agentic large language model driving the full-process clinical diagnosis
Xu, S., Huang, X., Wei, Z., Pang, L., Shen, H., & Cheng, X. (2026) · Nature Communications · doi:10.1038/s41467-026-71928-5 · Cited by 4 · see citing works
The digital therapist? LLMs and the future of clinical dialogue
Shi, E. R., Pei, R., & Barceló-Coblijn, L. (2026) · Language and Health · doi:10.1016/j.laheal.2026.100086 · Cited by 3 · see citing works
Pan, J., Jian, B., Hager, P., Zhang, Y., Liu, C., Jungmann, F., Li, H., Canisius, J., You, C., Wu, J., et al. (2026) · Nature Health · doi:10.1038/s44360-026-00152-8 · Cited by 5 · see citing works
Self-citationRafalko, N., Gianfrancesco, M. A., & Goldstein, N. D. (2025) · Global Epidemiology · doi:10.1016/j.gloepi.2025.100204 · Cited by 3 · see citing works
Zhang, J., Ubuzima, P., Huang, G., Lee, E.-H., Wang, Y., Xu, H., Xia, L., & Wu, T. (2025) · BMC Oral Health · doi:10.1186/s12903-025-07540-8 · Cited by 3 · see citing works
Zhang, X., Wang, T., Yan, C., Najdawi, F., Zhou, K., Ma, Y., Cheung, Y., & Malin, B. (2024) · medRxiv · doi:10.1101/2024.12.27.24319715 · Cited by 3 · see citing works
Yang, E., Garcia, T., Williams, H., Kumar, B., Ramé, M., Rivera, E., Ma, Y., Amar, J., Catalani, C., & Jia, Y. (2025) · doi:10.2196/preprints.75421 · Cited by 3 · see citing works
Erdem, O., Canbak, T., Acar, A., Ceylan, E. M., Çakıt, H., & Başak, F. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106341 · Cited by 2 · see citing works
Wang, Z., Fang, Y., & Li, Q. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08211-0 · Cited by 2 · see citing works
Ethical Considerations in Personal Health Large Language Models
Liu, J., & Liu, S. (2026) · Journal of Medical Internet Research · doi:10.2196/92240 · Cited by 2 · see citing works
Zhao, Y., Guo, R., Miao, Y., Luo, Y., Wang, H., & Wu, Y. X. (2026) · Journal of Medical Internet Research · doi:10.2196/89190 · Cited by 2 · see citing works
Zhou, Z., Ji, X., Zhang, J., Zhao, Z., Hei, X., & Choo, K. R. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93354-7_15 · Cited by 2 · see citing works
LLM-Guided Pain Management: Examining Socio-Demographic Gaps in Cancer vs non-Cancer casesPreprint
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Glicksberg, B. S., Hurd, Y. L., Apakama, D. U., Charney, A. W., Reich, D. L., Nadkarni, G. N., et al. (2025) · medRxiv · doi:10.1101/2025.03.04.25323396 · Cited by 2 · see citing works
Sun, X., Hong, Q., Zhang, M., Li, Y., Chen, T., Huang, Z., Liang, G., Tang, W., Xu, S., Ni, X., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102547 · Cited by 1 · see citing works
Cultural Bias and Linguistic Representation in AI Curriculum Design
Usman, N., Kusumanegara, A., Husnaini, H., Said, Y. R., & Ismayanti, D. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1062-6.ch008 · Cited by 1 · see citing works
Öberg, J., Vicente, R. P., Lindström, M., Midlöv, P., & Merlo, J. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00444-0 · Cited by 1 · see citing works
Semi-Automatic Extraction and Analysis of Health Equity Covariates in Registered Research Projects
Nananukul, N., & Kejriwal, M. (2025) · Applied Sciences · doi:10.3390/app152211853 · Cited by 1 · see citing works
Uncovering bias and variability in how large language models attribute cardiovascular risk
Chan, J. T. N., & Kwek, R. K. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1710594 · Cited by 1 · see citing works
Che, S., Zhu, M., Zhang, S., Jung, H. S., Lee, H., Wang, Z., & Miller, L. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104567 · Cited by 1 · see citing works
Ahmed, H., & Sadka, A. H. (2026) · IEEE Access · doi:10.1109/access.2026.3656396 · Cited by 1 · see citing works
Evaluation of algorithmic bias in large language models for retinal clinical recommendations
Gandhi, S., Balas, M., Popovic, M. M., Kertes, P. J., Muni, R. H., & Kohly, R. P. (2026) · AJO International · doi:10.1016/j.ajoint.2026.100233 · Cited by 2 · see citing works
Gandhi, S., & Balas, M. (2026) · BMC Medical Ethics · doi:10.1186/s12910-026-01496-2 · Cited by 1 · see citing works
Udayagiri, J., & Reddy, S., V (2026) · International Conference on Knowledge Engineering and Communication Systems (ICKECS) · doi:10.1109/ickecs70176.2026.11528016 · Cited by 1 · see citing works
Docking, J. J., Li, L. X., Menz, B. D., Bacchi, S., Hopkins, A. M., & Sorich, M. J. (2026) · Journal of Medical Internet Research · doi:10.2196/82256 · Cited by 1 · see citing works
Zhang, X., Wang, T., Yan, C., Najdawi, F., Zhou, K., Ma, Y., Cheung, Y.-M., Yeung, M. C. F., & Malin, B. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01694-8 · Cited by 1 · see citing works
Xiao, Y., Chen, Z., & Zhang, J. M. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2025.0125 · Cited by 1 · see citing works
Fosset, M., Pensier, J., Jung, B., Siddiqi, A., Mamdani, M. M., & Celi, L. A. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001621 · Cited by 1 · see citing works
Self-citationMcCradden, M. D., Tng, S., Jeyaseelan, D., Leane, C., Campbell, M., Braund, T., Earle-Bandaralage, L., Ebrahimi, M., Sharma, A. K., & TANG, J. C. Y. (2025) · medRxiv · doi:10.1101/2025.09.30.25336928 · Cited by 1 · see citing works
Trustworthy Medical Question Answering: An Evaluation-Centric Survey
Wang, Y., Wang, B., Mercer, R. E., Rudzicz, F., Roy, S. S., Ren, P., Chen, Z., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2506.03659
Open AI in Transplantation: A Friend or a Foe?
Wong, G., & Li, J. S. Y. (2024) · Transplantation · doi:10.1097/tp.0000000000005275
Centering Equity in Health AI: LLM Design Strategies for Impactful Solutions (Preprint)Preprint
Dewitt, A., Parker, A. G., Harrington, C., & Cole-Lewis, H. J. (2025) · doi:10.2196/preprints.71727
Self-citationDocking, J. J., Li, L. X., Menz, B. D., Bacchi, S., Hopkins, A. M., & Sorich, M. J. (2025) · medRxiv · doi:10.1101/2025.08.05.25333007
Grach, S., Badawi, A., Ahmad, F., & Dolatabadi, E. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105077
Pilowsky, J. K., Choi, J.-W., Nguyen, N., Williams, L., & Jones, S. L. (2025) · Australian Critical Care · doi:10.1016/j.aucc.2025.101487
Liao, N., Li, C., & Balch, C. M. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00310
Muniz, R., Lima, V., Saldanha, P., Ribeiro, A., de Paula, R., Cadeiras, M., da Silva, C., Silva, H., Rocha, P., & Pinheiro, D. (2025) · doi:10.5753/eniac.2025.13855
Large Language Models Reveal Narrative Architectures of Veteran Transition Stress (Preprint)Preprint
Galatzer‐Levy, I. R., Pan, X., Hart, R., & Bonanno, G. A. (2025) · doi:10.2196/preprints.90155
Trager, B. M., Hatch, O. J., Armstrong, K., Boyle, S. C., & LaBrie, J. W. (2026) · Journal of Psychology and AI · doi:10.1080/29974100.2026.2640258
Natural language processing for social good: Where we are, what is missing, and where we should go
Ranasinghe, T. (2026) · Natural language processing. · doi:10.1017/nlp.2026.10027
Asymmetric sociodemographic disparity in evidence-grounded clinical AIPreprint
Jia, E., Omar, M., Barash, Y., Brook, O. R., Ahmed, M., Kruskal, J. B., Gorenshtein, A., & Klang, E. (2026) · medRxiv · doi:10.64898/2026.05.12.26353061
Yu, X., & Qianqian, Y. (2026) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s614952
Assessing and mitigating demographic bias in large language models for diagnostic radiology
Horiuchi, D., Tatekawa, H., Takita, H., Mitsuyama, Y., Walston, S. L., Saito, K., Shimono, T., Miki, Y., & Ueda, D. (2026) · Japanese Journal of Radiology · doi:10.1007/s11604-026-02021-6
McCradden, M. D., Tng, S., Jeyaseelan, D., Leane, C., Campbell, M., Braund, T., Earle-Bandaralage, L., Ebrahimi, M., Sharma, A., & Tang, J. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-111656
From agentic artificial intelligence to accountable military healthcare: A readiness imperative
Vijayasimha, M., Srikanth, M., & Jha, R. K. (2026) · Medical Journal Armed Forces India · doi:10.1016/j.mjafi.2026.05.005
Alkam, T., Tarshizi, E., & Benschoten, A. H. V. (2026) · Journal of dementia and Alzheimer's disease · doi:10.3390/jdad3040048
Platforming Digital Community Health
Chen, A. Z. (2026) · ACM Interactive Health Conference · doi:10.1145/3786579.3801799
Wang, Y., & Steven, X. (2026) · International Conference on Artificial Intelligence and Digital Ethics (ICAIDE) · doi:10.1109/icaide69370.2026.11604541
Style-Based Bias Backdoor Attacks on Medical LLMs Under In-Context Learning
Jiang, Y., Xiao, S., Lin, J., Hao, F., & Gu, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3426-4_7
Can AI assist in reducing diagnostic error? A narrative review
Scott, I. A. (2026) · Diagnosis · doi:10.1515/dx-2026-0021
Zeng, J., Zhu, X., Shen, S., Li, S., Liu, X., Zhang, Q., & Cao, S. (2026) · Journal of Clinical Nursing · doi:10.1111/jocn.70478
- Authors
- Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., McCoy, L. G., Celi, L. A., Liu, Y., Schaekermann, M., Walton, A., Parrish, A., Nagpal, C., Singh, P., Dewitt, A., Mansfield, P., Prakash, S., Heller, K., Karthikesalingam, A., Semturs, C., Barral, J., Corrado, G., Matias, Y., Smith-Loud, J., Horn, I., & Singhal, K.
- Affiliations
- Google (United States) (US)
- University of Alberta (CA)
- Beth Israel Deaconess Medical Center (US)
- Corresponding author
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- ORCID
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Sayres, Rory
- Asiedu, Mercy
- Dieng, Awa
- Rashid, Qazi Mamunur
- Azizi, Shekoofeh
- Rostamzadeh, Negar
- McCoy, Liam G.
- Celi, Leo Anthony
- Liu, Yun
- Schaekermann, Mike
- Parrish, Alicia
- Nagpal, Chirag
- Singh, Preeti
- Dewitt, Akeiylah
- Mansfield, Philip
- Prakash, Sushant
- Heller, Katherine
- Karthikesalingam, Alan
- Semturs, Christopher
- Barral, Joelle
- Corrado, Greg
- Matias, Yossi
- Smith-Loud, Jamila
- Horn, Ivor
- Singhal, Karan
- Venue
- Nature Medicine
- Published
- 2024-09-23
- Volume
- 30
- Issue
- 12
- Pages
- 3590-3600
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11645264
- OpenAlex
- W4402748869
- Cited by
- 113
- References
- 66
- Citations by year
- 2026: 57 · 2025: 49 · 2024: 7
- FWCI
- 46.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- This study was funded by Google LLC and/or subsidiary thereof (Google).
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, health equity, algorithmic bias, health disparities, bias detection, human evaluation of AI, adversarial queries, participatory evaluation, health care equity
- MeSH terms
- Health Equity, Artificial Intelligence, Humans, Language, Bias, Healthcare Disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
