Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Open accessAbstract
Importance: Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas. Objective: To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty. Data Sources: A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024. Study Selection: Studies evaluating 1 or more LLMs in health care. Data Extraction and Synthesis: Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty. Results: Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented. Conclusions and Relevance: Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous reviews of LLM use within patient care have often overlooked pharmacy-specific applications.
“However, previous reviews of LLM use within patient care have often overlooked pharmacyspecific applications.”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [14]
This finding aligns with a previous review of ChatGPT in pharmacy practice, which highlighted a lack of standardized evaluation methods, and with broader healthcare reviews calling for standardized LLM evaluation approaches.
“This finding aligns with a previous review of ChatGPT in pharmacy practice, which highlighted a lack of standardized evaluation methods, 20 and with broader healthcare reviews calling for standardized LLM evaluation approaches.14,15”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [14]
Enthusiasm for new technologies can lead to unfocused experimentation, which may impede the full realization of the technology's potential.
“However, enthusiasm for new technologies can lead to unfocused experimentation, 14 which may impede the full realization of the technology's potential.107,108”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [14]
Recent analyses have emphasized a persistent gap between strong benchmark results and the scarcity of rigorous clinical trials for medical AI, underscoring the risk of overreliance on offline evaluations when making deployment decisions.
“Recent analyses have emphasized the persistent gap between strong benchmark results and the scarcity of rigorous clinical trials for medical AI, underscoring the risk of overreliance on offline evaluations when making deployment decisions23.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [23]
Large language models have potential as a transformative approach for identifying adverse events in patient notes.
“Recent progress of generative artificial intelligence (AI) technologies, such as Generative Pre-trained Transformer (GPT),15, 16, 17 underscores the potential of large language models (LLMs) as a transformative approach for identifying adverse events in patient notes.18,19”
From irAE-GPT: leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets (2026) · cites this paper as [18]
Many LLM evaluations focus on accuracy without addressing real-world variability, model drift, or contextual reasoning.
“As Bedi et al. [31] noted, many LLM evaluations focus on accuracy without addressing real-world variability, model drift, or contextual reasoning”
From Integrating large language models in biostatistical workflows for clinical and translational research (2025) · cites this paper as [31]
Real-world variability, model drift, or contextual reasoning may influence reliability of LLMs over time.
“factors that may influence reliability over time.”
From Integrating large language models in biostatistical workflows for clinical and translational research (2025) · cites this paper as [31]
Medical licensing exams and diagnostic tasks are prioritized in 64% of LLM healthcare evaluations, while many administrative and operational tasks are often overlooked.
“While medical licensing exams and diagnostic tasks are prioritized in 64% of LLM healthcare evaluations8, many administrative and operational tasks that represent significant human bottlenecks are often overlooked.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [8]
Only 5% of LLM evaluations use real-world electronic health record data.
“Only 5% of LLM evaluations use real-world electronic health record (EHR) data8.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [8]
The taxonomy was based on tasks identified in a JAMA review.
“We based our taxonomy on tasks identified in a JAMA review8.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [8]
Most research focuses on support for clinical reasoning, targeting diagnostics or treatment planning on synthetic benchmarks.
“However, most research focuses on support for clinical reasoning, targeting diagnostics or treatment planning on synthetic benchmarks [8].”
From Implementation of large language models in electronic health records (2025) · cites this paper as [8]
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283 · see citing works
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
Self-citationHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
Artificial Intelligence in U.S. Health Care Delivery
Sahni, N. R., & Carrus, B. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2204673 · Cited by 175 · see citing works
Large language models show human-like content biases in transmission chain experiments
Acerbi, A., & Stubbersfield, J. M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2313790120 · Cited by 137 · see citing works
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128 · see citing works
DRG-LLaMA : tuning LLaMA model to predict diagnosis-related group for hospitalized patients
Wang, H., Gao, C., Dantona, C., Hull, B., & Sun, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00989-3 · Cited by 99 · see citing works
Yim, W.-W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., & Yetisgen, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02487-3 · Cited by 64 · see citing works
Galido, P. V., Butala, S., Chakerian, M., & Agustines, D. (2023) · Cureus · doi:10.7759/cureus.38166 · Cited by 29 · see citing works
Heuer, A. J. (2021) · International Journal of Health Policy and Management · doi:10.34172/ijhpm.2021.129 · Cited by 11 · see citing works
DocLens: Multi-aspect Fine-grained Medical Text Evaluation
Xie, Y., Zhang, S., Cheng, H., Liu, P., Gero, Z., Wong, C., Naumann, T., Poon, H., & Rose, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.39 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83 · see citing works
Berkhout, W. E. M., van Wijngaarden, J. J., Workum, J. D., van de Sande, D., Hilling, D. E., Jung, C., Meyfroidt, G., Gommers, D., Buijsman, S. N. R., & van Genderen, M. E. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.22866 · Cited by 75 · see citing works
Large language model integrations in cancer decision-making: a systematic review and meta-analysis
Hao, Y., Qiu, Z., Holmes, J., Löckenhoff, C. E., Liu, W., Ghassemi, M., & Kalantari, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01824-7 · Cited by 67 · see citing works
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · see citing works
Self-citationCroxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M., Wills, G., Chen, G., Dligach, D., et al. (2025) · doi:10.1101/2025.04.22.25326219 · Cited by 25 · see citing works
Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R., Hong, P., Bates, D. W., & Zhou, L. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106091 · Cited by 26 · see citing works
Bejan, C. A., Wang, M., Venkateswaran, S., Bergmann, E. A., Hiles, L., Xu, Y., Chandler, G. S., Brondfield, S., Silverstein, J., Wright, F., et al. (2026) · eBioMedicine · doi:10.1016/j.ebiom.2026.106227 · Cited by 16 · see citing works
“18,19 These advancements reflect a broader evolution in AI from traditional machine learning and earlier deep learning models to LLMs, which can learn rich language representations through large-scale pretraining and require minimal task-specific engineering or labelled data.”
Implementation of large language models in electronic health records
Griot, M., Vanderdonckt, J., & Yuksel, D. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001141 · Cited by 23 · see citing works
Hu, Y., Liu, J., & Jiang, W. (2025) · Renal Failure · doi:10.1080/0886022x.2025.2555686 · Cited by 14 · see citing works
Gallano, G., Giglio, A., & Ferre, A. (2025) · Journal of Voice · doi:10.1016/j.jvoice.2025.04.010 · Cited by 9 · see citing works
Grambow, S. C., Desai, M., Weinfurt, K. P., Lindsell, C. J., Pencina, M. J., Rende, L., & Pomann, G.-M. (2025) · Journal of Clinical and Translational Science · doi:10.1017/cts.2025.10064 · Cited by 7 · see citing works
Innovating global regulatory frameworks for generative AI in medical devices is an urgent priority
Ong, J. C. L., Ning, Y., Liu, M., Ma, Y., Zhao, L., Singh, K., Chang, R. T., Vogel, S., Lim, J. C. W., Tan, I. S. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02552-2 · Cited by 7 · see citing works
Self-citationZheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
- Authors
- Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A. R., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A., Pfeffer, M. A., & Shah, N. H.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Harvard University (US)
- UC San Diego Health System (US)
- United States Food and Drug Administration (US)
- Corresponding author
- Shah, Nigam H.
- ORCID
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 2025-01-28
- Volume
- 333
- Issue
- 4
- Pages
- 319
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- PMC
- PMC11480901
- OpenAlex
- W4403420208
- Semantic Scholar
- cdb44a55439a135b6a8fb30c011f6e6e06440f64
- Cited by
- 662
- References
- 14
- Citations by year
- 2026: 420 · 2025: 234 · 2024: 8
- FWCI
- 14.94
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- JAMA (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, health care applications, LLM evaluation, natural language processing, natural language understanding, question answering, medical licensing examinations, medical question answering, bias and fairness, medical specialties, evaluation metrics, clinical decision support, clinical summarization, clinical prescribing, diagnosis, clinical decision-making
- MeSH terms
- Large Language Models, Delivery of Health Care, Medicine, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
