From text to treatment: the crucial role of validation for generative large language models in health care
Open accessAbstract
Generative large language models (LLMs) have made incredible progress and are speculated to become the next big revolution in health care. Researchers have described several compelling uses for LLMs in health care, including the automatic generation of clinical information letters1 and chatbots answering patient questions.2–4 Thorough validation of developed LLMs is of the utmost importance to their safe and effective application in health-care practice, because incomplete LLM outputs or unchecked LLM hallucinations can be harmful to patient care.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The science of ensuring that generative AI tools are fair, appropriate, valid, effective, and safe is still emerging.
“It is likely that some of the capabilities required to effectively validate predictive AI provide a foundation for monitoring generative AI, but the science of ensuring that generative AI tools are fair, appropriate, valid, effective, and safe is still emerging.23,24”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [24]
Some capabilities required to effectively validate predictive AI provide a foundation for monitoring generative AI.
“It is likely that some of the capabilities required to effectively validate predictive AI provide a foundation for monitoring generative AI, but the science of ensuring that generative AI tools are fair, appropriate, valid, effective, and safe is still emerging.23,24”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [24]
In the discussion of validation challenges (page unavailable), the paper contrasts standardized criteria for predictive AI with the more complex validation needs of generative AI. It calls for continuous validation of LLMs, but does not say that capabilities used to validate predictive AI provide a foundation for monitoring generative AI. I searched for that proposed transfer of capabilities and for monitoring. (LLM, conf=0.70)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
The effect of using a large language model to respond to patient messages
Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B. H., Chipidza, F. E., Leeman, J., Aerts, H. J. W. L., Miller, T., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00060-8 · Cited by 147 · see citing works
Reporting standards for the use of large language model-linked chatbots for health advice.
Huo, B., Cacciamani, G. E., Collins, G. S., McKechnie, T., Lee, Y., & Guyatt, G. (2023) · Nature Medicine · doi:10.1038/s41591-023-02656-2 · Cited by 53 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [1]1. Ali, SR ∙ Dobbs, TD ∙ Hutchings, HA ∙ et al. Using ChatGPT to write patient clinic letters Lancet Digit Health. 2023; 5:e179-e181 Full Text Full Text (PDF) Scopus (142) PubMed Google Scholar
- [8]8. Gehrmann, S ∙ Clark, E ∙ Sellam, T Repairing the cracked foundation: a survey of obstacles in evaluation practices for generated text J Artif Intell Res. 2023; 77:103-166 Crossref Scopus (33) Google Scholar
- [9]9. Fabbri, AR ∙ Kryściński, W ∙ McCann, B ∙ et al. Summeval: Re-evaluating summarization evaluation Trans Assoc Comput Linguist. 2021; 9:391-409 Crossref Scopus (308) Google Scholar
- [10]10. van der Lee, C ∙ Gatt, A ∙ van Miltenburg, E ∙ et al. Human evaluation of automatically generated text: current trends and best practice guidelines Comput Speech Lang. 2021; 67, 101151 Crossref Scopus (92) Google Scholar
Cited by
Sorted by citations
Benchmark evaluation of DeepSeek large language models in clinical decision-making
Sandmann, S., Hegselmann, S., Fujarski, M., Bickmann, L., Wild, B., Eils, R., & Varghese, J. (2025) · Nature Medicine · doi:10.1038/s41591-025-03727-2 · Cited by 233 · see citing works
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83 · see citing works
Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals
Everson, J., Nong, P., & Richwine, C. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.49463 · Cited by 33 · see citing works
Artificial intelligence language models for medical text analysis: A systematic review
Sorayaie Azar, A., Bagherzadeh Mohasefi, J., Wiil, U. K., Naemi, A., & Ebrahimi, A. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103441 · Cited by 2 · see citing works
- Authors
- de Hond, A., Leeuwenberg, T., Bartels, R., van Buchem, M., Kant, I., Moons, K. G., & van Smeden, M.
- Affiliations
- University Medical Center Utrecht (NL)
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2024-07
- Volume
- 6
- Issue
- 7
- Pages
- e441-e443
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W4399834334
- Cited by
- 53
- References
- 6
- Citations by year
- 2026: 25 · 2025: 25 · 2024: 3
- FWCI
- 5.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- The Lancet Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- UMC Utrecht
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- generative large language models, LLM validation, health care, patient safety, chatbots, clinical decision support
- MeSH terms
- Delivery of Health Care, Humans, Language, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
