Assessing the research landscape and clinical utility of large language models: a scoping review
Open accessPublished version of a preprint
The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Abstract
Importance: Large language models (LLMs) like OpenAI’s ChatGPT are powerful generative systems that rapidly synthesize natural language responses. Research on LLMs has revealed their potential and pitfalls, especially in clinical settings. However, the evolving landscape of LLM research in medicine has left several gaps regarding their evaluation, application, and evidence base. Objective: This scoping review aims to (1) summarize current research evidence on the accuracy and efficacy of LLMs in medical applications, (2) discuss the ethical, legal, logistical, and socioeconomic implications of LLM use in clinical settings, (3) explore barriers and facilitators to LLM implementation in healthcare, (4) propose a standardized evaluation framework for assessing LLMs’ clinical utility, and (5) identify evidence gaps and propose future research directions for LLMs in clinical applications. Evidence review: We screened 4,036 records from MEDLINE, EMBASE, CINAHL, medRxiv, bioRxiv, and arXiv from January 2023 (inception of the search) to June 26, 2023 for English-language papers and analyzed findings from 55 worldwide studies. Quality of evidence was reported based on the Oxford Centre for Evidence-based Medicine recommendations. Findings: Our results demonstrate that LLMs show promise in compiling patient notes, assisting patients in navigating the healthcare system, and to some extent, supporting clinical decision-making when combined with human oversight. However, their utilization is limited by biases in training data that may harm patients, the generation of inaccurate but convincing information, and ethical, legal, socioeconomic, and privacy concerns. We also identified a lack of standardized methods for evaluating LLMs’ effectiveness and feasibility. Conclusions and relevance: This review thus highlights potential future directions and questions to address these limitations and to further explore LLMs’ potential in enhancing healthcare delivery.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, large language models (LLMs) can be integrated into this principle to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks [23, 48–52].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although the number of studies comparing different AI methods in clinical research is growing, comprehensive evaluations that employ a patient-centered approach remain limited and are needed to fully assess their relative utility.11,12”
From Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently (2025) · cites this paper as [12]
Experimental — accuracy not yet measuredIn the citing paper
“However, there are many pitfalls of using AI for such tasks, including the lack of sources for the model’s outputs, “hallucinations” (i.e. producing factually incorrect statements that appear plausible), and answers based on outdated information.112”
From Use of artificial intelligence for liver diseases: A survey from the EASL congress 2024 (2024) · cites this paper as [112]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565 · cited 1×
“This scoping review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Scoping Review (PRISMA-ScR), and using the Arksey and O’Malley framework [10].”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 1×
“However, we foresee a few challenges of locally-run LLMs, such as potentially their suboptimal performance or the requirement for specialized hardware, as seen with GPT-3 Furthermore, LLMs were found to generate inaccurate or “hallucinated” information/citations [73].”
BackgroundLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 2×
“Outside of the immediate clinical setting, LLM interfaces such as ChatGPT can enable patients to ask health-related questions, potentially optimizing health resource utilization [6].”
BackgroundAyers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 4×
“Other LLMs besides Chat-GPT (i.e. BioBERT, GatorTron, Foresight, BART) were also discussed in this category [4, 20, 29– 63 Despite these benefits, issues with incorporating LLMs like ChatGPT in clinical decision-making processes that were uniquely discussed by extracted studies in this category…”
BackgroundMethodInfluentialBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 2×
“Additionally, ChatGPT may “hallucinate” false references and can be easily influenced by the rhetoric and assumptions of its users [15, 17].”
BackgroundCascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · cited 1×
“Overall, recent guidelines made to address the logistical and ethical concerns of medical LLMs include the draft guidance issued by the FDA on predetermined change control plans for artificial intelligence/machine learning models [74].”
BackgroundA large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 662 · cited 2×
“…phrasings and vignette structures, and The inability of language models (LLMs) to provide tailored recommendations based on regional guidelines and specific cut-off values for management, as highlighted by Yeo et al. (2023), is an important limitation that warrants further investigation [24].”
BackgroundChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · cited 1×
“This is because subtle differences in wording can lead to significantly varied responses with LLMs, even when the underlying intent is the same [37, 4, 20, 29– 63 ].”
BackgroundUtility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412
Lambert, S. I., Madi, M., Sopka, S., Lenes, A., Stange, H., Buszello, C.-P., & Stephan, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00852-5 · Cited by 404 · cited 2×
“One barrier to having such discussions, as well as the successful implementation of LLMs into clinic spaces, was recently described by Lambert et al. [76].”
BackgroundUsing AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · arXiv · doi:10.48550/arxiv.2303.09038 · Cited by 393
Using ChatGPT to write patient clinic letters
Ali, S. R., Dobbs, T. D., Hutchings, H. A., & Whitaker, I. S. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00048-1 · Cited by 375 · cited 6×
“Further research is needed to explore integrating voice-to-text dictation software to ChatGPT documentation workflows, evaluating Chat-GPT’s documentation ability in non-English languages, and whether it can be reconfigured to increase accuracy and enhance privacy [12–17].”
BackgroundInfluentialLyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · Visual Computing for Industry, Biomedicine, and Art · doi:10.1186/s42492-023-00136-5 · Cited by 367
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358 · cited 1×
“Key stakeholders include clinical informaticians, developers, healthcare providers, policymakers, industry representatives, and civil society as a whole [75].”
BackgroundAI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346 · cited 7×
“Lastly, articles that described the ethical, logistical, and legal implications of LLMs in medicine were mainly in the form of commentaries and brief reports with more flexible discussion on these topics [7, 18–21].”
BackgroundMethodInfluentialHirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 343
Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Mireshghallah, F., Goyal, K., Uniyal, A., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.03929 · Cited by 256 · cited 1×
“Ensuring that LLMs comply with data protection regulations and maintaining patient confidentiality is essential to further investigate and reinforce [65, 66].”
BackgroundGravel, J., D’Amours-Gravel, M., & Osmanlliu, E. (2023) · doi:10.1101/2023.03.16.23286914 · Cited by 220
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199
Evaluation and adaptation of clinical practice guidelines
Graham, I. D., & Harrison, M. B. (2005) · Evidence-Based Nursing · doi:10.1136/ebn.8.3.68 · Cited by 177 · cited 1×
“One challenge in this realm, however, is that certain medical conditions currently lack standardized guidelines [80, 81].”
BackgroundAI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176
AI chatbots not yet ready for clinical use
Yeung, J. A., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · doi:10.1101/2023.03.02.23286705 · Cited by 176
Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Balston, A., Ross, J., Idowu, E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00025-6 · Cited by 169
What Does it Mean for a Language Model to Preserve Privacy?
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534642 · Cited by 165 · cited 1×
“Ensuring that LLMs comply with data protection regulations and maintaining patient confidentiality is essential to further investigate and reinforce [65, 66].”
BackgroundRau, A., Rau, S., Zöller, D., Fink, A., Tran, H., Wilpert, C., Nattenmüller, J., Neubauer, J., Bamberg, F., Reisert, M., et al. (2023) · Radiology · doi:10.1148/radiol.230970 · Cited by 150
Zhu, L., Mou, W., & Chen, R. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04123-5 · Cited by 143 · cited 1×
“…is likely (i) more feasible to create safety measures for, and (ii) more realistically able to be implemented as chatbot features for patients who wish to gain a tailored, baseline knowledge of any questions they may have regarding their health before meeting with their care team [23, 25].”
BackgroundAssessing the utility of ChatGPT throughout the entire clinical workflow
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K. J., & Succi, M. D. (2023) · doi:10.1101/2023.02.21.23285886 · Cited by 134 · cited 1×
“…medical questions (e.g. one study showed ChatGPT outright refused to provide a diagnosis in such situations despite established standards of care [33], racial and cultural biases inherent in training datasets that can cause patient harm, LLMs’ inabilities to account for the complexities of body…”
BackgroundSezgin, E., Sirrianni, J., & Linwood, S. L. (2022) · JMIR Medical Informatics · doi:10.2196/32875 · Cited by 127
Wagner, M. W., & Ertl-Wagner, B. B. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231171125 · Cited by 125
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106
The opportunities and pitfalls of ChatGPT in clinical and translational medicine
Baumgartner, C. (2023) · Clinical and Translational Medicine · doi:10.1002/ctm2.1206 · Cited by 103
Haemmerli, J., Sveikata, L., Nouri, A., May, A., Egervari, K., Freyschlag, C., Lobrinus, J. A., Migliorini, D., Momjian, S., Sanda, N., et al. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100775 · Cited by 100
ChatGPT in Clinical Toxicology
Sabry Abdel-Messih, M., & Kamel Boulos, M. N. (2023) · JMIR Medical Education · doi:10.2196/46876 · Cited by 95
Nastasi, A. J., Courtright, K. R., Halpern, S. D., & Weissman, G. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-45223-y · Cited by 93
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Mireshghallah, F., Goyal, K., Uniyal, A., Berg-Kirkpatrick, T., & Shokri, R. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.570 · Cited by 92
Harskamp, R. E., & De Clercq, L. (2023) · doi:10.1101/2023.03.25.23285475 · Cited by 92
Harskamp, R. E., & De Clercq, L. (2024) · Acta Cardiologica · doi:10.1080/00015385.2024.2303528 · Cited by 91
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90
Artificial Intelligence in Medicine & ChatGPT: De-Tether the Physician
DiGiorgio, A. M., & Ehrenfeld, J. M. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01926-3 · Cited by 82
Utilizing ChatGPT-4 for Providing Medical Information on Blepharoplasties to Patients
Cox, A., Seth, I., Xie, Y., Hunter-Smith, D. J., & Rozen, W. M. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad096 · Cited by 79
Bridging the literacy gap for surgical consents: an AI-human expert collaborative approach
Ali, R., Connolly, I. D., Tang, O. Y., Mirza, F. N., Johnston, B., Abdulrazeq, H. F., Lim, R. K., Galamaga, P. F., Libby, T. J., Sodha, N. R., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01039-2 · Cited by 78 · cited 2×
“Reported benefits of using LLMs like ChatGPT for communication with patients included providing quick, readable (i.e. approximately 8th-grade level, according to Ali et al. [26]), and often-times accurate instructions and analysis of test results to patients and their providers [23–28].”
BackgroundMethodUfuk, F. (2023) · Radiology · doi:10.1148/radiol.230276 · Cited by 76
Applications of natural language processing at emergency department triage: A narrative review
Stewart, J. E., Lü, J., Goudie, A., Arendts, G., Meka, S., Freeman, S., Walker, K., Sprivulis, P., Sanfilippo, F. M., Bennamoun, M., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0279953 · Cited by 70
ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases
Mehnen, L., Gruarin, S., Vasileva, M., & Knapp, B. (2023) · doi:10.1101/2023.04.20.23288859 · Cited by 60
Faithful AI in Medicine: A Systematic Review with Large Language Models and Beyond
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · doi:10.21203/rs.3.rs-3661764/v1 · Cited by 57
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C. B., Martin, C. E., Flores, M. G., Zhang, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.03540 · Cited by 53
Consulting ChatGPT: Ethical dilemmas in language model artificial intelligence
Beltrami, E. J., & Grant-Kels, J. M. (2024) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2023.02.052 · Cited by 51 · cited 6×
“Lastly, articles that described the ethical, logistical, and legal implications of LLMs in medicine were mainly in the form of commentaries and brief reports with more flexible discussion on these topics [7, 18–21].”
BackgroundMethodInfluentialRevolutionary Potential of ChatGPT in Constructing Intelligent Clinical Decision Support Systems
Liao, Z., Wang, J., Shi, Z., Lu, L., & Tabata, H. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03288-w · Cited by 42
Putting ChatGPT’s Medical Advice to the (Turing) Test
Nov, O., Singh, N., & Mann, D. M. (2023) · medRxiv · doi:10.1101/2023.01.23.23284735 · Cited by 39
Artificial intelligence: Augmenting telehealth with large language models
Snoswell, C. L., Snoswell, A. J., Kelly, J. T., Caffery, L. J., & Smith, A. C. (2025) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x231169055 · Cited by 37
Chari, S., Acharya, P., Gruen, D. M., Zhang, O., Eyigöz, E., Ghalwash, M. F., Seneviratne, O. W., Suárez, F., Meyer, P., Chakraborty, P., et al. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102498 · Cited by 35
Knebel, D., Priglinger, S., Scherer, N., Klaas, J., Siedlecki, J., & Schworm, B. (2024) · Klinische Monatsblätter für Augenheilkunde · doi:10.1055/a-2149-0447 · Cited by 32
Knebel, D., Priglinger, S., Scherer, N., Siedlecki, J., & Schworm, B. (2023) · doi:10.1101/2023.04.16.23288645 · Cited by 32
Perlis, R. H. (2023) · medRxiv · doi:10.1101/2023.04.14.23288595 · Cited by 23 · cited 2×
“…to “hallucinating” responses that were not in line with its prompt, as well as engaging in “false mimicry” where ChatGPT would respond in a manner that fit a user’s implicit assumptions rather than seeking further clarification from the user or providing corrections/caveats to its responses [46].”
BackgroundRavipati, A., Pradeep, T., & Elman, S. A. (2023) · International Journal of Dermatology · doi:10.1111/ijd.16746 · Cited by 21
Danilov, G., Kotik, K., Shevchenko, E. V., Usachev, D., Shifrin, M. A., Strunina, Y., Tsukanova, T., Ishankulov, T., Lukshin, V., & Aa, P. (2022) · Studies in health technology and informatics · doi:10.3233/shti210882 · Cited by 17
How large language models can augment perioperative medicine: a daring discourse
Gabriel, R. A., Mariano, E. R., McAuley, J., & Wu, C. L. (2023) · Regional Anesthesia & Pain Medicine · doi:10.1136/rapm-2023-104637 · Cited by 16 · cited 1×
“Table 1) [15, 52].”
Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRsPreprint
Kraljević, Ž., Bean, D. M., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Baston, A., Ross, J., Idowu, E., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08072 · Cited by 16 · cited 1×
“…be warranted to proceed with application testing of LLMs, particularly in rigorous randomized clinical trials, before wider-scale implementation [68] address the concern of privacy, increased data transparency (particularly via research with open-sourced LLMs like Falcon 40B, where its data and…”
BackgroundPuthenpura, V., Nadkarni, S., DiLuna, M. L., Hieftje, K. D., & Marks, A. M. (2023) · Cureus · doi:10.7759/cureus.36408 · Cited by 15 · cited 1×
“Multiple of our extracted articles highlighted the importance of consulting various stakeholders for the successful implementation of LLMs in healthcare [16, 20].”
BackgroundNoaeen, M., Amini, S., Bhasker, S., Ghezelsefli, Z., Ahmed, A., Jafarinezhad, O., & Abad, Z. S. H. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10340232 · Cited by 14
Natural Language Processing for Covid-19 Consulting System
Tripathy, S., Singh, R., & Ray, M. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.112 · Cited by 13
Lantz, R. H. (2023) · Cureus · doi:10.7759/cureus.35742 · Cited by 11 · cited 5×
“Further research is needed to explore integrating voice-to-text dictation software to ChatGPT documentation workflows, evaluating Chat-GPT’s documentation ability in non-English languages, and whether it can be reconfigured to increase accuracy and enhance privacy [12–17].”
BackgroundInfluentialWilliams, M. C., & Shambrook, J. S. (2023) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2023.03.010 · Cited by 11
Applications of Natural Language Processing at Emergency Department Triage: A Systematic ReviewPreprint
Stewart, J. E., Lü, J., Goudie, A., Arendts, G., Meka, S. A., Freeman, S., Walker, K., Sprivulis, P., Sanfilippo, F. M., Bennamoun, M., et al. (2022) · medRxiv · doi:10.1101/2022.12.20.22283735 · Cited by 4
Research works printed but not linked: 7
- Suresh K, Rathi V, Nwosu O, Partain MP, Glicksman JT, Jowett N et al. Utility of GPT-4 as an informational patient resource in otolaryngology [Internet]. medRxiv. 2023. Available from: https://www.medrxiv.org/content/https://doi. org/10.1101/2023.05.14.23289944v1. · doi:10.1101/2023.05.14.23289944v1
- Kim JH. Search for medical information and treatment options for muscu- loskeletal disorders through an artificial intelligence chatbot: Focusing on shoulder impingement syndrome [Internet]. bioRxiv. 2022. Available from: https://www.medrxiv.org/content/https://doi.org/10.1101/2022.12.16.22283 512v2. · doi:10.1101/2022.12.16.22283512v2
- Comrie D. ChatGPT decision support system: Utility in creating public policy for concussion/repetitive brain trauma associated with neurodegenerative diseases [Internet]. medRxiv. 2023. Available from: https://www.medrxiv.org/ content/https://doi.org/10.1101/2023.04.17.23288681v1. · doi:10.1101/2023.04.17.23288681v1
- Ueda D, Walston SL, Matsumoto T, Deguchi R, Tatekawa H, Miki Y, Evaluat- ing. GPT-4-based ChatGPT's clinical potential on the NEJM quiz [Internet]. medRxiv. 2023. Available from: https://www.medrxiv.org/content/https://doi. org/10.1101/2023.05.04.23289493v1. · doi:10.1101/2023.05.04.23289493v1
- Guo E, Gupta M, Sinha S, Rössler K, Tatagiba M, Akagami R et al. NeuroGPT- X: Towards an accountable expert opinion tool for vestibular schwan- noma [Internet]. medRxiv. 2023. Available from: https://www.medrxiv.org/ content/https://doi.org/10.1101/2023.02.25.23286117v1. · doi:10.1101/2023.02.25.23286117v1
- Nastasi AJ, Courtright KR, Halpern SD, Weissman GE. Does ChatGPT provide appropriate and equitable medical advice? A vignette-based, clinical evalu- ation across care contexts [Internet]. bioRxiv. 2023. Available from: https:// www.medrxiv.org/content/https://doi.org/10.1101/2023.02.25.23286451v1. · doi:10.1101/2023.02.25.23286451v1
- Lau FF, Ronit A, Weis N, Winckelmann A. Reactive infectious mucosal erup- tions (RIME) secondary to Chlamydia pneumoniae infection. Rep Int Dev Res Cent Can. 2021;4(2):11.
Cited by
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
A systematic review of large language model (LLM) evaluations in clinical medicine
Shool, S., Adimi, S., Saboori Amleshi, R., Bitaraf, E., Golpira, R., & Tara, M. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02954-4 · Cited by 346
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 288
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E., Malin, B., & Yin, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/22769 · Cited by 165
“Several reviews have been published to discuss the appropriateness of a particular application of LLMs in a specific aspect [1,6-9] but none of them summarized the overall problems systematically [8].”
Wells, B. J., Nguyen, H. M., McWilliams, A., Pallini, M., Bovi, A., Kuzma, A., Kramer, J., Chou, S.-H., Hetherington, T., Corn, P., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01900-y · Cited by 90
“While the literature in this area is nascent, evaluation should still be performed and may require qualitative metrics such as user feedback and expert reviews, which can provide insights into performance, risks, and usefulness 19,20 .”
Metadata
- Authors
- Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C.
- Affiliations
- University of Toronto (CA)
- University of Calgary (CA)
- Corresponding author
- Park, Ye-Jean
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2024-03-12
- Volume
- 24
- Issue
- 1
- Pages
- 72-72
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC10936025
- OpenAlex
- W4392691776
- Cited by
- 187
- References
- 62
- Citations by year
- 2026: 66 · 2025: 83 · 2024: 38
- FWCI
- 17.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Libraries and Cultural Resources (University of Calgary) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, ChatGPT, clinical utility, clinical decision support, LLM evaluation framework, training data bias, patient privacy, legal and ethical issues, implementation barriers and facilitators, health system navigation, human oversight, evidence gaps
- MeSH terms
- Clinical Decision-Making, Health Facilities, Humans, Language, MEDLINE, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.