Clinical and Surgical Applications of Large Language Models: A Systematic Review
Open accessAbstract
Background: Large language models (LLMs) represent a recent advancement in artificial intelligence with medical applications across various healthcare domains. The objective of this review is to highlight how LLMs can be utilized by clinicians and surgeons in their everyday practice. Methods: A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Six databases were searched to identify relevant articles. Eligibility criteria emphasized articles focused primarily on clinical and surgical applications of LLMs. Results: The literature search yielded 333 results, with 34 meeting eligibility criteria. All articles were from 2023. There were 14 original research articles, four letters, one interview, and 15 review articles. These articles covered a wide variety of medical specialties, including various surgical subspecialties. Conclusions: LLMs have the potential to enhance healthcare delivery. In clinical settings, LLMs can assist in diagnosis, treatment guidance, patient triage, physician knowledge augmentation, and administrative tasks. In surgical settings, LLMs can assist surgeons with documentation, surgical planning, and intraoperative guidance. However, addressing their limitations and concerns, particularly those related to accuracy and biases, is crucial. LLMs should be viewed as tools to complement, not replace, the expertise of healthcare professionals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pressman et al. note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.
“For example, Pressman et al9 note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.”
From Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts (2026) · cites this paper as [9]
Section 4.5 reports that ChatGPT gave futile treatment recommendations in a salivary-gland clinic study and underperformed at triaging dermatologic lesions for Mohs surgery. It also warns that patients may encounter inaccurate, potentially harmful information. It does not report an instance of an LLM failing to identify a medical emergency, or establish that the cited treatment recommendations were given directly to patients. Page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 11,883 · cited 1× · see citing works
“However, this qualitative systematic literature review followed the organizational framework provided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [21] (Figure 2).”
MethodRecurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 7× · see citing works
“These models may prove instrumental for researchers, assisting in literature searches and the formulation of innovative research queries [7].”
BackgroundInfluentialAyers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
The rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636 · see citing works
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · see citing works
The benefits and challenges of chatgpt: An overview
Deng, J., & Lin, Y. (2023) · Frontiers in Computing and Intelligent Systems · doi:10.54097/fcis.v2i2.4465 · Cited by 611 · see citing works
Large language model AI chatbots require approval as medical devices
Gilbert, S., Harvey, H., Melvin, T., Vollebregt, E., & Wicks, P. (2023) · Nature Medicine · doi:10.1038/s41591-023-02412-6 · Cited by 207 · see citing works
Flagged first, then by citations
Research works printed but not linked: 50
- [1]Hamet P. Tremblay J. Artificial intelligence in medicine Metabolism 2017 69 S36 S40 10.1016/j.metabol.2017.01.011 28126242 · DOI · Paper page
- [3]Muftić F. Kadunić M. Mušinbegović A. Abd Almisreb A. Exploring Medical Breakthroughs: A Systematic Review of ChatGPT Applications in Healthcare Southeast Eur. J. Soft Comput. 2023 12 13 41 10.21533/scjournal · DOI
- [4]Jin Z. Analysis of the Technical Principles of ChatGPT and Prospects for Pre-trained Large Models Proceedings of the 2023 IEEE 3rd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA) Chongqing, China 26–28 May 2023 1755 1758
- [6]Abi-Rafeh J. Xu H.H. Kazan R. Tevlin R. Furnas H. Large Language Models and Artificial Intelligence: A Primer for Plastic Surgeons on the Demonstrated & Potential Applications, Promises, and Limitations of ChatGPT Aesthet. Surg. J. 2023 44 329 343 10.1093/asj/sjad260 37562022 · DOI
- [8]Tustumi F. Andreollo N.A. de Aguilar-Nascimento J.E. Future of the Language Models in Healthcare: The Role of ChatGPT. Review ABCD-Arq. Bras. Cir. Dig.-Braz. Arch. Dig. Surg. 2023 36 e1727 10.1590/0102-672020230002e1727 PMC10168663 37162073 · DOI · Paper page
- [11]Kung T.H. Cheatham M. Medenilla A. Sillos C. De Leon L. Elepaño C. Madriaga M. Aggabao R. Diaz-Candido G. Maningo J. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models PLOS Digit. Health 2023 2 e0000198 10.1371/journal.pdig.0000198 36812645 PMC9931230 · DOI · Paper page
- [12]Zhang Y. Pei H. Zhen S. Li Q. Liang F. Chat Generative Pre-Trained Transformer (ChatGPT) usage in healthcare Gastroenterol. Endosc. 2023 1 139 143 10.1016/j.gande.2023.07.002 · DOI
- [14]Jiang F. Jiang Y. Zhi H. Dong Y. Li H. Ma S. Wang Y. Dong Q. Shen H. Wang Y. Artificial intelligence in healthcare: Past, present and future Stroke Vasc. Neurol. 2017 2 230 243 10.1136/svn-2017-000101 29507784 PMC5829945 · DOI · Paper page
- [15]Yin J. Ngiam K.Y. Teo H.H. Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review J. Med. Internet Res. 2021 23 e25759 10.2196/25759 33885365 PMC8103304 · DOI · Paper page
- [16]Yu K.-H. Beam A.L. Kohane I.S. Artificial intelligence in healthcare Nat. Biomed. Eng. 2018 2 719 731 10.1038/s41551-018-0305-z 31015651 · DOI · Paper page
- [17]Argentiero A. Muscogiuri G. Rabbat M.G. Martini C. Soldato N. Basile P. Baggiano A. Mushtaq S. Fusini L. Mancini M.E. The Applications of Artificial Intelligence in Cardiovascular Magnetic Resonance-A Comprehensive Review J. Clin. Med. 2022 11 2866 10.3390/jcm11102866 35628992 PMC9147423 · DOI
- [18]Tran B.X. Latkin C.A. Vu G.T. Nguyen H.L.T. Nghiem S. Tan M.-X. Lim Z.-K. Ho C.S. Ho R.C. The Current Research Landscape of the Application of Artificial Intelligence in Managing Cerebrovascular and Heart Diseases: A Bibliometric and Content Analysis Int. J. Environ. Res. Public Health 2019 16 2699 10.3390/ijerph16152699 31362340 PMC6696240 · DOI
- [19]Uzun Ozsahin D. Ikechukwu Emegano D. Uzun B. Ozsahin I. The systematic review of artificial intelligence applications in breast cancer diagnosis Diagnostics 2022 13 45 10.3390/diagnostics13010045 36611337 PMC9818874 · DOI
- [20]Zhang Y. Weng Y. Lund J. Applications of explainable artificial intelligence in diagnosis and surgery Diagnostics 2022 12 237 10.3390/diagnostics12020237 35204328 PMC8870992 · DOI · Paper page
- [23]Bugaj M. Kliestik T. Lăzăroiu G. Generative Artificial Intelligence-based Diagnostic Algorithms in Disease Risk Detection, in Personalized and Targeted Healthcare Procedures, and in Patient Care Safety and Quality Contemp. Read. Law Soc. Justice 2023 15 9 26 10.22381/CRLSJ15120231 · DOI
- [24]Cadamuro J. Cabitza F. Debeljak Z. De Bruyne S. Frans G. Perez S.M. Ozdemir H. Tolios A. Carobene A. Padoan A. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI) Clin. Chem. Lab. Med. 2023 61 1158 1166 10.1515/cclm-2023-0355 37083166 · DOI
- [25]Chen T.C. Kaminski E. Koduri L. Singer A. Singer J. Couldwell M. Delashaw J. Dumont A. Wang A. Chat GPT as a Neuro-score Calculator: Analysis of a large language model’s performance on various neurological exam grading scales World Neurosurg. 2023 179 e342 e347 10.1016/j.wneu.2023.08.088 37634667 · DOI · Paper page
- [26]Cheng K. Li Z. He Y. Guo Q. Lu Y. Gu S. Wu H. Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example Ann. Biomed. Eng. 2023 51 1130 1135 10.1007/s10439-023-03203-3 37074486 PMC10116900 · DOI
- [27]Chiesa-Estomba C.M. Lechien J.R. Vaira L.A. Brunet A. Cammaroto G. Mayo-Yanez M. Sanchez-Barrueco A. Saga-Gutierrez C. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support Eur. Arch. Otorhinolaryngol. 2023 281 2081 2086 10.1007/s00405-023-08104-8 37405455 · DOI
- [28]Daher M. Koa J. Boufadel P. Singh J. Fares M.Y. Abboud J. Breaking Barriers: Can ChatGPT Compete with a Shoulder and Elbow Specialist in Diagnosis and Management? JSES Int. 2023 7 2534 2541 10.1016/j.jseint.2023.07.018 37969495 PMC10638599 · DOI
- [29]Dave T. Athaluri S.A. Singh S. ChatGPT in medicine: An overview of its applications, advantages, limitations, future prospects, and ethical considerations Front. Artif. Intell. 2023 6 1169595 10.3389/frai.2023.1169595 37215063 PMC10192861 · DOI · Paper page
- [30]Duey A.H. Nietsch K.S. Zaidat B. Ren R. Ndjonko L.C.M. Shrestha N. Rajjoub R. Ahmed W. Hoang T. Saturno M.P. Thromboembolic prophylaxis in spine surgery: An analysis of ChatGPT recommendations Spine J. 2023 23 1684 1691 10.1016/j.spinee.2023.07.015 37499880 · DOI
- [31]Gala D. Makaryus A.N. The Utility of Language Models in Cardiology: A Narrative Review of the Benefits and Concerns of ChatGPT-4 Int. J. Environ. Res. Public Health 2023 20 6438 10.3390/ijerph20156438 37568980 PMC10419098 · DOI
- [32]Gebrael G. Sahu K.K. Chigarira B. Tripathi N. Mathew Thomas V. Sayegh N. Maughan B.L. Agarwal N. Swami U. Li H. Enhancing Triage Efficiency and Accuracy in Emergency Rooms for Patients with Metastatic Prostate Cancer: A Retrospective Analysis of Artificial Intelligence-Assisted Triage Using ChatGPT 4.0 Cancers 2023 15 3717 10.3390/cancers15143717 37509379 PMC10378202 · DOI · Paper page
- [33]Grupac M. Zauskova A. Nica E. Generative Artificial Intelligence-based Treatment Planning in Clinical Decision-Making, in Precision Medicine, and in Personalized Healthcare Contemp. Read. Law Soc. Justice 2023 15 46 62 10.22381/CRLSJ15120233 · DOI
- [34]Haemmerli J. Sveikata L. Nouri A. May A. Egervari K. Freyschlag C. Lobrinus J.A. Migliorini D. Momjian S. Sanda N. ChatGPT in glioma adjuvant therapy decision making: Ready to assume the role of a doctor in the tumour board? BMJ Health Care Inform. 2023 30 e100775 10.1136/bmjhci-2023-100775 PMC10314415 37399360 · DOI
- [35]Javaid M. Haleem A. Singh R.P. ChatGPT for healthcare services: An emerging stage for an innovative perspective BenchCouncil Trans. Benchmarks Stand. Eval. 2023 3 100105 10.1016/j.tbench.2023.100105 · DOI · Paper page
- [36]Kottlors J. Bratke G. Rauen P. Kabbasch C. Persigehl T. Schlamann M. Lennartz S. Feasibility of differential diagnosis based on imaging patterns using a large language model Radiology 2023 308 e231167 10.1148/radiol.231167 37404149 · DOI
- [37]O’Hern K. Yang E. Vidal N.Y. ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms JAAD Int. 2023 12 168 170 10.1016/j.jdin.2023.06.002 37404248 PMC10316650 · DOI
- [38]Qu R.W. Qureshi U. Petersen G. Lee S.C. Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios OTO Open 2023 7 e67 10.1002/oto2.67 37614494 PMC10442607 · DOI
- [39]Rajjoub R. Arroyave J.S. Zaidat B. Ahmed W. Mejia M.R. Tang J. Kim J.S. Cho S.K. ChatGPT and its Role in the Decision-Making for the Diagnosis and Treatment of Lumbar Spinal Stenosis: A Comparative Analysis and Narrative Review Glob. Spine J. 2023 14 998 1017 10.1177/21925682231195783 PMC11192138 37560946 · DOI
- [40]Ravipati A. Pradeep T. Elman S.A. The role of artificial intelligence in dermatology: The promising but limited accuracy of ChatGPT in diagnosing clinical scenarios Int. J. Dermatol. 2023 62 e547 e548 10.1111/ijd.16746 37306147 · DOI
- [41]Rizwan A. Sadiq T. The Use of AI in Diagnosing Diseases and Providing Management Plans: A Consultation on Cardiovascular Disorders With ChatGPT Cureus 2023 15 e43106 10.7759/cureus.43106 37692649 PMC10483170 · DOI
- [42]Schukow C. Smith S.C. Landgrebe E. Parasuraman S. Folaranmi O.O. Paner G.P. Amin M.B. Application of ChatGPT in Routine Diagnostic Pathology: Promises, Pitfalls, and Potential Future Directions Adv. Anat. Pathol. 2023 31 15 21 10.1097/pap.0000000000000406 37501529 · DOI
- [43]Sharma S.C. Ramchandani J.P. Thakker A. Lahiri A. ChatGPT in Plastic and Reconstructive Surgery Indian. J. Plast. Surg. 2023 56 320 325 10.1055/s-0043-1771514 37705820 PMC10497341 · DOI
- [44]Sorin V. Klang E. Sklair-Levy M. Cohen I. Zippel D.B. Balint Lahat N. Konen E. Barash Y. Large language model (ChatGPT) as a support tool for breast tumor board NPJ Breast Cancer 2023 9 44 10.1038/s41523-023-00557-8 37253791 PMC10229606 · DOI
- [45]Srivastav S. Chandrakar R. Gupta S. Babhulkar V. Agrawal S. Jaiswal A. Prasad R. Wanjari M.B. ChatGPT in Radiology: The Advantages and Limitations of Artificial Intelligence for Medical Imaging Diagnosis Cureus 2023 15 e41435 10.7759/cureus.41435 37546142 PMC10404120 · DOI
- [46]Vaira L.A. Lechien J.R. Abbate V. Allevi F. Audino G. Beltramini G.A. Bergonzani M. Bolzoni A. Committeri U. Crimi S. Accuracy of ChatGPT-Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis Otolaryngol. Head Neck Surg. 2023 170 1492 1503 10.1002/ohn.489 37595113 · DOI
- [47]Xiao D. Meyers P. Upperman J.S. Robinson J.R. Revolutionizing Healthcare with ChatGPT: An Early Exploration of an AI Language Model’s Impact on Medicine at Large and its Role in Pediatric Surgery J. Pediatr. Surg. 2023 58 2410 2415 10.1016/j.jpedsurg.2023.07.008 37544801 · DOI
- [48]Xv Y. Peng C. Wei Z. Liao F. Xiao M. Can Chat-GPT a substitute for urological resident physician in diagnosing diseases?: A preliminary conclusion from an exploratory investigation World J. Urol. 2023 41 2569 2571 10.1007/s00345-023-04539-0 37505265 · DOI
- [49]Atkinson D. Generative Artificial Intelligence-based Treatment Planning in Patient Consultation and Support, in Digital Health Interventions, and in Medical Practice and Education Contemp. Read. Law Soc. Justice 2023 15 134 151 10.22381/CRLSJ15120238 · DOI
- [50]Cresswell A. Hart M. Suchanek O. Young T. Leaver L. Hibbs S. Mind the gap: Improving discharge communication between secondary and primary care BMJ Qual. Improv. Rep. 2015 4 u207936.w3197 10.1136/bmjquality.u207936.w3197 26734391 PMC4693041 · DOI
- [51]Ali M.J. ChatGPT and Lacrimal Drainage Disorders: Performance and Scope of Improvement Ophthalmic Plast. Reconstr. Surg. 2023 39 221 225 10.1097/IOP.0000000000002418 37166289 PMC10171282 · DOI
- [52]Mert S. Stoerzer P. Brauer J. Fuchs B. Haas-Lützenberger E.M. Demmer W. Giunta R.E. Nuernberger T. Diagnostic power of ChatGPT 4 in distal radius fracture detection through wrist radiographs Arch. Orthop. Trauma. Surg. 2024 144 2461 2467 10.1007/s00402-024-05298-2 38578309 PMC11093861 · DOI
- [53]Pressman S.M. Borna S. Gomez-Cabello C.A. Haider S.A. Forte A.J. AI in Hand Surgery: Assessing Large Language Models in the Classification and Management of Hand Injuries J. Clin. Med. 2024 13 2832 10.3390/jcm13102832 38792374 PMC11122623 · DOI
- [54]Gengatharan D. Saggi S.S. Bin Abd Razak H.R. Pre-operative Planning of High Tibial Osteotomy With ChatGPT: Are We There Yet? Cureus 2024 16 e54858 10.7759/cureus.54858 38533173 PMC10964394 · DOI
- [55]Ćirković A. Katz T. Exploring the Potential of ChatGPT-4 in Predicting Refractive Surgery Categorizations: Comparative Study JMIR Form. Res. 2023 7 e51798 10.2196/51798 38153777 PMC10784977 · DOI
- [56]Najafali D. Galbraith L.G. Camacho J.M. Arnold S.H. Alperovich M. King T.W. Cohen M.S. Morrison S.D. Dorafshar A.H. Addressing the Rhino in the Room: ChatGPT Creates "Novel" Patent Ideas for Rhinoplasty Eplasty 2024 24 e13 38685992 PMC11056627
- [59]Caruccio L. Cirillo S. Polese G. Solimando G. Sundaramurthy S. Tortora G. Can ChatGPT provide intelligent diagnoses? A comparative study between predictive models and ChatGPT to define a new medical diagnostic bot Expert. Syst. Appl. 2024 235 121186 10.1016/j.eswa.2023.121186 · DOI
- [60]Pressman S.M. Borna S. Gomez-Cabello C.A. Haider S.A. Haider C. Forte A.J. AI and Ethics: A Systematic Review of the Ethical Considerations of Large Language Model Use in Surgery Research Healthcare 2024 12 825 10.3390/healthcare12080825 38667587 PMC11050155 · DOI
Cited by
Sorted by citations
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83 · see citing works
McLaughlin, N. D., Srinivas, A. N., Lowe, Z. F., Botterbush, K. S., Patel, M. S., & Avila, M. J. (2026) · Neurosurgery Practice · doi:10.1227/neuprac.0000000000000244 · Cited by 1 · see citing works
“For example, Pressman et al 9 note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.”
- Authors
- Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J.
- Affiliations
- Mayo Clinic in Florida (US)
- Mayo Clinic (US)
- Corresponding author
- Forte, Antonio Jorge
- ORCID
- Venue
- Journal of Clinical Medicine
- Publisher
- MDPI AG
- Published
- 2024-05-22
- Volume
- 13
- Issue
- 11
- Pages
- 3041
- ISSN
- 2077-0383 (ISSN-L)
- PMC
- PMC11172607
- OpenAlex
- W4398223047
- Cited by
- 107
- References
- 8
- Citations by year
- 2026: 43 · 2025: 52 · 2024: 12
- FWCI
- 2.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Clinical Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Noaber Foundation (N/A)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- large language models, clinical application, surgical applications, diagnosis, treatment guidelines, triage, administrative functions, surgical documentation, surgical planning, intraoperative guidance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review