Cited by 107Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareArtificial Intelligence Applications

Clinical and Surgical Applications of Large Language Models: A Systematic Review

Open access
Pressman, Sophia M., Borna, Sahar, Gomez-Cabello, Cesar A., Haider, Syed Ali, Haider, Clifton R., Forte, Antonio Jorge·2024·Journal of Clinical Medicine

Abstract

Background: Large language models (LLMs) represent a recent advancement in artificial intelligence with medical applications across various healthcare domains. The objective of this review is to highlight how LLMs can be utilized by clinicians and surgeons in their everyday practice. Methods: A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Six databases were searched to identify relevant articles. Eligibility criteria emphasized articles focused primarily on clinical and surgical applications of LLMs. Results: The literature search yielded 333 results, with 34 meeting eligibility criteria. All articles were from 2023. There were 14 original research articles, four letters, one interview, and 15 review articles. These articles covered a wide variety of medical specialties, including various surgical subspecialties. Conclusions: LLMs have the potential to enhance healthcare delivery. In clinical settings, LLMs can assist in diagnosis, treatment guidance, patient triage, physician knowledge augmentation, and administrative tasks. In surgical settings, LLMs can assist surgeons with documentation, surgical planning, and intraoperative guidance. However, addressing their limitations and concerns, particularly those related to accuracy and biases, is crucial. LLMs should be viewed as tools to complement, not replace, the expertise of healthcare professionals.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/jcm13113041

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pressman et al. note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.

    “For example, Pressman et al9 note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.”

    From Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts (2026) · cites this paper as [9]

    Section 4.5 reports that ChatGPT gave futile treatment recommendations in a salivary-gland clinic study and underperformed at triaging dermatologic lesions for Mohs surgery. It also warns that patients may encounter inaccurate, potentially harmful information. It does not report an instance of an LLM failing to identify a medical emergency, or establish that the cited treatment recommendations were given directly to patients. Page numbers are unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. [1]Hamet P. Tremblay J. Artificial intelligence in medicine Metabolism 2017 69 S36 S40 10.1016/j.metabol.2017.01.011 28126242 · DOI · Paper page
  2. [3]Muftić F. Kadunić M. Mušinbegović A. Abd Almisreb A. Exploring Medical Breakthroughs: A Systematic Review of ChatGPT Applications in Healthcare Southeast Eur. J. Soft Comput. 2023 12 13 41 10.21533/scjournal · DOI
  3. [4]Jin Z. Analysis of the Technical Principles of ChatGPT and Prospects for Pre-trained Large Models Proceedings of the 2023 IEEE 3rd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA) Chongqing, China 26–28 May 2023 1755 1758
  4. [6]Abi-Rafeh J. Xu H.H. Kazan R. Tevlin R. Furnas H. Large Language Models and Artificial Intelligence: A Primer for Plastic Surgeons on the Demonstrated & Potential Applications, Promises, and Limitations of ChatGPT Aesthet. Surg. J. 2023 44 329 343 10.1093/asj/sjad260 37562022 · DOI
  5. [8]Tustumi F. Andreollo N.A. de Aguilar-Nascimento J.E. Future of the Language Models in Healthcare: The Role of ChatGPT. Review ABCD-Arq. Bras. Cir. Dig.-Braz. Arch. Dig. Surg. 2023 36 e1727 10.1590/0102-672020230002e1727 PMC10168663 37162073 · DOI · Paper page
  6. [11]Kung T.H. Cheatham M. Medenilla A. Sillos C. De Leon L. Elepaño C. Madriaga M. Aggabao R. Diaz-Candido G. Maningo J. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models PLOS Digit. Health 2023 2 e0000198 10.1371/journal.pdig.0000198 36812645 PMC9931230 · DOI · Paper page
  7. [12]Zhang Y. Pei H. Zhen S. Li Q. Liang F. Chat Generative Pre-Trained Transformer (ChatGPT) usage in healthcare Gastroenterol. Endosc. 2023 1 139 143 10.1016/j.gande.2023.07.002 · DOI
  8. [14]Jiang F. Jiang Y. Zhi H. Dong Y. Li H. Ma S. Wang Y. Dong Q. Shen H. Wang Y. Artificial intelligence in healthcare: Past, present and future Stroke Vasc. Neurol. 2017 2 230 243 10.1136/svn-2017-000101 29507784 PMC5829945 · DOI · Paper page
  9. [15]Yin J. Ngiam K.Y. Teo H.H. Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review J. Med. Internet Res. 2021 23 e25759 10.2196/25759 33885365 PMC8103304 · DOI · Paper page
  10. [16]Yu K.-H. Beam A.L. Kohane I.S. Artificial intelligence in healthcare Nat. Biomed. Eng. 2018 2 719 731 10.1038/s41551-018-0305-z 31015651 · DOI · Paper page
  11. [17]Argentiero A. Muscogiuri G. Rabbat M.G. Martini C. Soldato N. Basile P. Baggiano A. Mushtaq S. Fusini L. Mancini M.E. The Applications of Artificial Intelligence in Cardiovascular Magnetic Resonance-A Comprehensive Review J. Clin. Med. 2022 11 2866 10.3390/jcm11102866 35628992 PMC9147423 · DOI
  12. [18]Tran B.X. Latkin C.A. Vu G.T. Nguyen H.L.T. Nghiem S. Tan M.-X. Lim Z.-K. Ho C.S. Ho R.C. The Current Research Landscape of the Application of Artificial Intelligence in Managing Cerebrovascular and Heart Diseases: A Bibliometric and Content Analysis Int. J. Environ. Res. Public Health 2019 16 2699 10.3390/ijerph16152699 31362340 PMC6696240 · DOI
  13. [19]Uzun Ozsahin D. Ikechukwu Emegano D. Uzun B. Ozsahin I. The systematic review of artificial intelligence applications in breast cancer diagnosis Diagnostics 2022 13 45 10.3390/diagnostics13010045 36611337 PMC9818874 · DOI
  14. [20]Zhang Y. Weng Y. Lund J. Applications of explainable artificial intelligence in diagnosis and surgery Diagnostics 2022 12 237 10.3390/diagnostics12020237 35204328 PMC8870992 · DOI · Paper page
  15. [23]Bugaj M. Kliestik T. Lăzăroiu G. Generative Artificial Intelligence-based Diagnostic Algorithms in Disease Risk Detection, in Personalized and Targeted Healthcare Procedures, and in Patient Care Safety and Quality Contemp. Read. Law Soc. Justice 2023 15 9 26 10.22381/CRLSJ15120231 · DOI
  16. [24]Cadamuro J. Cabitza F. Debeljak Z. De Bruyne S. Frans G. Perez S.M. Ozdemir H. Tolios A. Carobene A. Padoan A. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI) Clin. Chem. Lab. Med. 2023 61 1158 1166 10.1515/cclm-2023-0355 37083166 · DOI
  17. [25]Chen T.C. Kaminski E. Koduri L. Singer A. Singer J. Couldwell M. Delashaw J. Dumont A. Wang A. Chat GPT as a Neuro-score Calculator: Analysis of a large language model’s performance on various neurological exam grading scales World Neurosurg. 2023 179 e342 e347 10.1016/j.wneu.2023.08.088 37634667 · DOI · Paper page
  18. [26]Cheng K. Li Z. He Y. Guo Q. Lu Y. Gu S. Wu H. Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example Ann. Biomed. Eng. 2023 51 1130 1135 10.1007/s10439-023-03203-3 37074486 PMC10116900 · DOI
  19. [27]Chiesa-Estomba C.M. Lechien J.R. Vaira L.A. Brunet A. Cammaroto G. Mayo-Yanez M. Sanchez-Barrueco A. Saga-Gutierrez C. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support Eur. Arch. Otorhinolaryngol. 2023 281 2081 2086 10.1007/s00405-023-08104-8 37405455 · DOI
  20. [28]Daher M. Koa J. Boufadel P. Singh J. Fares M.Y. Abboud J. Breaking Barriers: Can ChatGPT Compete with a Shoulder and Elbow Specialist in Diagnosis and Management? JSES Int. 2023 7 2534 2541 10.1016/j.jseint.2023.07.018 37969495 PMC10638599 · DOI
  21. [29]Dave T. Athaluri S.A. Singh S. ChatGPT in medicine: An overview of its applications, advantages, limitations, future prospects, and ethical considerations Front. Artif. Intell. 2023 6 1169595 10.3389/frai.2023.1169595 37215063 PMC10192861 · DOI · Paper page
  22. [30]Duey A.H. Nietsch K.S. Zaidat B. Ren R. Ndjonko L.C.M. Shrestha N. Rajjoub R. Ahmed W. Hoang T. Saturno M.P. Thromboembolic prophylaxis in spine surgery: An analysis of ChatGPT recommendations Spine J. 2023 23 1684 1691 10.1016/j.spinee.2023.07.015 37499880 · DOI
  23. [31]Gala D. Makaryus A.N. The Utility of Language Models in Cardiology: A Narrative Review of the Benefits and Concerns of ChatGPT-4 Int. J. Environ. Res. Public Health 2023 20 6438 10.3390/ijerph20156438 37568980 PMC10419098 · DOI
  24. [32]Gebrael G. Sahu K.K. Chigarira B. Tripathi N. Mathew Thomas V. Sayegh N. Maughan B.L. Agarwal N. Swami U. Li H. Enhancing Triage Efficiency and Accuracy in Emergency Rooms for Patients with Metastatic Prostate Cancer: A Retrospective Analysis of Artificial Intelligence-Assisted Triage Using ChatGPT 4.0 Cancers 2023 15 3717 10.3390/cancers15143717 37509379 PMC10378202 · DOI · Paper page
  25. [33]Grupac M. Zauskova A. Nica E. Generative Artificial Intelligence-based Treatment Planning in Clinical Decision-Making, in Precision Medicine, and in Personalized Healthcare Contemp. Read. Law Soc. Justice 2023 15 46 62 10.22381/CRLSJ15120233 · DOI
  26. [34]Haemmerli J. Sveikata L. Nouri A. May A. Egervari K. Freyschlag C. Lobrinus J.A. Migliorini D. Momjian S. Sanda N. ChatGPT in glioma adjuvant therapy decision making: Ready to assume the role of a doctor in the tumour board? BMJ Health Care Inform. 2023 30 e100775 10.1136/bmjhci-2023-100775 PMC10314415 37399360 · DOI
  27. [35]Javaid M. Haleem A. Singh R.P. ChatGPT for healthcare services: An emerging stage for an innovative perspective BenchCouncil Trans. Benchmarks Stand. Eval. 2023 3 100105 10.1016/j.tbench.2023.100105 · DOI · Paper page
  28. [36]Kottlors J. Bratke G. Rauen P. Kabbasch C. Persigehl T. Schlamann M. Lennartz S. Feasibility of differential diagnosis based on imaging patterns using a large language model Radiology 2023 308 e231167 10.1148/radiol.231167 37404149 · DOI
  29. [37]O’Hern K. Yang E. Vidal N.Y. ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms JAAD Int. 2023 12 168 170 10.1016/j.jdin.2023.06.002 37404248 PMC10316650 · DOI
  30. [38]Qu R.W. Qureshi U. Petersen G. Lee S.C. Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios OTO Open 2023 7 e67 10.1002/oto2.67 37614494 PMC10442607 · DOI
  31. [39]Rajjoub R. Arroyave J.S. Zaidat B. Ahmed W. Mejia M.R. Tang J. Kim J.S. Cho S.K. ChatGPT and its Role in the Decision-Making for the Diagnosis and Treatment of Lumbar Spinal Stenosis: A Comparative Analysis and Narrative Review Glob. Spine J. 2023 14 998 1017 10.1177/21925682231195783 PMC11192138 37560946 · DOI
  32. [40]Ravipati A. Pradeep T. Elman S.A. The role of artificial intelligence in dermatology: The promising but limited accuracy of ChatGPT in diagnosing clinical scenarios Int. J. Dermatol. 2023 62 e547 e548 10.1111/ijd.16746 37306147 · DOI
  33. [41]Rizwan A. Sadiq T. The Use of AI in Diagnosing Diseases and Providing Management Plans: A Consultation on Cardiovascular Disorders With ChatGPT Cureus 2023 15 e43106 10.7759/cureus.43106 37692649 PMC10483170 · DOI
  34. [42]Schukow C. Smith S.C. Landgrebe E. Parasuraman S. Folaranmi O.O. Paner G.P. Amin M.B. Application of ChatGPT in Routine Diagnostic Pathology: Promises, Pitfalls, and Potential Future Directions Adv. Anat. Pathol. 2023 31 15 21 10.1097/pap.0000000000000406 37501529 · DOI
  35. [43]Sharma S.C. Ramchandani J.P. Thakker A. Lahiri A. ChatGPT in Plastic and Reconstructive Surgery Indian. J. Plast. Surg. 2023 56 320 325 10.1055/s-0043-1771514 37705820 PMC10497341 · DOI
  36. [44]Sorin V. Klang E. Sklair-Levy M. Cohen I. Zippel D.B. Balint Lahat N. Konen E. Barash Y. Large language model (ChatGPT) as a support tool for breast tumor board NPJ Breast Cancer 2023 9 44 10.1038/s41523-023-00557-8 37253791 PMC10229606 · DOI
  37. [45]Srivastav S. Chandrakar R. Gupta S. Babhulkar V. Agrawal S. Jaiswal A. Prasad R. Wanjari M.B. ChatGPT in Radiology: The Advantages and Limitations of Artificial Intelligence for Medical Imaging Diagnosis Cureus 2023 15 e41435 10.7759/cureus.41435 37546142 PMC10404120 · DOI
  38. [46]Vaira L.A. Lechien J.R. Abbate V. Allevi F. Audino G. Beltramini G.A. Bergonzani M. Bolzoni A. Committeri U. Crimi S. Accuracy of ChatGPT-Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis Otolaryngol. Head Neck Surg. 2023 170 1492 1503 10.1002/ohn.489 37595113 · DOI
  39. [47]Xiao D. Meyers P. Upperman J.S. Robinson J.R. Revolutionizing Healthcare with ChatGPT: An Early Exploration of an AI Language Model’s Impact on Medicine at Large and its Role in Pediatric Surgery J. Pediatr. Surg. 2023 58 2410 2415 10.1016/j.jpedsurg.2023.07.008 37544801 · DOI
  40. [48]Xv Y. Peng C. Wei Z. Liao F. Xiao M. Can Chat-GPT a substitute for urological resident physician in diagnosing diseases?: A preliminary conclusion from an exploratory investigation World J. Urol. 2023 41 2569 2571 10.1007/s00345-023-04539-0 37505265 · DOI
  41. [49]Atkinson D. Generative Artificial Intelligence-based Treatment Planning in Patient Consultation and Support, in Digital Health Interventions, and in Medical Practice and Education Contemp. Read. Law Soc. Justice 2023 15 134 151 10.22381/CRLSJ15120238 · DOI
  42. [50]Cresswell A. Hart M. Suchanek O. Young T. Leaver L. Hibbs S. Mind the gap: Improving discharge communication between secondary and primary care BMJ Qual. Improv. Rep. 2015 4 u207936.w3197 10.1136/bmjquality.u207936.w3197 26734391 PMC4693041 · DOI
  43. [51]Ali M.J. ChatGPT and Lacrimal Drainage Disorders: Performance and Scope of Improvement Ophthalmic Plast. Reconstr. Surg. 2023 39 221 225 10.1097/IOP.0000000000002418 37166289 PMC10171282 · DOI
  44. [52]Mert S. Stoerzer P. Brauer J. Fuchs B. Haas-Lützenberger E.M. Demmer W. Giunta R.E. Nuernberger T. Diagnostic power of ChatGPT 4 in distal radius fracture detection through wrist radiographs Arch. Orthop. Trauma. Surg. 2024 144 2461 2467 10.1007/s00402-024-05298-2 38578309 PMC11093861 · DOI
  45. [53]Pressman S.M. Borna S. Gomez-Cabello C.A. Haider S.A. Forte A.J. AI in Hand Surgery: Assessing Large Language Models in the Classification and Management of Hand Injuries J. Clin. Med. 2024 13 2832 10.3390/jcm13102832 38792374 PMC11122623 · DOI
  46. [54]Gengatharan D. Saggi S.S. Bin Abd Razak H.R. Pre-operative Planning of High Tibial Osteotomy With ChatGPT: Are We There Yet? Cureus 2024 16 e54858 10.7759/cureus.54858 38533173 PMC10964394 · DOI
  47. [55]Ćirković A. Katz T. Exploring the Potential of ChatGPT-4 in Predicting Refractive Surgery Categorizations: Comparative Study JMIR Form. Res. 2023 7 e51798 10.2196/51798 38153777 PMC10784977 · DOI
  48. [56]Najafali D. Galbraith L.G. Camacho J.M. Arnold S.H. Alperovich M. King T.W. Cohen M.S. Morrison S.D. Dorafshar A.H. Addressing the Rhino in the Room: ChatGPT Creates "Novel" Patent Ideas for Rhinoplasty Eplasty 2024 24 e13 38685992 PMC11056627
  49. [59]Caruccio L. Cirillo S. Polese G. Solimando G. Sundaramurthy S. Tortora G. Can ChatGPT provide intelligent diagnoses? A comparative study between predictive models and ChatGPT to define a new medical diagnostic bot Expert. Syst. Appl. 2024 235 121186 10.1016/j.eswa.2023.121186 · DOI
  50. [60]Pressman S.M. Borna S. Gomez-Cabello C.A. Haider S.A. Haider C. Forte A.J. AI and Ethics: A Systematic Review of the Ethical Considerations of Large Language Model Use in Surgery Research Healthcare 2024 12 825 10.3390/healthcare12080825 38667587 PMC11050155 · DOI

Cited by

Sorted by citations

Authors
Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J.
Affiliations
  • Mayo Clinic in Florida (US)
  • Mayo Clinic (US)
Corresponding author
Forte, Antonio Jorge
Venue
Journal of Clinical Medicine
Publisher
MDPI AG
Published
2024-05-22
Volume
13
Issue
11
Pages
3041
ISSN
2077-0383 (ISSN-L)
OpenAlex
W4398223047
Cited by
107
References
8
Citations by year
2026: 43 · 2025: 52 · 2024: 12
FWCI
2.44
Type
journal article
Language
en
Open access
Locations
  • Journal of Clinical Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Noaber Foundation (N/A)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, clinical application, surgical applications, diagnosis, treatment guidelines, triage, administrative functions, surgical documentation, surgical planning, intraoperative guidance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review