CoChat
Sign Up
Cited by 52Artificial Intelligence in Healthcare and EducationTopic ModelingInformation Retrieval and Search Behavior

Analyzing evaluation methods for large language models in the medical field: a scoping review

Open access
Lee, Junbok, Park, Sungkyung, Shin, Jaeyong, Cho, Belong·2024·BMC Medical Informatics and Decision Making
DOI 10.1186/s12911-024-02709-7PMC PMC11606129

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Background: Owing to the rapid growth in the popularity of Large Language Models (LLMs), various performance evaluation studies have been conducted to confirm their applicability in the medical field. However, there is still no clear framework for evaluating LLMs. Objective: This study reviews studies on LLM evaluations in the medical field and analyzes the research methods used in these studies. It aims to provide a reference for future researchers designing LLM studies. Methods & materials: We conducted a scoping review of three databases (PubMed, Embase, and MEDLINE) to identify LLM-related articles published between January 1, 2023, and September 30, 2023. We analyzed the types of methods, number of questions (queries), evaluators, repeat measurements, additional analysis methods, use of prompt engineering, and metrics other than accuracy. Results: A total of 142 articles met the inclusion criteria. LLM evaluation was primarily categorized as either providing test examinations ( n = 53, 37.3%) or being evaluated by a medical professional ( n = 80, 56.3%), with some hybrid cases ( n = 5, 3.5%) or a combination of the two ( n = 4, 2.8%). Most studies had 100 or fewer questions ( n = 18, 29.0%), 15 (24.2%) performed repeated measurements, 18 (29.0%) performed additional analyses, and 8 (12.9%) used prompt engineering. For medical assessment, most studies used 50 or fewer queries ( n = 54, 64.3%), had two evaluators ( n = 43, 48.3%), and 14 (14.7%) used prompt engineering. Conclusions: More research is required regarding the application of LLMs in healthcare. Although previous studies have evaluated performance, future studies will likely focus on improving performance. A well-structured methodology is required for these studies to be conducted systematically.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12911-024-02709-7

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 144

  1. [1]Thirunavukarasu AJ, et al. Large language models in medicine. Nat Med. 2023;29:1930–40. 37460753
  2. [3]Abd-Alrazaq A, et al. Large Language models in Medical Education: opportunities, challenges, and future directions. JMIR Med Educ. 2023;9:e48291. 37261894 PMC10273039
  3. [4]Iannantuono GM, et al. Applications of large language models in cancer care: current evidence and future perspectives. Front Oncol. 2023;13:1268915. 37731643 PMC10507617
  4. [5]Qiu J et al. Large ai models in health informatics: Applications, challenges, and the future. IEEE Journal of Biomedical and Health Informatics (2023). (2023). 37738186
  5. [7]Wu T, et al. A brief overview of ChatGPT: the history, status quo and potential future development. IEEE/CAA J Automatica Sinica. 2023;10:1122–36.
  6. [10]Liu S, et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023;30:1237–45. 37087108 PMC10280357
  7. [11]Guo E, et al. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024;26:e48996. 38214966 PMC10818236
  8. [12]Subramanian CR, Yang DA, Khanna R. Enhancing health care communication with large language models—the role, challenges, and future directions. JAMA Netw Open. 2024;7:e240347–240347. 38466311
  9. [13]Karabacak M, Margetis K. Embracing large Language models for Medical Applications: opportunities and challenges. Cureus 15 (2023). PMC10292051 37378099
  10. [14]Choudhury A, Shamszare H. Investigating the impact of user trust on the adoption and use of ChatGPT: Survey Analysis. J Med Internet Res. 2023;25:e47184. 37314848 PMC10337387
  11. [15]Shahsavar Y, Choudhury A. User intentions to Use ChatGPT for self-diagnosis and health-related Purposes: cross-sectional survey study. JMIR Hum Factors. 2023;10:e47564. 37195756 PMC10233444
  12. [17]Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.
  13. [18]Colquhoun HL, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67:1291–4. 25034198
  14. [22]Ali R et al. Performance of ChatGPT, GPT-4, and Google bard on a neurosurgery oral boards preparation question bank. Neurosurgery (2022). 10.1227 (2022). 37306460
  15. [23]Ali R et al. Performance of ChatGPT and GPT-4 on neurosurgery written board examinations. medRxiv (2023). 2023.2003. 2025.23287743 (2023). 37581444
  16. [24]Antaki F et al. Evaluating the performance of chatgpt in ophthalmology: An analysis of its successes and shortcomings. Ophthalmology Science (2023). 100324 (2023). PMC10272508 37334036
  17. [25]Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology board-style examination: Insights into current strengths and limitations. Radiology (2023). 230582 (2023). 37191485
  18. [26]Cai LZ et al. Performance of Generative Large Language Models on Ophthalmology Board Style Questions. American Journal of Ophthalmology (2023). (2023). 37339728
  19. [27]Chen TC et al. Chat GPT as a Neuro-Score Calculator: Analysis of a Large Language Model’s Performance on Various Neurological Exam Grading Scales. World Neurosurgery (2023). (2023). 37634667
  20. [28]Cohen A et al. Performance of ChatGPT in Israeli Hebrew OBGYN national residency examinations. Archives of Gynecology and Obstetrics (2023). 1–6 (2023). 37668790
  21. [29]Cuthbert R, Simpson AI. Artificial intelligence in orthopaedics: can chat generative pre-trained transformer (ChatGPT) pass Sect. 1 of the Fellowship of the Royal College of Surgeons (Trauma & Orthopaedics) examination? Postgrad Med J. 2023;99:1110–4. 37410674
  22. [30]Deebel NA, Terlecki R. ChatGPT performance on the American Urological Association (AUA) Self-Assessment Study Program and the potential influence of artificial intelligence (AI) in urologic training. Urology (2023). (2023). 37209880
  23. [31]Friederichs H, Friederichs WJ, März M. ChatGPT in medical school: how successful is AI in progress testing? Med Educ Online. 2023;28:2220920. 37307503 PMC10262795
  24. [32]Gencer A, Aydin S. Can ChatGPT pass the thoracic surgery exam? Am J Med Sci. 2023;366:291–5. 37549788
  25. [33]Giannos P. Evaluating the limits of AI in medical specialisation: ChatGPT’s performance on the UK Neurology Specialty Certificate Examination. BMJ Neurol Open 5 (2023). PMC10277081 37337531
  26. [34]Gilson A, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312. 36753318 PMC9947764
  27. [35]Guerra GA et al. GPT-4 artificial intelligence model outperforms ChatGPT, medical students, and neurosurgery residents on neurosurgery written board-like questions. World Neurosurgery (2023). (2023). 37597659
  28. [36]Guigue PA et al. Performance of ChatGPT in French language Parcours d’Accès Spécifique Santé test and in OBGYN. International Journal of Gynecology & Obstetrics (2023). (2023). 37655838
  29. [37]Gupta R, et al. Performance of ChatGPT on the plastic surgery inservice training examination. Aesthetic Surg J. 2023;sjad128:2023. 37128784
  30. [38]Hoch CC et al. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. European Archives of Oto-Rhino-Laryngology (2023). 1–8 (2023). PMC10382366 37285018
  31. [39]Holmes J et al. Evaluating large language models on a highly-specialized topic, radiation oncology physics. arXiv preprint arXiv:2304.01938 (2023). (2023). PMC10388568 37529688 · arXiv
  32. [40]Hopkins BS, et al. ChatGPT versus the neurosurgical written boards: a comparative analysis of artificial intelligence/machine learning performance on neurosurgical board–style questions. J Neurosurg. 2023;139:904–11. 36964734
  33. [41]Huang RS, et al. Assessment of resident and AI chatbot performance on the University of Toronto family medicine residency progress test: comparative study. JMIR Med Educ. 2023;9:e50514. 37725411 PMC10548315
  34. [43]Hurley NC, Schroeder KM, Hess AS. Would doctors dream of electric blood bankers? Large language model-based artificial intelligence performs well in many aspects of transfusion medicine. Transfusion. 2023;63:1833–40. 37644845
  35. [44]Kaneda Y et al. Assessing the performance of GPT-3.5 and GPT-4 on the 2023 Japanese nursing examination. Cureus 15 (2023). PMC10475149 37667724
  36. [45]Kumah-Crystal Y, Mankowitz S, Embi P, Lehmann CU. ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification? Journal of the American Medical Informatics Association (2023). ocad104 (2023). PMC10436139 37335851
  37. [46]Kung JE, et al. Evaluating ChatGPT performance on the Orthopaedic In-Training examination. JBJS Open Access. 2023;8:e23. PMC10484364 37693092
  38. [47]Lewandowski M, Łukowicz P, Świetlik D, Barańska-Rybak W. An original study of ChatGPT-3.5 and ChatGPT-4 dermatological knowledge level based on the dermatology specialty certificate examinations. Clinical and Experimental Dermatology (2023). llad255 (2023). 37540015
  39. [48]Li Q, Min X. Unleashing the Power of Language Models in Clinical Settings: A Trailblazing Evaluation Unveiling Novel Test Design. medRxiv (2023). 2023.2007. 2011.23292512 (2023).
  40. [49]Long C et al. Evaluating ChatGPT-4 in Otolaryngology–Head and Neck Surgery Board Examination using the CVSA Model. medRxiv10.1101/2023.05.30.23290758 (2023). 2023.2005.2030.23290758 (2023).
  41. [50]Lum ZC. Can artificial intelligence pass the American Board of Orthopaedic Surgery examination? Orthopaedic residents versus ChatGPT. Clinical Orthopaedics and Related Research®. 2022:10.1097. PMC10344569 37220190
  42. [51]Madrid-García A, et al. Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish Access exam to Specialized Medical Training. medRxiv. 2023;20232007. 2021.23292821. (2023). PMC10719375 38092821
  43. [52]Massey PA, Montgomery C, Zhang AS. Comparison of ChatGPT–3.5, ChatGPT-4, and Orthopaedic Resident Performance on Orthopaedic Assessment Examinations. JAAOS-Journal of the American Academy of Orthopaedic Surgeons (2022). 10.5435 (2022). PMC10627532 37671415
  44. [54]Mihalache A, Popovic MM, Muni RH. Performance of an artificial intelligence chatbot in ophthalmic knowledge assessment. JAMA ophthalmology (2023). (2023). PMC10141269 37103928
  45. [55]Moshirfar M et al. Artificial Intelligence in Ophthalmology: a comparative analysis of GPT-3.5, GPT-4, and human expertise in answering StatPearls questions. Cureus 15 (2023). PMC10362981 37485215
  46. [56]Noda R et al. Performance of ChatGPT and Bard in Self-Assessment Questions for Nephrology Board Renewal. medRxiv (2023). 2023.2006. 2006.23291070 (2023). 38353783
  47. [58]Oztermeli AD, Oztermeli A. ChatGPT performance in the medical specialty exam: an observational study. Medicine. 2023;102:e34673. 37565917 PMC10419419
  48. [59]Passby L, Jenko N, Wernham A. Performance of ChatGPT on dermatology Specialty Certificate Examination multiple choice questions. Clin Exp Dermatol10.1093/ced/llad197 (2023). (2023). 37264670
  49. [60]Patil NS, Huang RS, van der Pol CB, Larocque N. Comparative performance of ChatGPT and bard in a text-based radiology knowledge assessment. Canadian Association of Radiologists Journal (2023). 08465371231193716 (2023). 37578849
  50. [62]Rosoł M et al. Evaluation of the performance of GPT-3.5 and GPT-4 on the Medical Final Examination. medRxiv (2023). 2023.2006. 2004.23290939 (2023). PMC10665355 37993519
  51. [63]Saad A, Iyengar KP, Kurisunkal V, Botchu R. Assessing ChatGPT’s ability to pass the FRCS orthopaedic part a exam: a critical analysis. Surgeon. 2023;21:263–6. 37517980
  52. [64]Schubert MC, Wick W, Venkataramani V. Evaluating the Performance of Large Language Models on a Neurology Board-Style Examination. medRxiv (2023). 2023.2007. 2013.23292598 (2023). PMC10704278 38060223
  53. [65]Shetty M, Ettlinger M, Lynch M. GPT-4, an artificial intelligence large language model, exhibits high levels of accuracy on dermatology specialty certificate exam questions. medRxiv (2023). 2023.2007. 2013.23292418 (2023).
  54. [66]Smith J, Choi PM, Buntine P. Will code one day run a code? Performance of language models on ACEM primary examinations and implications. Emergency Medicine Australasia (2023). (2023). 37414729
  55. [69]Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study. JMIR Med Educ. 2023;9:e48002. 37384388 PMC10365615
  56. [70]Tanaka Y et al. Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan. medRxiv (2023). 2023.2004. 2017.23288603 (2023). PMC10805303 38261580
  57. [71]Teebagy S, et al. Improved performance of chatgpt-4 on the OKAP examination: a comparative study with chatgpt-3.5. J Acad Ophthalmol. 2023;15:e184–7. PMC10495224 37701862
  58. [72]Thirunavukarasu AJ, et al. Trialling a large language model (ChatGPT) in general practice with the Applied Knowledge Test: observational study demonstrating opportunities and limitations in primary care. JMIR Med Educ. 2023;9:e46599. 37083633 PMC10163403
  59. [73]Valdez D et al. Performance of progressive generations of GPT on an exam designed for certifying physicians as Certified Clinical Densitometrists. medRxiv (2023). 2023.2007. 2025.23293171 (2023). 38401238
  60. [74]Wang H, et al. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Informatics. 2023;177:105173. 37549499
  61. [75]Abi-Rafeh J et al. Complications Following Facelift and Neck Lift: Implementation and Assessment of Large Language Model and Artificial Intelligence (ChatGPT) Performance Across 16 Simulated Patient Presentations. Aesthetic Plastic Surgery (2023). 1–8 (2023). 37589944
  62. [76]Ali MJ. ChatGPT and lacrimal drainage disorders: performance and scope of improvement. Ophthal Plast Reconstr Surg. 2023;39:221. 37166289 PMC10171282
  63. [77]Allahqoli L et al. The Diagnostic and Management Performance of the ChatGPT in Obstetrics and Gynecology. Gynecologic and Obstetric Investigation (2023). (2023). 37494894
  64. [80]Ayoub NF, Lee YJ, Grimm D, Divi V. Head-to‐Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition. Otolaryngology–Head and Neck Surgery (2023). (2023). 37529853
  65. [81]Barash Y, Klang E, Konen E, Sorin V. ChatGPT-4 assistance in optimizing emergency department radiology referrals and imaging selection. J Am Coll Radiol. 2023;20:998–1003. 37423350
  66. [82]Bellinger JR et al. BPPV Information on Google Versus AI (ChatGPT). Otolaryngology–Head and Neck Surgery (2023). (2023). 37622581
  67. [83]Benirschke RC et al. Assessment of a large language model’s utility in helping pathology professionals answer general knowledge pathology questions. American Journal of Clinical Pathology (2023). aqad106 (2023). 37658808
  68. [84]Bernstein IA, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. JAMA Netw Open. 2023;6:e2330320–2330320. 37606922 PMC10445188
  69. [85]Birkun AA, Gautam A. Large language model-based chatbot as a source of advice on first aid in heart attack. Current Problems in Cardiology (2023). 102048 (2023). 37640177
  70. [86]Biswas S et al. Assessing the utility of ChatGPT as an artificial intelligence-based large language model for information to answer questions on myopia. Ophthalmic and Physiological Optics (2023). (2023). 37476960
  71. [87]Cadamuro J, et al. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI). Clin Chem Lab Med (CCLM). 2023;61:1158–66. 37083166
  72. [88]Caglar U et al. Evaluating the performance of ChatGPT in answering questions related to pediatric urology. Journal of pediatric urology (2023). (2023). 37596194
  73. [89]Cakir H et al. Evaluating the performance of ChatGPT in answering questions related to urolithiasis. International Urology and Nephrology (2023). 1–5 (2023). 37658948
  74. [90]Chen S et al. The utility of ChatGPT for cancer treatment information. medRxiv (2023). 2023.2003. 2016.23287316 (2023).
  75. [91]Chiesa-Estomba CM et al. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. European Archives of Oto-Rhino-Laryngology (2023). 1–6 (2023). 37405455
  76. [92]Clough RA et al. Transforming healthcare documentation: Harnessing the potential of AI to generate discharge summaries. BJGP open (2023). (2023). PMC11169980 37699649
  77. [93]Cocci A et al. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate cancer and prostatic diseases (2023). 1–6 (2023). 37923807
  78. [94]Coskun B, Ocakoglu G, Yetemen M, Kaygisiz O. Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer? Urology 180, 35–58 (2023). 37406864
  79. [95]Coskun BN et al. Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatology International (2023). 1–7 (2023). 37747564
  80. [96]Davis R, et al. Evaluating the effectiveness of artificial intelligence–powered large language models application in disseminating appropriate and readable health information in urology. J Urol. 2023;210:688–94. 37428117
  81. [97]Delsoz M et al. Performance of chatgpt in diagnosis of corneal eye diseases. medRxiv (2023). (2023). 38391243
  82. [98]Delsoz M et al. The Use of ChatGPT to Assist in Diagnosing Glaucoma Based on Clinical Case Reports. Ophthalmology and Therapy (2023). 1–12 (2023). PMC10640454 37707707
  83. [99]Duey AH, et al. Thromboembolic prophylaxis in spine surgery: an analysis of ChatGPT recommendations. Spine J. 2023;23:1684–91. 37499880
  84. [100]Fink MA, et al. Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer. Radiology. 2023;308:e231362. 37724963
  85. [101]Gorelik Y, Ghersin I, Maza I, Klein A. Harnessing language models for streamlined postcolonoscopy patient management: a novel approach. Gastrointest Endosc. 2023;98:639–41. e634. 37385548
  86. [102]Haemmerli J et al. ChatGPT in glioma adjuvant therapy decision making: ready to assume the role of a doctor in the tumour board? BMJ Health Care Inf 30 (2023). PMC10314415 37399360
  87. [103]Henson JB et al. Evaluation of the Potential Utility of an Artificial Intelligence Chatbot in Gastroesophageal Reflux Disease Management. Official journal of the American College of Gastroenterology| ACG (2022). 10.14309 (2022). PMC10834834 37410934
  88. [104]Hirosawa T, et al. Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: a pilot study. Int J Environ Res Public Health. 2023;20:3378. 36834073 PMC9967747
  89. [105]Hirosawa T, Mizuta K, Harada Y, Shimizu T. Comparative Evaluation of Diagnostic Accuracy between Google Bard and Physicians. Am J Med. 2023;136:1119–23. e1118. 37643659
  90. [106]Hristidis V, et al. ChatGPT vs Google for queries related to dementia and other Cognitive decline: comparison of results. J Med Internet Res. 2023;25:e48966. 37490317 PMC10410383
  91. [107]Hu X et al. What can GPT-4 do for Diagnosing Rare Eye Diseases? A Pilot Study. Ophthalmology and Therapy (2023). 1–8 (2023). PMC10640532 37656399
  92. [108]Hung Y-C, et al. Comparison of Patient Education materials generated by Chat Generative Pre-trained Transformer Versus experts: an innovative way to increase readability of Patient Education materials. Ann Plast Surg. 2023;91:409–12. 37713147
  93. [109]Johnson D et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Research square (2023). (2023).
  94. [110]Kaarre J, et al. Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information. Knee Surg Sports Traumatol Arthrosc. 2023;31:5190–8. 37553552 PMC10598178
  95. [111]Kao H-J et al. Assessing ChatGPT’s capacity for clinical decision support in pediatrics: a comparative study with pediatricians using KIDMAP of Rasch analysis. Medicine 102 (2023). PMC10289633 37352054
  96. [112]King RC et al. A multidisciplinary assessment of ChatGPTs knowledge of amyloidosis. medRxiv (2023). 2023.2007. 2017.23292780 (2023).
  97. [113]King RC et al. Appropriateness of ChatGPT in answering heart failure related questions. medRxiv (2023). 2023.2007. 2007.23292385 (2023). 38821760
  98. [114]Kiyohara Y et al. Large language models to differentiate vasospastic angina using patient information. medRxiv (2023). 2023.2006. 2026.23291913 (2023).
  99. [115]Krusche M, Callhoff J, Knitza J, Ruffer N. Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatology International (2023). 1–4 (2023). PMC10796566 37742280
  100. [116]Kuckelman IJ et al. Assessing ai-powered patient education: a case study in radiology. Academic Radiology (2023). (2023). 37709612
  101. [117]Kumari A et al. Large language models in hematology case solving: a comparative study of ChatGPT-3.5, Google Bard, and Microsoft Bing. Cureus 15 (2023). PMC10511207 37736448
  102. [118]Kuroiwa T, et al. The potential of ChatGPT as a Self-Diagnostic Tool in Common Orthopedic diseases: exploratory study. J Med Internet Res. 2023;25:e47621. 37713254 PMC10541638
  103. [119]Kusunose K, Kashima S, Sata M. Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension guidelines. Circ J. 2023;87:1030–3. 37286486
  104. [120]Lahat A et al. Evaluating the Utility of a Large Language Model in Answering Common Patients’ Gastrointestinal Health-Related Questions: Are We There Yet? Diagnostics 13, 1950 (2023). PMC10252924 37296802
  105. [121]Lim ZW et al. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine 95 (2023). PMC10470220 37625267
  106. [122]Liu S et al. Assessing the value of ChatGPT for clinical decision support optimization. MedRxiv (2023). 2023.2002. 2021.23286254 (2023).
  107. [124]Lyons RJ et al. Artificial intelligence chatbot performance in triage of ophthalmic conditions. Canadian Journal of Ophthalmology (2023). (2023). 37572695
  108. [126]Mika AP et al. Assessing ChatGPT Responses to Common Patient Questions Regarding Total Hip Arthroplasty. JBJS 105, 1519–1526 (2023). 37459402
  109. [127]Mishra A et al. Exploring the intersection of artificial intelligence and neurosurgery: Let us be cautious with ChatGPT. Neurosurgery (2022). 10.1227 (2022). 37417886
  110. [128]Momenaei B et al. Appropriateness and Readability of ChatGPT-4 generated Responses for Surgical Treatment of Retinal Diseases. Ophthalmology Retina (2023). (2023). 37277096
  111. [129]Nakaura T et al. Preliminary assessment of automated radiology report generation with generative pre-trained transformers: comparing results to radiologist-generated reports. Japanese Journal of Radiology (2023). 1–11 (2023). PMC10811038 37713022
  112. [130]O’Hagan R et al. Trends in accuracy and appropriateness of alopecia areata information obtained from a popular online large language model, ChatGPT. Dermatology (Basel, Switzerland) (2023). (2023). 37722370
  113. [131]Qu RW, Qureshi U, Petersen G, Lee SC. Diagnostic and management applications of ChatGPT in structured otolaryngology clinical scenarios. OTO open. 2023;7:e67. 37614494 PMC10442607
  114. [132]Rahsepar AA, et al. How AI responds to common lung Cancer questions: ChatGPT vs Google Bard. Radiology. 2023;307:e230922. 37310252
  115. [133]Rao A et al. Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot. Journal of the American College of Radiology (2023). (2023). PMC10733745 37356806
  116. [134]Rao A, et al. Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study. J Med Internet Res. 2023;25:e48659. 37606976 PMC10481210
  117. [135]Rau A et al. A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines. medRxiv (2023). 2023.2004. 2010.23288354 (2023). 37489981
  118. [136]Reese JT et al. On the limitations of large language models in clinical diagnosis. medRxiv (2023). (2023).
  119. [137]Rogasch JM et al. ChatGPT: Can You Prepare My Patients for [18F] FDG PET/CT and Explain My Reports? Journal of Nuclear Medicine (2023). (2023). PMC10690125 37709536
  120. [138]Rojas-Carabali W et al. Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis. Ocular Immunology and Inflammation (2023). 1–6 (2023). 37722842
  121. [139]Russe MF, et al. Performance of ChatGPT, human radiologists, and context-aware ChatGPT in identifying AO codes from radiology reports. Sci Rep. 2023;13:14215. 37648742 PMC10468502
  122. [140]Salazar GZ et al. Efficacy of AI chats to determine an emergency: a comparison between OpenAI’s ChatGPT, Google Bard, and Microsoft Bing AI Chat. Cureus 15 (2023). PMC10506659 37727841
  123. [141]Samaan JS, et al. ChatGPT’s ability to comprehend and answer cirrhosis related questions in Arabic. Arab J Gastroenterol. 2023;24:145–8. 37673708
  124. [142]Samaan JS et al. Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obesity surgery (2023). 1–7 (2023). PMC10234918 37106269
  125. [143]Sarbay İ, Berikol GB, Özturan İU. Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): a preliminary, scenario-based cross-sectional study. Turkish J Emerg Med. 2023;23:156. PMC10389099 37529789
  126. [145]Stevenson E, Walsh C, Hibberd L. Can artificial intelligence replace biochemists? A study comparing interpretation of thyroid function test results by ChatGPT and Google Bard to practising biochemists. Annals of Clinical Biochemistry (2023). 00045632231203473 (2023). 37699796
  127. [146]Sütcüoğlu BM, Güler M. Appropriateness of premature ovarian insufficiency recommendations provided by ChatGPT. Menopause. 2023;30:1033–7. 37671567
  128. [147]Suthar PP et al. Artificial Intelligence (AI) in Radiology: A Deep Dive Into ChatGPT 4.0’s Accuracy with the American Journal of Neuroradiology’s (AJNR) Case of the Month. Cureus 15 (2023). PMC10516448 37746411
  129. [148]Ueda D, et al. Diagnostic performance of ChatGPT from Patient History and Imaging findings on the diagnosis please quizzes. Radiology. 2023;308:e231040. 37462501
  130. [149]Uz C, Umay E. Dr ChatGPT: Is it a reliable and useful source for common rheumatic diseases? International Journal of Rheumatic Diseases (2023). (2023). 37218530
  131. [150]Vaira LA et al. Accuracy of ChatGPT-Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis. Otolaryngology–Head and Neck Surgery (2023). (2023). 37595113
  132. [151]Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Canadian Association of Radiologists Journal (2023). 08465371231171125 (2023). 37078489
  133. [152]Wang C, Liu S, Li A, Liu J. Text dialogue analysis based ChatGPT for primary screening of mild cognitive impairment. medRxiv (2023). 2023.2006. 2027.23291884 (2023). PMC10787336 38157230
  134. [153]Whiles BB, et al. Caution! AI bot has entered the patient chat: ChatGPT has limitations in providing accurate urologic healthcare advice. Urology. 2023;180:278–84. 37467806
  135. [154]Yeo YH et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. medRxiv (2023). 2023.2002. 2006.23285449 (2023). PMC10366809 36946005
  136. [155]Angel M, Rinehart J, Canneson M, Baldi PF. Clinical Knowledge and Reasoning Abilities of AI Large Language Models in Anesthesiology: A Comparative Study on the ABA Exam. medRxiv (2023). 2023.2005. 2010.23289805 (2023). PMC11373264 38640076
  137. [157]Copeland-Halperin LR, O’Brien L, Copeland M. Evaluation of Artificial intelligence–generated responses to common plastic surgery questions. Plast Reconstr Surgery–Global Open. 2023;11:e5226. PMC10468106 37654681
  138. [159]Beaulieu-Jones BR et al. Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge Assessments. medRxiv (2023). (2023). PMC10947829 38246839
  139. [160]Fang C et al. How does ChatGPT4 preform on Non-English National Medical Licensing Examination? An Evaluation in Chinese Language. medRxiv (2023). 2023.2005. 2003.23289443 (2023). PMC10691691 38039286
  140. [161]Huynh LM et al. New Artificial Intelligence ChatGPT Performs Poorly on the 2022 Self-assessment Study Program for Urology. Urology Practice. 2023. 10.1097/UPJ.0000000000000406. 37276372 · DOI
  141. [162]Kung TH, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2:e0000198. 36812645 PMC9931230
  142. [163]Strong E et al. Performance of ChatGPT on free-response, clinical reasoning exams. medRxiv (2023). 2023.2003. 2024.23287731 (2023).
  143. [164]Athaluri SA et al. Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus 15 (2023). PMC10173677 37182055
  144. [165]Gilbert S et al. Large language model AI chatbots require approval as medical devices. Nature Medicine (2023). 1–3 (2023). 37391665

Cited by

Sorted by citations

Authors
Lee, J., Park, S., Shin, J., & Cho, B.
Affiliations
  • Seoul National University (KR)
  • Seoul National University of Science and Technology (KR)
  • Yonsei University (KR)
Corresponding author
  • Shin, Jaeyong
  • Cho, Belong
ORCID
Venue
BMC Medical Informatics and Decision Making
Publisher
Springer Science and Business Media LLC
Published
2024-11-29
Volume
24
Issue
1
Pages
366-366
ISSN
1472-6947 (ISSN-L)
DOI
10.1186/s12911-024-02709-7
PMC
PMC11606129
OpenAlex
W4404837143
Cited by
52
References
20
Citations by year
2026: 27 · 2025: 25
FWCI
1.19
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Ministry of Trade, Industry & Energy (20018246)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
large language models, LLM evaluation, medical applications of AI, evaluation methodology, repeated measures, prompt engineering, evaluation metrics, artificial intelligence in healthcare
MeSH terms
Evaluation Studies as Topic, Humans, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review