The DOI matches its Crossref record; 12 of the 13 references it cites were found in a registry or an index, and 1 are not checked yet.
ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models
Open accessPublished version of a preprint
Abstract
Purpose: This study aimed to assess the performance of ChatGPT, specifically the GPT-3.5 and GPT-4 models, in understanding complex surgical clinical information and its potential implications for surgical education and training. Methods: The dataset comprised 280 questions from the Korean general surgery board exams conducted between 2020 and 2022. Both GPT-3.5 and GPT-4 models were evaluated, and their performances were compared using McNemar test. Results: GPT-3.5 achieved an overall accuracy of 46.8%, while GPT-4 demonstrated a significant improvement with an overall accuracy of 76.4%, indicating a notable difference in performance between the models (P < 0.001). GPT-4 also exhibited consistent performance across all subspecialties, with accuracy rates ranging from 63.6% to 83.3%. Conclusion: ChatGPT, particularly GPT-4, demonstrates a remarkable ability to understand complex surgical clinical information, achieving an accuracy rate of 76.4% on the Korean general surgery board exam. However, it is important to recognize the limitations of large language models and ensure that they are used in conjunction with human expertise and judgment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GAI demonstrates potential in multidisciplinary knowledge acquisition within medical education by providing high-quality knowledge across various medical subfields [87-94].”
From Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Only a handful of these studies delved into the potential implications of generative AI's performance on these tests [8,33]. Those that did posited that this technology could be useful for self-directed learning or exam preparation [8,11,34]. However, none of these studies provided an explicit exploration of this process.”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, GPT-4 consistently performs well across different subspecialties, suggesting its capability to comprehend complex clinical information and enhance surgical education and training.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [143]
The cited source says what the claim attributes to it.
In the citing paper
“The results indicate an overall accuracy of 46.8% for GPT-3.5 and 76.4% for GPT-4, demonstrating a significant performance difference between the two models.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [143]
The cited source says what the claim attributes to it.
In the citing paper
“Several studies focused on the role of generative AI models in tests of medical knowledge [8-11,13,26,27,31-39].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“The precision and trustworthiness of generative AI are fundamental concerns thoroughly elaborated in many publications [8,11-15,20,22,24,32,35,41,42].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“A few studies outline potential scenarios in which educators or learners might interact with AI systems, such as in self-directed learning, simulation environments, and writing assistance [8,11,12,14,19,22,23,25,28,34,35,38,40,64].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [11]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Moreover, concerns have arisen that generative AI may produce biased content or lack representation of all populations [8,11-15,19,20,22,24,32,35,41,42].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [11]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“4% accuracy rate on the Korean surgical board exam, it is important to remember that LLMs are generative models, often referred to as “stochastic parrots” [11].”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
Medical Education Must Move From the Information Age to the Age of Artificial Intelligence
Wartman, S. A., & Combs, C. D. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002044 · Cited by 585
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 565
Han, E.-R., Yeo, S., Kim, M.-J., Lee, Y.-H., Park, K.-H., & Roh, H. (2019) · BMC Medical Education · doi:10.1186/s12909-019-1891-5 · Cited by 393
American Surgical Association Blue Ribbon Committee Report on Surgical Education: 2004
Debas, H. T., Bass, B. L., Brennan, M. F., Flynn, T. C., Folse, J. R., Freischlag, J. A., Friedmann, P., Greenfield, L. J., Jones, R. S., Lewis, F. R., et al. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000150066.83563.52 · Cited by 319
ChatGPT passing USMLE shines a spotlight on the flaws of medical education
Mbakwe, A. B., Lourentzou, I., Celi, L. A., Mechanic, O. J., & Dagan, A. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000205 · Cited by 273
Martelli, G., Boracchi, P., Palo, M. D., Pilotti, S., Oriana, S., Zucali, R., Daidone, M. G., & Palo, G. D. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000167759.15670.14 · Cited by 199
GPT Takes the Bar ExamPreprint
Bommarito, M., & Katz, D. (2022) · Social Science Research Network · doi:10.48550/arxiv.2212.14402 · Cited by 195
Bommarito, M. J., & Katz, D. M. (2022) · doi:10.2139/ssrn.4314839 · Cited by 181
Being Affable, Available, and Able Is Not Enough
Kapadia, M. R., & Kieran, K. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.5884 · Cited by 26
Research works printed but not linked: 1
- Touvron, 2023, arXiv [Preprint]
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 679
Preiksaitis, C., & Rose, C. (2023) · JMIR Medical Education · doi:10.2196/48785 · Cited by 552
“The precision and trustworthiness of generative AI are fundamental concerns thoroughly elaborated in many publications [8,11-15, 20,22, 24, 32, The propensity of these systems to generate and propagate misinformation is a notable risk.”
BackgroundInfluential
Metadata
- Authors
- Oh, N., Choi, G.-S., & Lee, W. Y.
- Affiliations
- Samsung Medical Center (KR)
- Corresponding author
- Lee, Woo Yong
- Venue
- Annals of Surgical Treatment and Research
- Publisher
- XMLink
- Published
- 2023
- Volume
- 104
- Issue
- 5
- Pages
- 269
- ISSN
- 2288-6575 (ISSN-L)
- 2288-6796
- PMC
- PMC10172028
- OpenAlex
- W4372047097
- Cited by
- 201
- References
- 12
- Citations by year
- 2026: 16 · 2025: 62 · 2024: 89 · 2023: 34
- FWCI
- 8.07
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Annals of Surgical Treatment and Research (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- Samsung
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- ChatGPT, GPT-4, large language models, surgical training, model performance comparison, GPT-3.5, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
