Can ChatGPT outperform a neurosurgical trainee? a prospective comparative study.
Open accessAbstract
PURPOSE: This study aimed to compare the performance of ChatGPT, a large language model (LLM), with human neurosurgical applicants in a neurosurgical national selection interview, to assess the potential of artificial intelligence (AI) and LLMs in healthcare and provide insights into their integration into the field. METHODS: In a prospective comparative study, a set of neurosurgical national selection-style interview questions were asked to eight human participants and ChatGPT in an online interview. All participants were doctors currently practicing in the UK who had applied for a neurosurgical National Training Number. Interviews were recorded, anonymised, and scored by three neurosurgical consultants with experience as interviewers for national selection. Answers provided by ChatGPT were used as a template for a virtual interview. Interview transcripts were subsequently scored by neurosurgical consultants using criteria utilised in real national selection interviews. Overall interview score and subdomain scores were compared between human participants and ChatGPT. RESULTS: For overall score, ChatGPT fell behind six human competitors and did not achieve a mean score higher than any individuals who achieved training positions. Several factors, including factual inaccuracies and deviations from expected structure and style may have contributed to ChatGPT's underperformance. CONCLUSIONS: LLMs such as ChatGPT have huge potential for integration in healthcare. However, this study emphasises the need for further development to address limitations and challenges. While LLMs have not surpassed human performance yet, collaboration between humans and AI systems holds promise for the future of healthcare.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Currently, CHAS are inconsistent in their design, analysis, and reporting.
“Currently, these studies are inconsistent in their design, analysis, and reporting.14,15,16”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1× · see citing works
“23 In this small study, we run a Turing Test of sorts, through comparison of human neurosurgical applicants with ChatGPT in a typical neurosurgical national selection style interview, generating several key findings.”
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
Artificial intelligence in education: Addressing ethical challenges in K-12 settings
Akgun, S., & Greenhow, C. (2022) · AI and Ethics · doi:10.1007/s43681-021-00096-7 · Cited by 1,473 · cited 1× · see citing works
“30 Evidently, prior to wider uptake in healthcare, LLMs such as ChatGPT have a range of issues to overcome, including moderating risk of bias and transparency issues, and answering key questions regarding ethical and equitable use.”
Artificial intelligence in surgery: promises and perils
Hashimoto, D. A., Rosman, G., Rus, D., & Meireles, O. R. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002693 · Cited by 1,468 · see citing works
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728 · see citing works
The benefits and challenges of chatgpt: An overview
Deng, J., & Lin, Y. (2023) · Frontiers in Computing and Intelligent Systems · doi:10.54097/fcis.v2i2.4465 · Cited by 611 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · see citing works
Rao, A. S., Kim, J., Kamineni, M., Pang, M., Lie, W., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.05.003 · Cited by 481 · see citing works
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
ChatGPT in Healthcare: A Taxonomy and Systematic ReviewPreprint
Li, J., Dada, A., Puladi, B., et al. (2023) · medRxiv · doi:10.1101/2023.03.30.23287899 · Cited by 411 · see citing works
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407 · cited 1× · see citing works
“24 29 Further, the use of LLMs in medical education and interview preparation is likely to expand.”
Using ChatGPT to write patient clinic letters
Ali, S. R., Dobbs, T. D., Hutchings, H. A., & Whitaker, I. S. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00048-1 · Cited by 375 · see citing works
Use and Misuse of the Likert Item Responses and Other Ordinal Measures
Bishop, P. A., & Herron, R. L. (2015) · International journal of exercise science · doi:10.70252/lanz1453 · Cited by 372 · see citing works
Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations
Ali, R., Tang, O. Y., Connolly, I. D., Sullivan, P. L. Z., Shin, J. H., Fridley, J., Asaad, W. F., Cielo, D., Oyelese, A. A., Doberstein, C. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002632 · Cited by 273 · see citing works
Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257 · see citing works
ChatGPT and antimicrobial advice: the end of the consulting infection doctor?
Howard, A., Hope, W., & Gerada, A. (2023) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(23)00113-5 · Cited by 202 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
Wadhwa, H., Shah, S. S., Shan, J., Cheng, J. K., Beniwal, A., Chen, J., Gill, S. A., Mummaneni, N. V., McDermott, M. W., Berger, M. S., et al. (2019) · Journal of neurosurgery · doi:10.3171/2019.8.jns191256 · Cited by 140 · see citing works
Hopkins, B. S., Nguyen, V., Dallas, J., Texakalidis, P., Yang, M., Renn, A., Guerra, G. A., Kashif, Z., Cheok, S. K., Zada, G., et al. (2023) · Journal of neurosurgery · doi:10.3171/2023.2.jns23419 · Cited by 84 · see citing works
Lee, R. Y., Kross, E. K., Torrence, J., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Engelberg, R. A., & Curtis, J. R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1204 · Cited by 74 · see citing works
Shelmerdine, S. C., Martin, H. M., Shirodkar, K. K., Shamshuddin, S., & Weir-McCall, J. R. (2022) · BMJ · doi:10.1136/bmj-2022-072826 · Cited by 45 · see citing works
Hatef, E., Rouhizadeh, M., Nau, C., Xie, F., Rouillard, C., Abu-Nasser, M., Padilla, A., Lyons, L. J., Kharrazi, H. H. K., Weiner, J. P., et al. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac006 · Cited by 31 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Large language models in neurosurgery: a systematic review and meta-analysis
Patil, A., Serrato, P., Chisvo, N. S., Arnaout, O. M., See, P. A., & Huang, K. T. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-06372-9 · Cited by 26 · see citing works
Pap, I. A., & Oniga, S. (2024) · Sensors · doi:10.3390/s24186140 · Cited by 20 · see citing works
Almekkawi, A. K., Caruso, J. P., Anand, S., Hawkins, A., Rauf, R., Al-Shaikhli, M., Aoun, S. G., & Bagley, C. A. (2024) · World Neurosurgery · doi:10.1016/j.wneu.2024.11.114 · Cited by 18 · see citing works
Large Language Models in Neurological Practice: Real-World Study
Maiorana, N. V., Marceglia, S. R. F., Treddenti, M., Tosi, M., Guidetti, M., Creta, M. F., Bocci, T., Oliveri, S., Boneschi, F. M., & Priori, A. (2025) · Journal of Medical Internet Research · doi:10.2196/73212 · Cited by 14 · see citing works
Huang, K. T., Mehta, N. H., Gupta, S., See, A. P., & Arnaout, O. M. (2024) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2024.03.021 · Cited by 14 · see citing works
Chopra, G., Bhaskar, P., Purohit, A., & Strzelecki, A. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13536-2 · Cited by 10 · see citing works
Employing large language models safely and effectively as a practicing neurosurgeon
Patil, A., Serrato, P., Cleaver, G., Limbania, D., See, A. P., & Huang, K. T. (2025) · Acta Neurochirurgica · doi:10.1007/s00701-025-06515-6 · Cited by 8 · see citing works
Wu, A. Q., Davis, P. H., Gajjar, A. A., Patel, S., Sollenberger, C. H., Hoglund, Z. T., Ghanta, P., Jabarkheel, R., Scott, K. W., Catapano, J. S., et al. (2026) · Cureus · doi:10.7759/cureus.106840 · Cited by 1 · see citing works
Is it time for the neurologist to use Large Language Models in everyday practice?Preprint
Maiorana, N. V., Marceglia, S. R. F., Treddenti, M., Tosi, M., Guidetti, M., Creta, M. F., Bocci, T., Oliveri, S., Boneschi, F. M., & Priori, A. (2025) · medRxiv · doi:10.1101/2025.01.23.25320945 · Cited by 1 · see citing works
From diagnostics to education: Multi-domain evaluation of LLM chatbots in neurology
Battineni, G., Chintalapudi, N., Dhulipalla, V. R., & Amenta, F. (2025) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2025.11.004
AI-Powered Patient Education and Triage Tools in Community Neurosurgery
ŞİMŞEK, K., Hızal, S., & Pehlivan, M. T. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9650-7.ch015
Shanker, V. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9650-7.ch001
- Authors
- Williams, S. C., Starup-Hansen, J., Funnell, J. P., Hanrahan, J. G., Valetopoulou, A., Singh, N., Sinha, S., Muirhead, W. R., & Marcus, H. J.
- Affiliations
- St George's Hospital (GB)
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
- Wellcome / EPSRC Centre for Interventional and Surgical Sciences (GB)
- Imperial College Healthcare NHS Trust (GB)
- Sheffield Teaching Hospitals NHS Foundation Trust (GB)
- Corresponding author
- Williams, Simon C.
- ORCID
- Venue
- British Journal of Neurosurgery
- Publisher
- Informa UK Limited
- Published
- 2024-02-02
- Volume
- 39
- Issue
- 5
- Pages
- 627-636
- ISSN
- 0268-8697 (ISSN-L)
- 1360-046X
- PMC
- PMC12090375
- OpenAlex
- W4391484746
- Cited by
- 13
- References
- 22
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 4
- FWCI
- 1.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- British Journal of Neurosurgery (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- St George's Online Research Archive (St George's University of London) (submittedVersion)
- Funders
- WEISS (NS/A000050/1)
- National Institute for Health and Care Research (NIHR) Biomedical Research Centre at University College London
- NIHR Academic Clinical Fellowship
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- ChatGPT, large language models, artificial intelligence in healthcare, virtual interviews, human-AI collaboration
- MeSH terms
- Generative Artificial Intelligence, Artificial Intelligence, Female, United Kingdom, Humans, Interviews as Topic, Male, Neurosurgery, Prospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
