Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations
Abstract
Background ChatGPT is a powerful artificial intelligence large language model with great potential as a tool in medical practice and education, but its performance in radiology remains unclear. Purpose To assess the performance of ChatGPT on radiology board–style examination questions without images and to explore its strengths and limitations. Materials and Methods In this exploratory prospective study performed from February 25 to March 3, 2023, 150 multiple-choice questions designed to match the style, content, and difficulty of the Canadian Royal College and American Board of Radiology examinations were grouped by question type (lower-order [recall, understanding] and higher-order [apply, analyze, synthesize] thinking) and topic (physics, clinical). The higher-order thinking questions were further subclassified by type (description of imaging findings, clinical management, application of concepts, calculation and classification, disease associations). ChatGPT performance was evaluated overall, by question type, and by topic. Confidence of language in responses was assessed. Univariable analysis was performed. Results ChatGPT answered 69% of questions correctly (104 of 150). The model performed better on questions requiring lower-order thinking (84%, 51 of 61) than on those requiring higher-order thinking (60%, 53 of 89) (P = .002). When compared with lower-order questions, the model performed worse on questions involving description of imaging findings (61%, 28 of 46; P = .04), calculation and classification (25%, two of eight; P = .01), and application of concepts (30%, three of 10; P = .01). ChatGPT performed as well on higher-order clinical management questions (89%, 16 of 18) as on lower-order questions (P = .88). It performed worse on physics questions (40%, six of 15) than on clinical questions (73%, 98 of 135) (P = .02). ChatGPT used confident language consistently, even when incorrect (100%, 46 of 46). Conclusion Despite no radiology-specific pretraining, ChatGPT nearly passed a radiology board–style examination without images; it performed well on lower-order thinking questions and clinical management questions but struggled with higher-order thinking questions involving description of imaging findings, calculation and classification, and application of concepts. © RSNA, 2023 See also the editorial by Lourenco et al and the article by Bhayana et al in this issue.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MCQs provide a robust framework for assessing higher cognitive functions by promoting critical thinking and the application of knowledge.
“Moreover, MCQs provide a robust framework for assessing higher cognitive functions, by promoting critical thinking and the application of knowledge.12,13”
From The Scientific Knowledge of Bard and ChatGPT in Endocrinology, Diabetes, and Diabetes Technology: Multiple-Choice Questions Examination-Based Performance (2025) · cites this paper as [13]
Multiple-choice questions provide a broad range of assessments of critical thinking and provide an ample evaluation of higher cognitive functions.
“Multiple-choice questions provide a broad range of assessments of critical thinking and provide an ample evaluation of higher cognitive functions.12,13”
From The Scientific Knowledge of Bard and ChatGPT in Endocrinology, Diabetes, and Diabetes Technology: Multiple-Choice Questions Examination-Based Performance (2025) · cites this paper as [13]
ChatGPT has been shown to pass the USMLE, the German State Examination in Medicine, and a radiology board-style examination without images.
“However, even when trained for general purposes, ChatGPT has previously been shown to pass the United States Medical Licensing Examination (USMLE), the German State Examination in Medicine, or even a radiology board-style examination without images117–120.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [120]
Across other medical subspecialties, proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly, while proprietary LLM 2 answered 57% to 84% correctly.
“This improvement may explain some of the variation in performance observed across examinations from other medical subspecialties, where proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly,6,7,8,9,11,35 while proprietary LLM 2 answered 57% to 84% correctly.10,12”
From Performance of Large Language Models on Medical Oncology Examination Questions (2024) · cites this paper as [8]
Cites
(2001) · Cited by 13,930 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · see citing works
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728 · see citing works
Sawin, E. I. (1957) · The Elementary School Journal · doi:10.1086/459563 · Cited by 95 · see citing works
Measuring consumer-perceived humanness of online organizational agents
Lu, L., McDonald, C., Kelleher, T., Lee, S. S., Chung, Y., Mueller, S., Vielledent, M., & Yue, C. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107092 · Cited by 58 · see citing works
Shelmerdine, S. C., Martin, H. M., Shirodkar, K. K., Shamshuddin, S., & Weir-McCall, J. R. (2022) · BMJ · doi:10.1136/bmj-2022-072826 · Cited by 45 · see citing works
Effect of Correction-for-Guessing Formula on Psychometric Characteristics of Test
Şenel, S., Pehlivan, E. B., & Alatlı, B. K. (2015) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2015.04.221 · Cited by 10 · see citing works
Writing Good Multiple-Choice Questions: A Brief Guide for Radiologists
DiSantis, D. J. (2013) · Radiographics · doi:10.1148/rg.337125749 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192 · see citing works
The clinicians’ guide to large language models: a general perspective with a focus on hallucinations
Roustan, D., & Bastardot, F. (2025) · Interactive Journal of Medical Research · doi:10.2196/59823 · Cited by 191 · see citing works
Holderried, F., Stegemann–Philipps, C., Herschbach, L., Moldt, J.-A., Nevins, A., Griewatz, J., Holderried, M., Herrmann-Werner, A., Festl-Wietek, T., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/53961 · Cited by 156 · see citing works
Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/59213 · Cited by 152 · see citing works
Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy
Gertz, R. J., Dratsch, T., Bunck, A. C., Lennartz, S., Iuga, A.-I., Hellmich, M. G., Persigehl, T., Pennig, L., Gietzen, C. H., Fervers, P., et al. (2024) · Radiology · doi:10.1148/radiol.232714 · Cited by 143 · see citing works
Performance of large language models on a neurology board-style examination
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.46721 · Cited by 115 · see citing works
Le Guellec, B., Lefèvre, A., Geay, C., Shorten, L., Bruge, C., Hacein-Bey, L., Amouyel, P., Pruvo, J.-P., Kuchcinski, G., & Hamroun, A. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.230364 · Cited by 115 · see citing works
Zong, H., Li, J., Wu, E., Wu, R., Lu, J., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05125-7 · Cited by 114 · see citing works
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103 · see citing works
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81 · see citing works
Harnessing the Power of Generative AI for Clinical Summaries: Perspectives From Emergency Physicians
Barak-Corren, Y., Wolf, R., Rozenblum, R., Creedon, J. K., Lipsett, S. C., Lyons, T. W., Michelson, K. A., Miller, K. A., Shapiro, D. J., Reis, B. Y., et al. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.039 · Cited by 75 · see citing works
The impact of ChatGPT and LLMs on medical imaging stakeholders: Perspectives and use cases
Yang, J., Li, H. B., & Wei, D. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100007 · Cited by 46 · see citing works
Meo, S. A., Al-Khlaiwi, T., AbuKhalaf, A. A., Meo, A. S., & Klonoff, D. C. (2025) · Journal of Diabetes Science and Technology · doi:10.1177/19322968231203987 · Cited by 40 · see citing works
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41 · see citing works
Sriwastwa, A., Ravi, P., Emmert, A., Chokshi, S., Kondor, S., Dhal, K., Patel, P., Chepelev, L. L., Rybicki, F. J., & Gupta, R. (2023) · 3D Printing in Medicine · doi:10.1186/s41205-023-00186-8 · Cited by 28 · see citing works
Wada, A., Akashi, T., Shih, G., Hagiwara, A., Nishizawa, M., Hayakawa, Y., Kikuta, J., Shimoji, K., Sano, K., Kamagata, K., et al. (2024) · Diagnostics · doi:10.3390/diagnostics14141541 · Cited by 25 · see citing works
- Authors
- Bhayana, R., Krishna, S., & Bleakney, R. R.
- Affiliations
- Mount Sinai Hospital (CA)
- Venue
- Radiology
- Published
- 2023-06-01
- Volume
- 307
- Issue
- 5
- Pages
- e230582-e230582
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W4376640725
- Cited by
- 460
- References
- 12
- Citations by year
- 2026: 72 · 2025: 120 · 2024: 190 · 2023: 78
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- ChatGPT, large language models, multiple-choice questions, higher-order thinking skills, lower-order thinking, Bloom's taxonomy, radiology education, artificial intelligence in medical education, radiology physics, clinical management, Royal College of Radiologists, American Board of Radiology, radiology residency
- MeSH terms
- Artificial Intelligence, Canada, Humans, Prospective Studies, Radiography, Radiology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
