GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Open accessAbstract
BACKGROUND Artificial intelligence (AI) is a burgeoning technological advancement, with considerable promise for influencing the field of medicine. As a preliminary step toward integrating AI into medical practice, it is imperative to ascertain whether model performance is comparable with that of physicians. We present a systematic comparison of performance by a large language model (LLM) versus that of a large cohort of physicians. The cohort includes all residents who took the medical specialist license examination in Israel in 2022 across the core medical disciplines: internal medicine, general surgery, pediatrics, psychiatry, and obstetrics and gynecology (OB/GYN). We provide the examinations as an accessible benchmark dataset for the medical machine learning and natural language processing communities, which may be adapted for future LLM studies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large Language Models have demonstrated expert-level accuracy on medical board examinations, performing at a level comparable to medical professionals.
“Recent efforts have included the use of official board examinations to evaluate LLM performance across different medical specialties such as pediatrics3, oncology4, ophthalmology5, radiology6, or plastic surgery7, often demonstrating that these models can perform at a level comparable to medical professionals8.”
From Large Language Models lack essential metacognition for reliable medical reasoning (2025) · cites this paper as [8]
All analyses were performed on publicly available, de-identified benchmark datasets used in accordance with their respective licenses and terms of use.
“All analyses were performed on publicly available, de-identified benchmark datasets—MIRAGE [23], PubMedQA [24], and the medical licensing examination questions of Katz et al. [25]—used in accordance with their respective licenses and terms of use.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [25]
The Katz examination provides a natural, externally-validated difficulty anchor through its physician-performance benchmark.
“the Katz examination provides a natural, externally-validated difficulty anchor through its physician-performance benchmark.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [25]
The physician performance figures used for comparison are aggregate, previously published statistics reported by Katz et al. and were not collected or re-identified by us.
“The physician performance figures used for comparison are aggregate, previously published statistics reported by Katz et al. [25] and were not collected or re-identified by us.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [25]
All models perform below the passing score of 65% and below median physician scores (70%-80% depending on specialty).
“All models perform below the passing score of 65% and below the median physician scores which ranges from 70%-80% depending on the specialty [25].”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [25]
Model scores overlapped with the bottom quartile of physicians, whose performance ranges from 40%-65% across specialties.
“However, the scores for models did overlap with the bottom quartile of physicians, who had performance ranges from 40%-65% across specialties (range inferred from Fig 2 in [25]).”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [25]
No model configuration achieved the 65% passing threshold on the medical licensing examination, and all models performed below the median physician score of 70-80%.
“No model configuration achieved the 65% passing threshold on the medical licensing examination, and all models performed below the median physician score of 70–80% [25].”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [25]
Cites
Harrison's Principles of Internal Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_2 · Cited by 12,528 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Sabiston Textbook of Surgery: The Biological Basis of Modern Surgical Practice
Townsend, C. M. (2007) · Andalas University Repository (Andalas University) · Cited by 807 · see citing works
Lång, K., Josefsson, V., Larsson, A.-M., Larsson, S., Högberg, C., Sartor, H., Hofvind, S., Andersson, I., & Rosso, A. (2023) · The Lancet Oncology · doi:10.1016/s1470-2045(23)00298-x · Cited by 598 · see citing works
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449 · see citing works
The Advent of Generative Language Models in Medical Education
Karabacak, M., Ozkara, B. B., Margetis, K., Wintermark, M., & Bisdas, S. (2023) · JMIR Medical Education · doi:10.2196/48163 · Cited by 208 · see citing works
Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187 · see citing works
Hoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08051-4 · Cited by 154 · see citing works
Performance of ChatGPT on Specialty Certificate Examination in Dermatology multiple-choice questions
Passby, L., Jenko, N., & Wernham, A. G. H. (2023) · Clinical and Experimental Dermatology · doi:10.1093/ced/llad197 · Cited by 98 · see citing works
Giannos, P. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000451 · Cited by 84 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmc2305286 · Cited by 75 · see citing works
GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors
Jang, D., Yun, T.-R., Lee, C.-Y., Kwon, Y. K., & Kim, C.-E. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000416 · Cited by 59 · see citing works
Huang, R. S., Lu, K. J. Q., Meaney, C., Kemppainen, J., Punnett, A., & Leung, F.-H. (2023) · JMIR Medical Education · doi:10.2196/50514 · Cited by 56 · see citing works
Performance of ChatGPT on free-response, clinical reasoning examsPreprint
Strong, E., DiGiammarino, A., Weng, Y., Basaviah, P., Hosamani, P., Kumar, A. D., Nevins, A., Kugler, J., Hom, J., & Chen, J. H. (2023) · medRxiv · doi:10.1101/2023.03.24.23287731 · Cited by 46 · see citing works
Meaney, C. A., Huang, R. S., Lu, K. J. Q., Fischer, A. W., Leung, F., Kulasegaram, K. M., Tzanetos, K., & Punnett, A. (2023) · medRxiv · doi:10.1101/2023.09.14.23295571 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Educational Strategies for Clinical Supervision of Artificial Intelligence Use
Abdulnour, R.-E. E., Gin, B., & Boscardin, C. K. (2025) · New England Journal of Medicine · doi:10.1056/nejmra2503232 · Cited by 209 · see citing works
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
Benchmarking Large Language Models in Evidence-Based Medicine
Li, J., Deng, Y., Sun, Q., Zhu, J., Tian, Y., Li, J., & Zhu, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483816 · Cited by 35 · see citing works
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Workum, J. D., van de Sande, D., Gommers, D., & van Genderen, M. E. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1504805 · Cited by 25 · see citing works
Benchmarking the Confidence of Large Language Models in Clinical Questions
Omar, M., Agbareia, R., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2024) · doi:10.1101/2024.08.11.24311810 · Cited by 10 · see citing works
Bejan, C. A., Reed, A. M., Mikula, M., Zhang, S., Xu, Y., Fabbri, D., Embí, P. J., & Hsi, R. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-86632-5 · Cited by 8 · see citing works
The economics of accuracy for medical reasoning with large language models
Bhattacharyya, K., & Kamabattula, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001182
“…publicly-available dataset comprises 655 multiple-choice questions drawn from the 2022 Israeli board residency examinations, administered by the Israeli Medical Association, spanning five core specialties: internal medicine, general surgery, pediatrics, psychiatry, and obstetrics/gynecology [25].”
Influential
- Authors
- Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I.
- Affiliations
- Tel Aviv University (IL)
- Tel Aviv Sourasky Medical Center (IL)
- Technion – Israel Institute of Technology (IL)
- University of Washington (US)
- St. Elizabeth's Medical Center (US)
- Jerusalem Multidisciplinary College (IL)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-04-25
- Volume
- 1
- Issue
- 5
- Pages
- AIdbp2300192-AIdbp2300192
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4394767601
- Cited by
- 135
- References
- 17
- Citations by year
- 2026: 44 · 2025: 66 · 2024: 25
- FWCI
- 12.65
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, GPT, medical residents, benchmark dataset, internal medicine, general surgery, obstetrics and gynecology, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
