ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions.
Open accessAbstract
PURPOSE: With the increasing adoption of artificial intelligence (AI) in various domains, including healthcare, there is growing acceptance and interest in consulting AI models to provide medical information and advice. This study aimed to evaluate the accuracy of ChatGPT's responses to practice quiz questions designed for otolaryngology board certification and decipher potential performance disparities across different otolaryngology subspecialties. METHODS: A dataset covering 15 otolaryngology subspecialties was collected from an online learning platform funded by the German Society of Oto-Rhino-Laryngology, Head and Neck Surgery, designed for board certification examination preparation. These questions were entered into ChatGPT, with its responses being analyzed for accuracy and variance in performance. RESULTS: The dataset included 2576 questions (479 multiple-choice and 2097 single-choice), of which 57% (n = 1475) were answered correctly by ChatGPT. An in-depth analysis of question style revealed that single-choice questions were associated with a significantly higher rate (p < 0.001) of correct responses (n = 1313; 63%) compared to multiple-choice questions (n = 162; 34%). Stratified by question categories, ChatGPT yielded the highest rate of correct responses (n = 151; 72%) in the field of allergology, whereas 7 out of 10 questions (n = 65; 71%) on legal otolaryngology aspects were answered incorrectly. CONCLUSION: The study reveals ChatGPT's potential as a supplementary tool for otolaryngology board certification preparation. However, its propensity for errors in certain otolaryngology areas calls for further refinement. Future research should address these limitations to improve ChatGPT's educational use. An approach, with expert collaboration, is recommended for the reliable and accurate integration of such AI models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Across other medical subspecialties, proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly, while proprietary LLM 2 answered 57% to 84% correctly.
“This improvement may explain some of the variation in performance observed across examinations from other medical subspecialties, where proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly,6,7,8,9,11,35 while proprietary LLM 2 answered 57% to 84% correctly.10,12”
From Performance of Large Language Models on Medical Oncology Examination Questions (2024) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.1144 · Cited by 305 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.1 · Cited by 278 · see citing works
The exciting potential for ChatGPT in obstetrics and gynecology
Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A., & Chervenak, F. A. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.009 · Cited by 234 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
ChatGPT in Clinical Toxicology
Sabry Abdel-Messih, M., & Kamel Boulos, M. N. (2023) · JMIR Medical Education · doi:10.2196/46876 · Cited by 95 · see citing works
Application of radiomics and machine learning in head and neck cancers
Peng, Z., Wang, Y.-M., Wang, Y.-X., Jiang, S., Fan, R.-H., Zhang, H., & Jiang, W.-H. (2021) · International Journal on Biological Sciences · doi:10.7150/ijbs.55716 · Cited by 91 · see citing works
Development of Artificial Intelligence for Parathyroid Recognition During Endoscopic Thyroid Surgery
Wang, B., Zheng, J., Yu, J., Lin, S., Yan, S., Zhang, L., Wang, S., Cai, S., Ahmed, A. H. A., Lin, L., et al. (2022) · The Laryngoscope · doi:10.1002/lary.30173 · Cited by 45 · see citing works
Vimont, A., Leleu, H., & Durand-zaleski, I. (2021) · European Journal of Health Economics · doi:10.1007/s10198-021-01363-4 · Cited by 36 · see citing works
Knoedler, L., Miragall, M. F., Kauke‐Navarro, M., et al. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12101739 · Cited by 36 · see citing works
Self-citationKnoedler, L., Baecher, H., Kauke‐Navarro, M., Prantl, L., Machens, H., Scheuermann, P., Palm, C., Baumann, R., Kehrer, A., Panayi, A. C., et al. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11174998 · Cited by 32 · see citing works
Self-citationAutomatic Question Answering for Medical MCQs: Can It Go Further than Information Retrieval?
Ha, L. A., & Yaneva, V. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_049 · Cited by 19 · see citing works
A machine learning approach to high‐risk cardiac surgery risk scoring
Rogers, M. P., Janjua, H. M., Fishberger, G., Harish, A., Sujka, J. A., Toloza, E. M., DeSantis, A. J., Hooker, R. L., Pietrobon, R. S., Lozonschi, L., et al. (2022) · Journal of Cardiac Surgery · doi:10.1111/jocs.17110 · Cited by 11 · see citing works
Beyond the Buzzwords: Artificial Intelligence in Laryngology
Melley, L. E., & Sataloff, R. T. (2021) · Journal of Voice · doi:10.1016/j.jvoice.2021.03.001 · Cited by 5 · see citing works
HNO-Wissen bei HNO-Ärzten: Eine Analyse der Quizfragen in der HNO-App
Johannsen, J., Shabli, S., Klußmann, J. P., et al. (2023) · Laryngo-Rhino-Otologie · doi:10.1055/a-2036-7843 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I. (2024) · NEJM AI · doi:10.1056/aidbp2300192 · Cited by 135 · see citing works
ChatGPT in Teaching and Learning: A Systematic Review
Ali, D., Fatemi, Y., Boskabadi, E., Nikfar, M., Ugwuoke, J., & Ali, H. (2024) · Education Sciences · doi:10.3390/educsci14060643 · Cited by 146 · see citing works
Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) · JMIR Medical Education · doi:10.2196/50658 · Cited by 99 · see citing works
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81 · see citing works
Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines
Frosolini, A., Franz, L., Benedetti, S., Vaira, L. A., de Filippis, C., Gennaro, P., Marioni, G., & Gabriele, G. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08205-4 · Cited by 63 · see citing works
ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69 · see citing works
Self-citationLechien, J. R., Briganti, G., & Vaira, L. A. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08441-8 · Cited by 54 · see citing works
Embracing the future: Integrating ChatGPT into China’s nursing education system
Ni, Z., Peng, R., Zheng, X., & Xie, P. (2024) · International Journal of Nursing Sciences · doi:10.1016/j.ijnss.2024.03.006 · Cited by 35 · see citing works
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25 · see citing works
Arbel, Y., Gimmon, Y., & Shmueli, L. (2025) · Physical Therapy · doi:10.1093/ptj/pzaf010 · Cited by 21 · see citing works
Benchmarking the Confidence of Large Language Models in Clinical Questions
Omar, M., Agbareia, R., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2024) · doi:10.1101/2024.08.11.24311810 · Cited by 10 · see citing works
- Authors
- Hoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M.
- Affiliations
- TUM Klinikum (DE)
- University of Cologne (DE)
- Brigham and Women's Hospital (US)
- Harvard University (US)
- Queen Mary University of London (GB)
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Hoch, Cosima C.
- ORCID
- Venue
- European Archives of Oto-Rhino-Laryngology
- Published
- 2023-06-07
- Volume
- 280
- Issue
- 9
- Pages
- 4271-4278
- ISSN
- 0937-4477 (ISSN-L)
- 1434-4726
- PMC
- PMC10382366
- OpenAlex
- W4379599010
- Cited by
- 154
- References
- 18
- Citations by year
- 2026: 18 · 2025: 37 · 2024: 70 · 2023: 28
- FWCI
- 6.11
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- European Archives of Oto-Rhino-Laryngology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- Funders
- Technische Universität München
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, otolaryngology subspecialties, AI accuracy, medical education, allergology
- MeSH terms
- Artificial Intelligence, Certification, Educational Status, Humans, Otolaryngology, Referral and Consultation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
