Performance of a large language model on practice questions for the neonatal board examination.
Open accessAbstract
This Diagnostic/Prognostic Study evaluates the performance of a large language model in generating answers to practice questions for the neonatal-perinatal board examination.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Across other medical subspecialties, proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly, while proprietary LLM 2 answered 57% to 84% correctly.
“This improvement may explain some of the variation in performance observed across examinations from other medical subspecialties, where proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly,6,7,8,9,11,35 while proprietary LLM 2 answered 57% to 84% correctly.10,12”
From Performance of Large Language Models on Medical Oncology Examination Questions (2024) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199 · see citing works
Self-citationThe Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106 · see citing works
Self-citationMachine Learning and Statistics in Clinical Research Articles—Moving Past the False Dichotomy
Finlayson, S. G., Beam, A. L., & van Smeden, M. (2023) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2023.0034 · Cited by 24 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [4]Morton S, Ehret D, Ghanta S, Sajti E, Walsh B. In: Brodsky D, Martin CR, eds. Neonatology Review: Q & A. 3rd ed. Lulu; 2015. [Google Scholar]
- [5]Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. arXiv. Published online December 26, 2022. https://arxiv.org/abs/2212.13138 · arXiv
Cited by
Sorted by citations
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81 · see citing works
Riedel, M., Kaefinger, K., Stuehrenberg, A., Ritter, V., Amann, N., Graf, A., Recker, F., Klein, E., Kiechle, M., Riedel, F., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1296615 · Cited by 49 · see citing works
- Authors
- Beam, K., Sharma, P., Kumar, B., Wang, C., Brodsky, D., Martin, C. R., & Beam, A.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- Boston Children's Hospital (US)
- Harvard University (US)
- Weill Cornell Medicine (US)
- Corresponding author
- Beam, Andrew
- Venue
- JAMA Pediatrics
- Publisher
- American Medical Association (AMA)
- Published
- 2023-09-01
- Volume
- 177
- Issue
- 9
- Pages
- 977
- ISSN
- 1072-4710 (ISSN-L)
- 2168-6203
- PMC
- PMC10352922
- OpenAlex
- W4384561376
- Cited by
- 43
- References
- 4
- Citations by year
- 2026: 12 · 2025: 10 · 2024: 16 · 2023: 5
- FWCI
- 1.60
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Archives of Pediatrics and Adolescent Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (T32HD098061)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- large language models, practice questions, answer generation, neonatal-perinatal medicine, medical licensing examination preparation
- MeSH terms
- Certification, Humans, Infant, Newborn, Language, Specialty Boards
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
