ChatGPT in medical school: how successful is AI in progress testing?
Open accessAbstract
Background: As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine.Given that knowledge acquisition is a basic determinant of physicians' performance, teaching and testing different levels of medical knowledge is a central task of medical schools.To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test.Methods: A total of 400 multiple-choice questions (MCQs) from the progress test in Germanspeaking countries were entered into ChatGPT's user interface to obtain the percentage of correctly answered questions.We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question.Results: Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct.On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words.There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = -0.08,95% CI [-0.18, 0.02], t(393) = -1.55,p = 0.121; for word count rho = -0.03,95% CI [-0.13, 0.07], t(393) = -0.54,p = 0.592).There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002).Conclusion: ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1-3.The ChatGPT answers can be compared with the performance of medical students in the second half of their studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Despite remarkable performance on standardized tests, generative AI models still commit significant errors, and their performance is often on par with that of novice learners.
“Despite the remarkable performance of these models on standardized tests, they still commit significant errors, and their performance is often on par with that of novice learners [32,35,36].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [36]
The Results report that ChatGPT answered 65.5% of progress-test questions correctly, supporting the claim that it made substantial errors despite strong test performance. But the Discussion says it outperformed almost all first- to third-year medical students; only students from the fourth year onward achieved similar results. This paper therefore does not support describing its performance as on par with novice learners. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAn overdependence on generative AI could potentially curtail learners' capacities for critical thinking and intricate problem-solving.
“An overdependence on this technology could potentially curtail learners’ capacities for critical thinking and intricate problem-solving [15,24,25,36].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [36]
Dependence on AI could potentially compromise learning opportunities or skill development that arises from task completion without assistance.
“Dependence on AI could potentially compromise learning opportunities or skill development that arises from task completion without assistance [15,24,25,36].”
From Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023) · cites this paper as [36]
The integration of ChatGPT into medical practices and healthcare settings presents challenges and ethical considerations.
“The integration of ChatGPT into medical practices and healthcare settings presents challenges and ethical considerations.”
One challenge is that ChatGPT's responses are based on the data on which it was trained, which may result in biased or inaccurate information.
“One challenge is that the responses of ChatGPT are based on the data on which it was trained, which may result in biased or inaccurate information.”
Ethical concerns arise when content generated by AI is mistaken for the expertise of a human.
“Ethical concerns arise when content generated by AI is mistaken for the expertise of a human.”
Medical education can benefit significantly from AI-driven tools like ChatGPT.
“Medical education can benefit significantly from AI-driven tools like ChatGPT.”
ChatGPT's ability to explain intricate medical terminology in a comprehensible manner aids in knowledge acquisition, fostering continuous learning and improving medical literacy.
“Its ability to explain intricate medical terminology in a comprehensible manner aids in knowledge acquisition, fostering continuous learning and improving medical literacy [6,9,13,14,15,16,17,18,19,20,21,22,23].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Ericsson, K. A. (2004) · Academic Medicine · doi:10.1097/00001888-200410001-00022 · Cited by 2,988 · cited 1× · see citing works
“This approach aligns with the medical thinking and problem-solving skills that are crucial in a professional context [32].”
BackgroundHow numeracy influences risk comprehension and medical decision making.
Reyna, V. F., Nelson, W. L., Han, P. K. J., & Dieckmann, N. F. (2009) · Psychological Bulletin · doi:10.1037/a0017327 · Cited by 1,172 · see citing works
American Medical Education 100 Years after the Flexner Report
Cooke, M., Irby, D. M., Sullivan, W., & Ludmerer, K. M. (2006) · New England Journal of Medicine · doi:10.1056/nejmra055445 · Cited by 883 · see citing works
Measuring Risk Literacy: The Berlin Numeracy Test
Cokely, E. T., Galešić, M., Schulz, E., Ghazal, S., & García‐Retamero, R. (2012) · Judgment and Decision Making · doi:10.1017/s1930297500001819 · Cited by 752 · see citing works
Peters, E. M. (2012) · Current Directions in Psychological Science · doi:10.1177/0963721411429960 · Cited by 369 · see citing works
Different written assessment methods: what can be said about their strengths and weaknesses?
Schuwirth, L. W. T., & Van Der Vleuten, C. P. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01916.x · Cited by 309 · see citing works
Fifteen years of experience with progress testing in a problem-based learning curriculum
van der Vleuten, C., Verwijnen, G. M., & Wijnen, W. (1996) · Medical Teacher · doi:10.3109/01421599609034142 · Cited by 304 · see citing works
Palmer, E. J., & Devitt, P. G. (2007) · BMC Medical Education · doi:10.1186/1472-6920-7-49 · Cited by 294 · see citing works
Rethinking Health Numeracy: A Multidisciplinary Literature Review
Ancker, J. S., & Kaufman, D. R. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2464 · Cited by 276 · see citing works
Cognition and the question of test item format
Martinez, M. E. (1999) · Educational Psychologist · doi:10.1207/s15326985ep3404_2 · Cited by 242 · see citing works
Objectives, evaluation, and the improvement of education
Anderson, L. W. (2005) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2005.05.004 · Cited by 189 · see citing works
A systemic framework for the progress test: Strengths, constraints and issues: AMEE Guide No. 71
Wrigley, W. J., van der Vleuten, C. P. M., Freeman, A., & Muijtjens, A. M. M. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.704437 · Cited by 159 · see citing works
Rodriguez, M. C. (2003) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2003.tb01102.x · Cited by 145 · see citing works
Fischer, M. R., Bauer, D., Mohn, K., & NKLM-Projektgruppe (2015) · Europe PMC (PubMed Central) · doi:10.3205/zma000977 · Cited by 139 · see citing works
Doctor scores on national qualifying examinations predict quality of care in future practice
Wenghofer, E. F., KLASS, D. J., Abrahamowicz, M., et al. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03534.x · Cited by 135 · see citing works
Progress testing internationally
Freeman, A., van der Vleuten, C. P. M., Nouns, Z. M., et al. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.485231 · Cited by 119 · see citing works
Progress testing in German speaking countries
Nouns, Z. M., & Georg, W. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.485656 · Cited by 72 · see citing works
der Veken, J. V., Valcke, M., Maeseneer, J. D., et al. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03397.x · Cited by 53 · see citing works
Johnson, T. R., Khalil, M. K., Peppler, R. D., Davey, D. D., & Kibble, J. D. (2014) · AJP Advances in Physiology Education · doi:10.1152/advan.00047.2014 · Cited by 44 · see citing works
How to write short cases for assessing problem-solving skills
Schuwirth, L. (1999) · Medical Teacher · doi:10.1080/01421599979761 · Cited by 39 · see citing works
Bianchi, F., Stobbe, K., & Eva, K. W. (2008) · Medical Teacher · doi:10.1080/01421590701754144 · Cited by 39 · see citing works
Morrison, C. A., Ross, L. P., Fogle, T. T., Butler, A., Miller, J., & Dillon, G. F. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181ed3f5c · Cited by 33 · see citing works
Karay, Y., & Schauber, S. K. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1472370 · Cited by 30 · see citing works
Glew, R. H., RIPKEY, D. R., & Swanson, D. B. (1997) · Academic Medicine · doi:10.1097/00001888-199712000-00022 · Cited by 29 · see citing works
Görlich, D., & Friederichs, H. (2021) · Medical Education Online · doi:10.1080/10872981.2021.1972505 · Cited by 26 · see citing works
Self-citationWang, L., Laird-Fick, H. S., Parker, C. J., et al. (2021) · BMC Medical Education · doi:10.1186/s12909-021-02633-8 · Cited by 25 · see citing works
Friederichs, H., Birkenstein, R., Becker, J. C., Marschall, B., & Weissenstein, A. (2020) · BMC Family Practice · doi:10.1186/s12875-020-01214-w · Cited by 22 · see citing works
Self-citationPeeraer, G., Winter, B. Y. D., Muijtjens, A. M. M., et al. (2008) · Medical Teacher · doi:10.1080/01421590802512920 · Cited by 15 · see citing works
Anderson, D. L., & Price, D. J. D. S. (1962) · Technology and Culture · doi:10.2307/3101441 · Cited by 13 · see citing works
Schauber, S. K., Hautz, S. C., Kämmer, J. E., Stroben, F., & Hautz, W. E. (2021) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10052-z · Cited by 12 · see citing works
Patient numeracy: what do patients need to recognize, think, or do with health numbers?
Zikmund‐Fisher, B. J., Mayman, G., & Fagerlin, A. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139644358.005 · Cited by 11 · see citing works
Friederichs, H., Schölling, M., Marschall, B., & Weissenstein, A. (2013) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039.2013.821909 · Cited by 7 · see citing works
Self-citationsierra-local: A lightweight standalone application for drug resistance prediction
Ho, J. C., Ng, G. T., Renaud, M. S., et al. (2019) · The Journal of Open Source Software · doi:10.21105/joss.01186 · Cited by 7 · see citing works
Physicians’ understanding and use of numeric information
Anderson, B. L., & Schulkin, J. A. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139644358.004 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [13]Nouns Z, Hanfler S, Brauns K, et al. Do progress tests predict the outcome of national exams. Short Communication. AMEE Conference, 2004. Edinburgh.
- [16]Wang L, Laird-Fick HS, Parker CJ, et al. Using Markov chain model to evaluate medical students’ trajectory on progress tests and predict USMLE step 1 scores. 2021; (Preprint). doi: 10.21203/rs.3.rs-147714/v1 PMC8033658 33836741 · DOI
Cited by
Sorted by citations
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
Preiksaitis, C., & Rose, C. (2023) · JMIR Medical Education · doi:10.2196/48785 · Cited by 468 · see citing works
“Despite the remarkable performance of these models on standardized tests, they still commit significant errors, and their performance is often on par with that of novice learners [32,35,36].”
BackgroundInfluentialYounis, H. A., Eisa, T. A. E., Nasser, M., Sahib, T. M., Noor, A. A., Alyasiri, O. M., Salisu, S., Hayder, I. M., & Younis, H. A. (2024) · Diagnostics · doi:10.3390/diagnostics14010109 · Cited by 242 · see citing works
Samala, A. D., Rawas, S., Wang, T., Reed, J. M., Kim, J., Howard, N.-J., & Ertz, M. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-12936-0 · Cited by 229 · see citing works
Weidener, L., & Fischer, M. (2024) · JMIR Medical Education · doi:10.2196/51247 · Cited by 163 · see citing works
Large language models for generating medical examinations: systematic review
Artsi, Y., Sorin, V., Konen, E., Glicksberg, B. S., Nadkarni, G., & Klang, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05239-y · Cited by 128 · see citing works
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103 · see citing works
ChatGPT for assessment writing
Zuckerman, M., Flood, R., Tan, R. J. B., Kelp, N., Ecker, D. J., Menke, J., & Lockspeiser, T. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2249239 · Cited by 83 · see citing works
Faisal, E. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1325601 · Cited by 44 · see citing works
Meo, S. A., Al-Khlaiwi, T., AbuKhalaf, A. A., Meo, A. S., & Klonoff, D. C. (2025) · Journal of Diabetes Science and Technology · doi:10.1177/19322968231203987 · Cited by 40 · see citing works
Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33 · see citing works
Glimpses of the Use of Generative AI and ChatGPT in Medical Education
Human Biology Division, S. O. M., Mitra, N. K., Chitra, E., & School of Health Sciences, I. U. (2024) · Education in Medicine Journal · doi:10.21315/eimj2024.16.2.11 · Cited by 15 · see citing works
- Authors
- Friederichs, H., Friederichs, W. J., & März, M.
- Affiliations
- Bielefeld University (DE)
- RWTH Aachen University (DE)
- Humboldt-Universität zu Berlin (DE)
- Corresponding author
- Friederichs, Hendrik
- Venue
- Medical Education Online
- Publisher
- Informa UK Limited
- Published
- 2023-06-12
- Volume
- 28
- Issue
- 1
- Pages
- 2220920-2220920
- ISSN
- 1087-2981 (ISSN-L)
- PMC
- PMC10262795
- OpenAlex
- W4380356334
- Cited by
- 122
- References
- 35
- Citations by year
- 2026: 15 · 2025: 36 · 2024: 51 · 2023: 19
- FWCI
- 4.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Medical Education Online (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RWTH Publications (RWTH Aachen) (publishedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
- Funders
- funding
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, generative AI, medical education, progress testing, medical students, multiple-choice questions, factual knowledge, question difficulty, response time, word count, licensing examinations
- MeSH terms
- Machine Learning, Artificial Intelligence, Education, Medical, Educational Measurement, Humans, Schools, Medical, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
