Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study

May combine two claims.

Excluding image-dependent questions did not significantly alter examination difficulty, as evidenced by similar student scores.

“The exclusion of these questions did not significantly alter examination difficulty, as evidenced by similar student scores (Figure 2D).”

• AssumptionRests on an assumption: ChatGPT (GPT-3.5 and GPT-4) cannot analyze visual content.
• AssumptionRests on an assumption: The MEDI-LEARN examination statistics accurately reflect the number of correct student answers and the specialization of each question.

May combine two claims.

GPT-4's average score ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the October 2021, April 2022, and October 2022 examinations, respectively.

“GPT-4 achieved an average score of 85% and ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the same examinations in October 2021, April 2022, and October 2022, respectively.”

• cites [37] DietrichJLeoncioWcitation: Software Citation ToolsZenodo2024-01-28https://zenodo.org/records/3909438Not checked yet

  Not checked in the free run.

• cites [38] 2008, R: A Language and Environment for Statistical ComputingNot checked yet

  Not checked in the free run.

• cites [39] Herbst 2021 - ErgebnisinformartionInstitut für medizinische und pharmazeutische Prüfungsfragen2024-01-28https://www.impp.de/pruefungen/medizin/archiv-medizin.html?file=files/PDF/Pr%C3%BCfungsergebnisse/Pr%C3%BCfungsergebnisse/ErgMedM2H2021APPO2012.pdfNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The official correct answers provided by IMPP are the correct answers.
• AssumptionRests on an assumption: The MEDI-LEARN examination statistics accurately reflect the number of correct student answers and the specialization of each question.

May combine two claims.

Upon excluding image-based questions, GPT-4 continued to outperform GPT-3.5, with scores approaching 91.44%.

“Upon excluding image-based questions, GPT-4 continued to outperform GPT-3.5, with scores approaching 91.44%.”

• AssumptionRests on an assumption: ChatGPT (GPT-3.5 and GPT-4) cannot analyze visual content.
• AssumptionRests on an assumption: The official correct answers provided by IMPP are the correct answers.

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of images remained constant throughout the examinations (P=.07).

“GPT-3.5’s accuracy (P=.66), the number of images (P=.07), and students’ accuracy (P=.77) remained constant throughout the examinations, whereas GPT-4’s accuracy (P=.02), the specialties (P<.001), and question type (P=.04) varied (Multimedia Appendix 4 and Figures 2A, 2B, and 2E).”

• AssumptionRests on an assumption: ChatGPT (GPT-3.5 and GPT-4) cannot analyze visual content.

Probably not a substantial claim: may not pass S3 (load-bearing).

The specialties varied across the examinations (P<.001).

“GPT-3.5’s accuracy (P=.66), the number of images (P=.07), and students’ accuracy (P=.77) remained constant throughout the examinations, whereas GPT-4’s accuracy (P=.02), the specialties (P<.001), and question type (P=.04) varied (Multimedia Appendix 4 and Figures 2A, 2B, and 2E).”

May combine two claims.

In comparison to German medical students, GPT-3.5 scored in the 8.6th percentile, while GPT-4 ranked in the 92.8th, 99.5th, and 92.6th percentiles in the October 2021, April 2022, and October 2022 German medical licensing examinations.

“In comparison to the German medical students, however, GPT-3.5 scored in the 8.6th percentile, while GPT-4 ranked in the 92.8th, 99.5th, and 92.6th percentiles in the October 2021, April 2022, and October 2022 examinations [39,45,46].”

• Also stated in: § Conclusions · hedged

  “the results of this study indicate that only GPT-4 consistently passed all 3 medical examinations, ranking in the 92.8th to 99.5th percentile in comparison to medical students.”

• cites [39] Herbst 2021 - ErgebnisinformartionInstitut für medizinische und pharmazeutische Prüfungsfragen2024-01-28https://www.impp.de/pruefungen/medizin/archiv-medizin.html?file=files/PDF/Pr%C3%BCfungsergebnisse/Pr%C3%BCfungsergebnisse/ErgMedM2H2021APPO2012.pdfNot checked yet

  Not checked in the free run.

• cites [45] Frühjahr 2022 - ErgebnisinformartionInstitut für medizinische und pharmazeutische Prüfungsfragen2024-01-28https://www.impp.de/pruefungen/medizin/archiv-medizin.html?file=files/PDF/Pr%C3%BCfungsergebnisse/Pr%C3%BCfungsergebnisse/ErgMedM2F2022APPO2012.pdfNot checked yet

  Not checked in the free run.

• cites [46] Herbst 2022 - ErgebnisinformartionInstitut für medizinische und pharmazeutische Prüfungsfragen2024-01-28https://www.impp.de/pruefungen/medizin/archiv-medizin.html?file=files/PDF/Prüfungsergebnisse/Prüfungsergebnisse/ErgMedM2H2022.pdfNot checked yet

  Not checked in the free run.

May combine two claims.

GPT-4, unlike GPT-3.5, did not answer questions containing images on repetition, showing an improvement over the previously incorrect content produced by GPT-3.5.

“GPT-4, unlike GPT-3.5, also did not answer questions containing images on repetition, showing an improvement in the previously incorrect content produced by GPT-4’s predecessor [17].”

• cites [17] OpenAIGPT-4 Technical Report2024-01-28https://cdn.openai.com/papers/gpt-4.pdfNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The collected parameters did not differ in terms of GPT-3.5's and GPT-4's accuracy in the excluded cases.

“Although the collected parameters did not differ in terms of GPT-3.5’s and GPT-4’s accuracy in the excluded cases, the decision to exclude image-based questions may have introduced a sampling bias.”

May combine two claims.

GPT-3.5 displayed a high degree of accuracy within psychiatry, while GPT-4 demonstrated its strength in internal medicine and surgery.

“GPT-3.5 initially displayed a high degree of accuracy within the field of psychiatry, while GPT-4 demonstrated its strength in internal medicine and surgery.”

May combine two claims.

GPT-3.5 was imprecise and inconsistent.

“While GPT-4’s predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims. The quote only partly states this claim.

GPT-4 demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results.

“While GPT-4’s predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results.”

• AssumptionRests on an assumption: The 3 written German medical licensing examinations used are representative of the written German medical licensing examination.

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 passed a Japanese medical examination.

“Still, with assistance and adaptations, GPT-3.5 nearly passed the United States Medical Licensing Examination [13,14], and GPT-4 passed a Japanese medical examination [15].”

• cites [15] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-3.5 nearly passed the United States Medical Licensing Examination.

“Still, with assistance and adaptations, GPT-3.5 nearly passed the United States Medical Licensing Examination [13,14], and GPT-4 passed a Japanese medical examination [15].”

• cites [13] sourceNot checked yet

  Not checked in the free run.

• cites [14] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Both GPT-3.5 and GPT-4 were not primarily trained on medical data.

“These concerns cast doubt on the medical proficiency of these LLMs, as both were not primarily trained on medical data and are the first to admit that as a language AI model, passing a medical examination is outside their skillset (Multimedia Appendix 1).”

• cites Multimedia Appendix 1Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

SMOG has shown acceptable interrater reliability for patient education materials.

“To assess the readability of a question, we used the Simple Measure of Gobbledygook (SMOG) as it has shown acceptable interrater reliability for patient education materials in the literature [21].”

• cites [21] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The German medical licensing examination has a high interexamination reliability of over 0.9.

“While the German medical licensing examination covers various medical subdisciplines in 320 multiple-choice questions [18], it has a high interexamination reliability of over 0.9 [19].”

• cites [19] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Concerns about lack of transparency regarding GPT-4's model parameters, training process, and underlying data structure remain unaddressed.

“However, despite the introduction of the more advanced iteration GPT-4, concerns about the lack of transparency regarding this AI’s model parameters, training process, and underlying data structure remain unaddressed [8,12].”

• cites [8] Kurz, C, 2023, Dtsch Arztebl InternationalNot checked yet

  Not checked in the free run.

• cites [12] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-3.5 iteration has already shown promise in several routine medical tasks and medical research.

“ChatGPT’s GPT-3.5 iteration has already shown promise in several routine medical tasks and medical research [4-7], even raising ethical concerns in the literature [2,3,8].”

• cites [4] sourceNot checked yet

  Not checked in the free run.

• cites [5] sourceNot checked yet

  Not checked in the free run.

• cites [6] sourceNot checked yet

  Not checked in the free run.

• cites [7] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-3.5's and GPT-4's performance fell short compared to that of medical students in a Japanese medical examination.

“Despite these improvements, GPT-3.5’s or GPT-4’s performance still fell short in comparison to that of medical students in a Japanese medical examination according to the study by Takagi et al [15].”

• cites [15] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Language is a limiting factor for ChatGPT's accuracy, so globally consistent application depends on users' proficiency in English.

“Since ChatGPT appears to perform optimally with English inputs, language emerges as a limiting factor for its accuracy, suggesting that globally consistent application is dependent upon users' proficiency in English.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study did not exclude all image-based questions a priori; instead, it evaluated the relevance of images for each question and compared the results both with and without images.

“Furthermore, we did not exclude all image-based questions a priori. Instead, we evaluated the relevance of the images for each question and compared the results both with and without images.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Multiple-choice questions are inadequate for assessing medical knowledge because they encourage binary true/false thinking that fails to capture the complexity of medical practice.

“They also highlight the inadequacy of multiple-choice questions as a means of assessing medical knowledge, as they tend to encourage binary thinking as “true” and “false,” which often fails to capture the complex reality of the medical practice [11].”

• cites [11] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Testing for differences was done to minimize the sampling bias from excluding image-based questions and to maintain the integrity of the results.

“By testing for differences, efforts were made to minimize this bias and maintain the integrity of the results.”

May combine two claims.

GPT-4 has been benchmarked against medical licensing examinations, which explains its excellent scores and its exceeding achievements in internal medicine and surgery, areas overrepresented in the examinations assessed in this study.

“It seems evident that GPT-4 has been benchmarked against medical licensing examinations, explaining not only GPT-4’s excellent scores but also exceeding achievements in internal medicine and surgery, which, for instance, have been overrepresented in the medical examinations assessed in this study [12,17].”

• cites [12] sourceNot checked yet

  Not checked in the free run.

• cites [17] OpenAIGPT-4 Technical Report2024-01-28https://cdn.openai.com/papers/gpt-4.pdfNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Assessing the possible reasons for the performance differences between GPT-3.5 and GPT-4 is complicated by GPT-4's confidential architecture.

“However, assessing the possible reasons for the performance differences between GPT-3.5 and GPT-4 is complicated by the confidential architecture of GPT-4 [54], posing challenges for research on future applications.”

• cites [54] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

GPT-3.5 and GPT-4 have only been tested on medical examination questions and not on questions by nonprofessionals, which limits general recommendations for unsupervised patient education or the general population.

“it must be noted that GPT-3.5 and GPT-4 have only been tested on medical examination questions and not questions by nonprofessionals, limiting general recommendations for unsupervised patient education or the general population.”

Probably not a substantial claim: may not pass S2 (checkable).

GPT-4's achievements shed light on the limitations of current testing paradigms in medical education, which often favor rote memorization over a critical and context-aware approach.

“In turn, GPT-4’s excellent achievements shed light on the limitations of current testing paradigms in medical education that often favor rote memorization over a critical and context-aware approach.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4's performance is not limited to the medical field, as it achieved comparable percentile scores in the Uniform Bar Exam.

“GPT-4’s impressive performance is not limited to the medical field, as it demonstrated comparable percentile scores in the Uniform Bar Exam, showcasing it potential as a versatile tool across diverse academic disciplines [17].”

• cites [17] OpenAIGPT-4 Technical Report2024-01-28https://cdn.openai.com/papers/gpt-4.pdfNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The decision to exclude image-based questions may have introduced a sampling bias.

“Although the collected parameters did not differ in terms of GPT-3.5’s and GPT-4’s accuracy in the excluded cases, the decision to exclude image-based questions may have introduced a sampling bias.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

GPT-3.5 and GPT-4 have demonstrated similar performances in examinations in other countries and languages, which suggests some degree of generalizability.

“However, it is worth noting that GPT-3.5 and GPT-4 have demonstrated similar performances in examinations in other countries and languages, which suggests some degree of generalizability.”

Classified as probably not a substantial claim. May combine two claims.

Neither GPT-3.5 nor GPT-4 verifies the information they provide.

“Although GPT-3.5 and GPT-4 allow the simple and fast retrieval of medical information from any internet-capable device that fits in one's pocket [9,10], neither GPT-3.5 nor GPT-4 verifies the information they provide.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

A basic understanding in data science seems necessary alongside traditional medical expertise to navigate changing informational landscapes.

“To navigate these changing informational landscapes, a basic understanding in data science seems necessary alongside traditional medical expertise [56].”

• cites [56] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

ChatGPT's output needs to be approached with a critical mindset, because misinformation may be more difficult to detect than in the output of other search engines that offer multiple sources and take login credentials into account.

“Thus, ChatGPT's output needs to be approached with a critical mindset, recognizing that misinformation may be more difficult to detect than in the output of other search engines that offer multiple sources in response to a query and take login credentials into account [8,55].”

• cites [8] Kurz, C, 2023, Dtsch Arztebl InternationalNot checked yet

  Not checked in the free run.

• cites [55] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There is potential for using GPT-3.5 and GPT-4 in a medical education tutoring environment, as evidenced by their successful application in anatomy.

“For instance, there is potential for using GPT-3.5 and GPT-4 in a medical education tutoring environment, as evidenced by its successful application in anatomy [47].”

• cites [47] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The trends observed in this study appear to be consistent with those reported in other published and preprint studies on GPT-3.5's and GPT-4's performance.

“However, the trends observed in this study appear to be consistent with those reported in other published and preprint studies on GPT-3.5’s and GPT-4’s performance.”

Classified as probably not a substantial claim. May combine two claims.

Although GPT-3.5's and GPT-4's accuracy in medical examinations seems consistent across different countries and languages, their inconsistencies, potential biases, and number of incorrect answers restrain a recommendation for their use by the general population for medical purposes.

“Although GPT-3.5’s and GPT-4’s accuracy in medical examinations seems consistent across different countries and languages, its inconsistencies, potential biases, and number of incorrect answers restrain a recommendation for its use by the general population for medical purposes.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

GPT-3.5's and GPT-4's elaborate explanations and potential to yield nonobvious insights may benefit medical professionals in training.

“However, its elaborate explanations and potential to yield nonobvious insights may benefit medical professionals in training.”

Classified as probably not a substantial claim.

When using GPT-3.5 or GPT-4 for medical applications, their differing accuracy across specialties must be taken into account.

“However, when using either GPT-3.5 or GPT-4 for medical applications, its differing accuracy across specialties must also be taken into account [48].”

• cites [48] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Health care professionals could potentially benefit from GPT-4's conclusive and often nonobvious insights to multiple-choice questions, as these users have the ability to verify crucial details.

“Thus, health care professionals could potentially benefit, especially from GPT-4’s conclusive and often nonobvious insights to multiple-choice questions, as these users have the ability to verify crucial details [13,14,41].”

• cites [13] sourceNot checked yet

  Not checked in the free run.

• cites [14] sourceNot checked yet

  Not checked in the free run.

• cites [41] sourceNot checked yet

  Not checked in the free run.