CoChat
Sign Up
Cited by 98Artificial Intelligence in Healthcare and EducationArtificial Intelligence in EducationInnovations in Medical Education

Artificial Intelligence in Medical Education: Comparative Analysis of ChatGPT, Bing, and Medical Students in Germany

Open access
Roos, Jonas, Kasapovic, Adnan, Jansen, Tom, Kaczmarczyk, Robert·2023·JMIR Medical Education
DOI 10.2196/46482PMC PMC10507517

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Large language models (LLMs) have demonstrated significant potential in diverse domains, including medicine. Nonetheless, there is a scarcity of studies examining their performance in medical examinations, especially those conducted in languages other than English, and in direct comparison with medical students. Analyzing the performance of LLMs in state medical examinations can provide insights into their capabilities and limitations and evaluate their potential role in medical education and examination preparation. Objective: This study aimed to assess and compare the performance of 3 LLMs, GPT-4, Bing, and GPT-3.5-Turbo, in the German Medical State Examinations of 2022 and to evaluate their performance relative to that of medical students. Methods: The LLMs were assessed on a total of 630 questions from the spring and fall German Medical State Examinations of 2022. The performance was evaluated with and without media-related questions. Statistical analyses included 1-way ANOVA and independent samples t tests for pairwise comparisons. The relative strength of the LLMs in comparison with that of the students was also evaluated. Results: GPT-4 achieved the highest overall performance, correctly answering 88.1% of questions, closely followed by Bing (86.0%) and GPT-3.5-Turbo (65.7%). The students had an average correct answer rate of 74.6%. Both GPT-4 and Bing significantly outperformed the students in both examinations. When media questions were excluded, Bing achieved the highest performance of 90.7%, closely followed by GPT-4 (90.4%), while GPT-3.5-Turbo lagged (68.2%). There was a significant decline in the performance of GPT-4 and Bing in the fall 2022 examination, which was attributed to a higher proportion of media-related questions and a potential increase in question difficulty. Conclusions: LLMs, particularly GPT-4 and Bing, demonstrate potential as valuable tools in medical education and for pretesting examination questions. Their high performance, even relative to that of medical students, indicates promising avenues for further development and integration into the educational and clinical landscape.

AI snapshot

Written from the paper’s full text.

TL;DR

This study compared GPT-4, Bing, and GPT-3.5-Turbo on 630 German Medical State Examination 2022 questions and found GPT-4 and Bing outperformed medical students overall, while GPT-3.5-Turbo lagged. GPT-4 and Bing declined significantly in the fall 2022 examination, attributed partly to more media-related questions and possible increased difficulty.

Key findings

  • Including media questions, GPT-4 answered 555/630 (88.1%) correctly, Bing 542/630 (86.0%), GPT-3.5-Turbo 414/630 (65.7%), and students averaged 469.7/630 (74.6%). GPT-4 and Bing significantly outperformed students in both spring and fall 2022; GPT-3.5-Turbo was not significantly different from students in spring but was significantly lower in fall.
  • Excluding media questions (590 total), Bing answered 517/590 (90.7%) correctly, GPT-4 515/590 (90.4%), and GPT-3.5-Turbo 389/590 (68.2%). No student comparison was possible for nonmedia questions because official individual-question student data were unavailable.
  • GPT-4 and Bing showed significant spring-to-fall declines: GPT-4 +29 correct answers (+6.9%) and Bing +34 (+8.6%) in spring versus fall when including media; GPT-3.5-Turbo showed no significant difference between examinations. Relative model strength versus students with media was 116.6% for GPT-4 and 115.4% for Bing in spring, and 110.4% and 107.5% in fall; GPT-3.5-Turbo ranged from 88.5% to 95.1%.

Methods

  • Retrospective analysis of 630 of 640 German Medical State Examination multiple-choice questions from spring and fall 2022; 1 spring and 9 fall questions were excluded post hoc for factual incorrectness.
  • GPT-3.5-Turbo and GPT-4 were queried via OpenAI’s Python API on June 9, 2023; Bing was queried between June 9 and 13, 2023, using EdgeGPT version 0.10.7 with precise settings. Prompts were in German based on a scheme from prior studies; student results came from the Institute for Medical and Pharmaceutical Examination Questions.
  • One-way ANOVA and independent samples t tests were used for comparisons; analyses were repeated excluding media questions (38/311 spring and 22/319 fall excluded), and model strength was calculated relative to student performance.

Limitations

  • The paper states that the evaluation was limited to OpenAI’s models, single prompts and answers, and only 1 year’s German Medical State Examinations; it notes the models’ inability to process media, lack of diversity in examination content and languages, rapid LLM evolution that may make results outdated, and that misinformation, inaccuracy, intrinsic biases, and errors were not investigated.
  • Evident from the text: because no officially published student performance on individual questions was available, the models’ nonmedia performance could not be compared with students; media questions were inaccessible to the models, which may confound comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/46482

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 12 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the fall 2022 examination, Bing significantly outperformed the students (P=.02, t620=2.26).

    “In contrast, both GPT-4 (P=.001, t620=3.23) and Bing (P=.02, t620=2.26) again significantly outperformed the students (Figure 1).”
    Quote in the paperSubstantial claim
  • Findingown data

    In the fall 2022 examination, GPT-3.5-Turbo's performance was significantly lower than that of the students (P=.002, t620=-3.10).

    “In the fall 2022 examination, GPT-3.5-Turbo’s performance was significantly lower than that of the students (P=.002, t620=–3.10).”
    Quote in the paperSubstantial claim
  • Findingown data

    In the spring 2022 examination, GPT-4 significantly outperformed the students (P<.001, t636=5.751).

    “However, both GPT-4 (P<.001, t636=5.751) and Bing (P<.001, t636=5.22) significantly outperformed the students.”
    Quote in the paperSubstantial claim
  • Findingown data

    The students had an average correct answer rate of 469.7 out of 630 (74.6%) overall including media.

    “The students performed in between (469.7/630, 74.6%).”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT-4 performed significantly better than GPT-3.5-Turbo overall including media (t1258=9.77, P<.001).

    “However, both Bing (t1258=8.67, P<.001) and GPT-4 (t1258= 9.77, P<.001) performed significantly better than GPT-3.5-Turbo.”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 10

  • The quote only partly states this claim.

    Findingown data

    One-way ANOVA revealed significant differences among the 3 models on the 630 questions including media (F2=64.1, P<.001).

    “One-way ANOVA revealed significant differences among the 3 models (F2=64.1, P<.001).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    In the fall 2022 examination including media, GPT-3.5-Turbo answered 195 out of 311 (62.7%) questions correctly.

    “On the fall 2022 examination, both the LLMs and the students performed slightly worse: GPT-4 answered 263 out of 311 (84.6%) questions correctly, Bing answered 254 out of 311 (81.7%) questions correctly, and GPT-3.5-Turbo answered 195 out of 311 (62.7%) questions correctly.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The consistent performance of students and GPT-3.5-Turbo suggests there may have been more questions with a higher degree of difficulty in the fall 2022 examination.

    “The consistent performance of the students and GPT-3.5-Turbo also suggests that there may have been more questions with a higher degree of difficulty in the fall 2022 examination; this also aligns with a current negative trend in the results of the state examinations, with a large variation among individual examinations [20].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • The quote only partly states this claim.

    Interpretationown data

    The significant decline in GPT-4 and Bing performance in the fall 2022 examination was attributed to a potential increase in question difficulty.

    “There was a significant decline in the performance of GPT-4 and Bing in the fall 2022 examination, which was attributed to a higher proportion of media-related questions and a potential increase in question difficulty.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In contrast to ChatGPT, Bing AI actively searches the internet for pertinent content and can provide relevant sources for its specific responses.

    “In contrast to ChatGPT, Bing AI actively searches the internet for pertinent content and can provide relevant sources for its specific responses [14].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Introducing the new BingBing2023-06-22https://www.bing.com/new

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Potential biases or errors intrinsic to the models were not explored.

    “Lastly, potential biases or errors intrinsic to the models were not explored.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The training data sets for GPT-3.5-Turbo consist of 93% English-language content.

    “The training data sets for GPT-3.5-Turbo consist of 93% English-language content [10].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Brown, T, arXiv

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GPT-3.5-Turbo and GPT-4 have been trained on data sets up until approximately September 2021.

    “GPT-3.5-Turbo and GPT-4 have been trained on data sets up until approximately September 2021 [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] BreenAChatGPT Just Got a Game-Changing Update — Here's What to KnowEntrepreneur20232023-06-22https://www.entrepreneur.com/business-news/openais-chatgpt-just-got-a-powerful-update-what-to-know/448427

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study did not investigate the potential for misinformation or inaccuracy in the responses generated by the LLMs.

    “The study did not investigate the potential for misinformation or inaccuracy in the responses generated by the LLMs, which is critical given that medical students might rely on these tools for preparation for examinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The inability of the models to process media content and the lack of diversity in examination content and languages confines the scope of insights.

    “The inability of the models to process media content and the lack of diversity in examination content and languages confines the scope of insights.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Their high performance, even relative to that of medical students, indicates promising avenues for further development and integration into the educational and clinical landscape.

    “Their high performance, even relative to that of medical students, indicates promising avenues for further development and integration into the educational and clinical landscape.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    LLMs, particularly GPT-4 and Bing, demonstrate potential as valuable tools in medical education and for pretesting examination questions.

    “LLMs, particularly GPT-4 and Bing, demonstrate potential as valuable tools in medical education and for pretesting examination questions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Bing is an AI chatbot developed by Microsoft and uses technology from GPT-4 to enhance its accuracy and performance capabilities.

    “Bing is an AI chatbot developed by Microsoft and has been unveiled in 2023. In the development process, technology from GPT-4 was used to enhance its accuracy and performance capabilities [12,13].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [12] LaukkonenJWhat Is Microsoft's Bing AI Chatbot?Lifewire20232023-06-22https://www.lifewire.com/what-is-bing-ai-chatbot-7371141

      Not checked in the free run.

    • cites [13] MehdiYReinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the webOfficial Microsoft Blog20232023-03-29https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There is a current negative trend in the results of the state examinations, with a large variation among individual examinations.

    “this also aligns with a current negative trend in the results of the state examinations, with a large variation among individual examinations [20].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ChatGPT is already being increasingly used in the field of radiology, where it can aid in education and assist in making clinical decisions.

    “ChatGPT is already being increasingly used in the field of radiology, where it can aid in education and assist in making clinical decisions [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Compared to GPT-3.5-Turbo, GPT-4 can understand more nuanced instructions and questions and is expected to provide false information less frequently; it is more likely to refuse queries that could result in harmful responses.

    “Compared to GPT-3.5-Turbo, its successor GPT-4 can understand more nuanced instructions and questions and is expected to provide false information less frequently. Moreover, it is more likely to refuse queries that could result in harmful responses [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] MartindaleJGPT-4 vs. ChatGPT: just how much better is the latest version?Digital Trends20232023-06-22https://www.digitaltrends.com/computing/gpt-4-vs-chatgpt/

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ChatGPT suffers from hallucination problems; as it does not have access to an external knowledge base, more extrinsic hallucinations are generated.

    “However, like other LLMs, ChatGPT suffers from hallucination problems. As it does not have access to an external knowledge base, more extrinsic hallucinations are generated [7].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Questions that were excluded post hoc for factual incorrectness were excluded from further analysis (1 from spring and 9 from fall).

    “Questions that were excluded post hoc for factual incorrectness were excluded from further analysis (1 from among 320 questions from the spring examination and 9 from 320 questions from the fall examination).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    For analysis without media content, 38 out of 311 (12.2%) questions were excluded for the spring examination and 22 out of 319 (6.9%) questions for the fall examination.

    “For analysis without media content (eg, questions containing images), 38 out of 311 (12.2%) questions were excluded for the spring examination and 22 out of 319 (6.9%) questions for the fall examination.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The evaluation was limited to OpenAI's models and focused on single prompts and answers from just 1 year's German Medical State Examinations.

    “In this study, the evaluation was limited to OpenAI's models and focused on single prompts and answers from just 1 year’s German Medical State Examinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is perceived potential in using LLMs such as ChatGPT as web-based teaching assistants to offer students detailed and relevant information.

    “Moreover, there is perceived potential in using LLMs such as ChatGPT as web-based teaching assistants to offer students detailed and relevant information [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] source

      Not checked in the free run.

21 of 21

  • The paper relies on this without support.

    The questions taken from Amboss are representative of the official German Medical State Examinations.

    “The questions were taken from the learning platform Amboss [15].”

  • The paper relies on this without support.

    The students' results obtained from the Institute for Medical and Pharmaceutical Examination Questions are comparable to the model performance on the same set of questions.

    “The students’ results were obtained from the Institute for Medical and Pharmaceutical Examination Questions [18].”

  • The paper relies on this without support.

    The LLM responses collected on June 9-13, 2023 are representative of the models' capabilities.

    “We used OpenAI’s Python application programming interface [16] to query prompts for the base models of GPT-3.5-Turbo and GPT-4 on June 9, 2023, and the Bing queries were made between the June 9 and 13, 2023, using the custom Python library EdgeGPT (version 0.10.7; Binedge.ai) and the precise settings [17].”

  • The paper relies on this without support.

    The current versions of the LLMs cannot process media content.

    “An explanation for the poorer performance is the higher proportion of media-related questions in the fall 2022 examination, as there is no image recognition in the current version of the LLMs.”

  • The paper relies on this without support.

    The German Medical State Examination questions are a valid measure of medical knowledge for comparing LLMs and medical students.

    “This study critically examines the capabilities of ChatGPT and Bing in medical education, by addressing how Bing, GPT-4, and GPT-3.5-Turbo perform in answering multiple-choice questions on the German Medical State Examination of 2022.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several studies examined the performance of LLMs on various medical exams; comparisons between older and newer versions indicated that newer versions exhibited better accuracy; one study comparing ChatGPT and Google's Bard in answering multiple-choice questionnaires related to endocrinology and diabetes found ChatGPT to be slightly superior, but both models lacked up-to-date information; another

    “Several studies examined the performance of LLMs on various medical exams [97–99]. Comparisons between older and newer versions indicated that newer versions exhibited better accuracy, highlighting the need for ongoing assessment of AI tools in medical education [100–102]. One study comparing ChatGPT and Google’s Bard in answering multiple-choice questionnaires (MCQs) related to endocrinology and”

    From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [100]

    Split into 2 atoms — 1 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Bing's reported nonmedia percentage does not use its stated denominator

    Minor

    The reported 90.7% is consistent with 517 out of 570, but 517 out of the stated 590 is 87.6%. The other two models' nonmedia percentages likewise use 570 rather than the stated 590.

    Reported 90.7 · recomputed 87.627119

  • Spring and fall media exclusions are reversed

    Minor

    The statistical-analysis section assigns 38 exclusions and 311 total questions to spring. Table 1 instead gives spring 319 total and 297 without media, implying 22 exclusions; its fall figures imply 38.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Brown, T, arXiv

Cited by

Sorted by citations

Related

Authors
Roos, J., Kasapovic, A., Jansen, T., & Kaczmarczyk, R.
Affiliations
  • University Hospital Bonn (DE)
  • Technical University of Munich (DE)
Corresponding author
Kaczmarczyk, Robert
ORCID
Venue
JMIR Medical Education
Publisher
JMIR Publications Inc.
Published
2023-09-04
Volume
9
Pages
e46482
ISSN
2369-3762 (ISSN-L)
DOI
10.2196/46482
PMC
PMC10507517
OpenAlex
W4386407038
Semantic Scholar
e30c825af7e13054b5cb2bc7b237d94a62ef5b39
Cited by
98
References
10
Citations by year
2026: 11 · 2025: 32 · 2024: 51 · 2023: 4
FWCI
3.90
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Keywords
large language models, ChatGPT, Bing, GPT-4, GPT-3.5-turbo, medical licensing examinations, medical students, LLM performance comparison, question difficulty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review