CoChat
Sign Up
Cited by 276Artificial Intelligence in Healthcare and EducationInnovations in Medical EducationArtificial Intelligence in Education

ChatGPT passing USMLE shines a spotlight on the flaws of medical education

Open access
Mbakwe, Amarachi B., Lourentzou, Ismini, Celi, Leo Anthony, Mechanic, Oren J., Dagan, Alon·2023·PLOS Digital Health
DOI 10.1371/journal.pdig.0000205PMC PMC9931307
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pdig.0000205

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can synthesise knowledge across vast medical corpora but are prone to hallucination, outdated pre-training and poor traceability of sources.

    “LLMs can synthesise knowledge across vast medical corpora but are prone to hallucination, outdated pre-training and poor traceability of sources [5].”

    From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [5]

    The main text says LLMs are trained on large amounts of text and that ChatGPT can answer medical-exam questions, while warning that biased training content can shape its output. It does not address hallucination, outdated pre-training, or poor traceability of sources; those specific risks are not supported by this paper. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The need to rethink and revise current assessment tools in health care education comes in light of ChatGPT's ability to pass reputable exams (e.g., USMLE) and the possibility of ChatGPT misuse resulting in academic dishonesty.

    “The need to rethink and revise the current assessment tools in health care education comes in light of ChatGPT’s ability to pass reputable exams (e.g., USMLE) and possibility of ChatGPT misuse, which would result in academic dishonesty [24,34,58,59,62,76,85,86,87].”

    From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [34]

    The paper’s main text says ChatGPT passed an examination and argues that its success on exams such as the USMLE calls for rethinking how medical students are evaluated and for more relevant assessments. It does not discuss ChatGPT misuse causing academic dishonesty; searches for cheating and academic dishonesty found no supporting passage. Page numbers are not available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • AI contributes to the creation of personalized learning plans by considering each student's learning style, interests, and goals, thereby maintaining student motivation and engagement, which ultimately leads to enhanced academic outcomes.

    “Furthermore, AI contributes to the creation of personalized learning plans by considering each student's learning style, interests, and goals, thereby maintaining student motivation and engagement, which ultimately leads to enhanced academic outcomes (Paranjape et al. 2019; Mbakwe et al. 2023).”

    From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Mbakwe et al. 2023

    The paper's discussion of medical education calls for curiosity, critical thinking, and rethinking student assessment (main text; page not provided). Searches for personalized learning plans, learning styles, motivation, engagement, and academic outcomes found no passage linking AI to those effects. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • ChatGPT extends its reach to the realm of patient education and engagement, offering a more straightforward and patient-centered approach.

    “Furthermore, ChatGPT extends its reach to the realm of patient education and engagement, offering a more straightforward and patient-centered approach.27)”

    From Application of ChatGPT for Orthopedic Surgeries and Patient Care (2024) · cites this paper as [27]

  • Multiple-choice questions are inadequate for assessing medical knowledge because they encourage binary true/false thinking that fails to capture the complexity of medical practice.

    “They also highlight the inadequacy of multiple-choice questions as a means of assessing medical knowledge, as they tend to encourage binary thinking as “true” and “false,” which often fails to capture the complex reality of the medical practice [11].”

    From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Mbakwe, A. B., Lourentzou, I., Celi, L. A., Mechanic, O. J., & Dagan, A.
Affiliations
  • Virginia Tech (US)
  • Beth Israel Deaconess Medical Center (US)
  • Florida International University (US)
Corresponding author
Celi, Leo Anthony
ORCID
Venue
PLOS Digital Health
Publisher
Public Library of Science (PLoS)
Published
2023-02-09
Volume
2
Issue
2
Pages
e0000205
ISSN
2767-3170 (ISSN-L)
DOI
10.1371/journal.pdig.0000205
PMC
PMC9931307
OpenAlex
W4319663047
Cited by
273
References
2
Citations by year
2026: 20 · 2025: 57 · 2024: 94 · 2023: 101
Type
journal article
Language
en
Open access
Licence
Locations
  • PLOS Digital Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
ChatGPT, USMLE, medical licensing examinations, large language models, medical education, artificial intelligence in medical education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review