Cited by 101Artificial Intelligence in Healthcare and EducationArtificial Intelligence in EducationInnovations in Medical Education

Harnessing the potential of large language models in medical education: promise and pitfalls

Open access
Benítez, Trista M, Xu, Yueyuan, Boudreau, J Donald, Kow, Alfred Wei Chieh, Bello, Fernando, Van Phuoc, Le, Wang, Xiaofei, Sun, Xiaodong +7 more·2024·Journal of the American Medical Informatics Association

Abstract

Objectives: To provide balanced consideration of the opportunities and challenges associated with integrating Large Language Models (LLMs) throughout the medical school continuum. Process: Narrative review of published literature contextualized by current reports of LLM application in medical education. Conclusions: LLMs like OpenAI’s ChatGPT can potentially revolutionize traditional teaching methodologies. LLMs offer several potential advantages to students, including direct access to vast information, facilitation of personalized learning experiences, and enhancement of clinical skills development. For faculty and instructors, LLMs can facilitate innovative approaches to teaching complex medical concepts and fostering student engagement. Notable challenges of LLMs integration include the risk of fostering academic misconduct, inadvertent overreliance on AI, potential dilution of critical thinking skills, concerns regarding the accuracy and reliability of LLM-generated content, and the possible implications on teaching staff.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocad252

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The quality and difficulty of GPT-4 generated USMLE questions often fall below expectations.

    “Lastly, the quality and difficulty of the questions often do not meet expectations (Artsi et al., 2024; Benítez et al., 2024) (Limit3).”

    From MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025) · cites this paper as Benítez et al., 2024

    The Board exam preparation section says ChatGPT-generated USMLE-style questions were generally lower in complexity and not advanced enough to serve as a stand-alone question bank. That supports the concern about question difficulty and adequacy, but the paper does not identify the question-generating model as GPT-4. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Benítez, T. M., Xu, Y., Boudreau, J. D., Kow, A. W. C., Bello, F., Van Phuoc, L., Wang, X., Sun, X., Leung, G. K.-K., Lan, Y., Wang, Y., Cheng, D., Tham, Y.-C., Wong, T. Y., & Chung, K. C.
Affiliations
  • University of Michigan (US)
  • Tsinghua University (CN)
  • McGill University (CA)
  • National University of Singapore (SG)
  • College of Health Sciences, VinUniversity , Hanoi, 100000, Vietnam
  • Beihang University (CN)
  • Shanghai Jiao Tong University (CN)
  • Department of Surgery, School of Clinical Medicine, LKS Faculty of Medicine, The University of Hong Kong, Queen Mary Hospital , Hong Kong, 999077, China
  • Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Beijing Ophthalmology and Visual Sciences Key Laboratory, Capital University of Medical Science , Beijing, 100730, China
  • Chinese University of Hong Kong, Shenzhen (CN)
  • Singapore National Eye Center (SG)
Corresponding author
  • Tham, Yih-Chung
  • Chung, Kevin C
Venue
Journal of the American Medical Informatics Association
Published
2024-01-24
Volume
31
Issue
3
Pages
776-783
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4391225179
Cited by
100
References
51
Citations by year
2026: 25 · 2025: 55 · 2024: 19 · 2013: 1
FWCI
2.26
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion, cc-by-nc-nd)
Funders
  • National Key Research and Development Program of China (2022YFC2502802)
  • National Natural Science Foundation of China (8238810007)
  • National Institutes of Health
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Keywords
large language models, ChatGPT, medical education, AI in teaching, personalized learning, clinical skills development, academic misconduct, overreliance on AI, critical thinking, medical curriculum
MeSH terms
Clinical Competence, Education, Medical, Humans, Language, Learning, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review