CoChat

The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study

Open access
Yokose, Masashi, Hirosawa, Takanobu, Sakamoto, Tetsu, Kawamura, Ren, Suzuki, Yudai, Harada, Yukinori, Shimizu, Taro · 2025 · JMIR Formative Research

Published version of a preprint

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

Abstract

Background: The Objective Structured Clinical Examination (OSCE) has been widely used to evaluate students in medical education. However, it is resource-intensive, presenting challenges in implementation. We hypothesized that generative artificial intelligence (AI) such as ChatGPT-4 could serve as a complementary assessor and alleviate the burden of physicians in evaluating OSCE. Objective: By comparing the evaluation scores between generative AI and physicians, this study aims to evaluate the validity of generative AI as a complementary assessor for OSCE. Methods: This experimental study was conducted at a medical university in Japan. We recruited 11 fifth-year medical students during the general internal medicine clerkship from April 2023 to December 2023. Participants conducted a mock medical interview with a patient experiencing abdominal pain and wrote patient notes. Four physicians independently evaluated the participants by reviewing medical interview videos and patient notes, while ChatGPT-4 was provided with interview transcripts and notes. Evaluations were conducted using the 6-domain rubric (patient care and communication, history taking, physical examination, patient notes, clinical reasoning, and management). Each domain was scored using a 6-point Likert scale, ranging from 1 (very poor) to 6 (excellent). Median scores were compared using the Wilcoxon signed-rank test, and the agreement between ChatGPT-4 and physicians was assessed using intraclass correlation coefficients (ICCs). All P values <.05 were considered statistically significant. Results: Although ChatGPT-4 assigned higher scores than physicians in terms of physical examination (median 4.0, IQR 4.0-5.0 vs median 4.0, IQR 3.0-4.0; P=.02), patient notes (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 4.0-4.0; P=.002), clinical reasoning (median 5.0, IQR 5.0-5.0 vs median 4.0, IQR 3.0-4.0; P<.001), and management (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 2.5-4.5; P=.002), there were no significant differences in the scores of patient care and communication (median 5.0, IQR 5.0-5.0 vs median 5.0, IQR 4.0-5.0; P=.06) and history taking (median 5.0, IQR 4.0-5.0 vs median 5.0, IQR 4.0-5.0; P>.99), respectively. ICC values were low in all domains, except for history taking, where the agreement was still poor (ICC=0.36, 95% CI –0.32 to 0.78). Conclusions: ChatGPT-4 produced higher evaluation scores than physicians in several OSCE domains, though the agreement between them was poor. Although these preliminary results suggest that generative AI may be able to support assessment in some domains of OSCE, further research is needed to establish its reproducibility and validity. Generative AI like ChatGPT-4 shows potential as a complementary assessor for OSCE. Trial Registration: University Hospital Medical Information Network Clinical Trials Registry UMIN000050489; https://center6.umin.ac.jp/cgi-open-bin/ctr/ctr_his_list.cgi?recptno=R000057513

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/79465

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Study flow diagram of the experimental evaluation of generative artificial intelligence in medical education. The study was conducted at Dokkyo Medical University, Japan, between April and December 2023. A total of 11 fifth-year medical students were recruited during their general internal medicine clerkship. Each participant completed a mock medical interview with a standardized patient. Physicians evaluated each student by using both the video recordings and written patient notes, while ChatGPT-4 conducted its evaluation based on interview transcripts and patient notes.
Figure 1 Study flow diagram of the experimental evaluation of generative artificial intelligence in medical education. The study was conducted at Dokkyo Medical University, Japan, between April and December 2023. A total of 11 fifth-year medical students were recruited during their general internal medicine clerkship. Each participant completed a mock medical interview with a standardized patient. Physicians evaluated each student by using both the video recordings and written patient notes, while ChatGPT-4 conducted its evaluation based on interview transcripts and patient notes.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. Lee, Goldman, 2023, Goldman-Cecil Medicine, 2-Volume Set, 27th Edition
  2. Le, T, 2012, First Aid for the USMLE Step 2 CS, Fourth Edition

Cited by

See all 13

Metadata

Authors
Yokose, M., Hirosawa, T., Sakamoto, T., Kawamura, R., Suzuki, Y., Harada, Y., & Shimizu, T.
Affiliations
  • Dokkyo Medical University (JP)
  • Yamagata Prefectural Government (JP)
Corresponding author
Yokose, Masashi
Venue
JMIR Formative Research
Published
2025-12-04
Volume
9
Pages
e79465
ISSN
2561-326X (ISSN-L)
OpenAlex
W4416618182
Cited by
13
References
12
Citations by year
2026: 13
FWCI
3.13
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Formative Research (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Innovations in Medical Education
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
generative AI, ChatGPT-4, OSCE, medical student assessment, AI-assisted assessment, physician assessment, assessment validity, patient interviews, clinical notes, clinical reasoning, medical history, physical examination, large language models in education, human-AI agreement
MeSH terms
Generative Artificial Intelligence, Artificial Intelligence, Clinical Competence, Educational Measurement, Female, Humans, Japan, Male, Students, Medical, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.