The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Background: The Objective Structured Clinical Examination (OSCE) has been widely used to evaluate students in medical education. However, it is resource-intensive, presenting challenges in implementation. We hypothesized that generative artificial intelligence (AI) such as ChatGPT-4 could serve as a complementary assessor and alleviate the burden of physicians in evaluating OSCE. Objective: By comparing the evaluation scores between generative AI and physicians, this study aims to evaluate the validity of generative AI as a complementary assessor for OSCE. Methods: This experimental study was conducted at a medical university in Japan. We recruited 11 fifth-year medical students during the general internal medicine clerkship from April 2023 to December 2023. Participants conducted a mock medical interview with a patient experiencing abdominal pain and wrote patient notes. Four physicians independently evaluated the participants by reviewing medical interview videos and patient notes, while ChatGPT-4 was provided with interview transcripts and notes. Evaluations were conducted using the 6-domain rubric (patient care and communication, history taking, physical examination, patient notes, clinical reasoning, and management). Each domain was scored using a 6-point Likert scale, ranging from 1 (very poor) to 6 (excellent). Median scores were compared using the Wilcoxon signed-rank test, and the agreement between ChatGPT-4 and physicians was assessed using intraclass correlation coefficients (ICCs). All P values <.05 were considered statistically significant. Results: Although ChatGPT-4 assigned higher scores than physicians in terms of physical examination (median 4.0, IQR 4.0-5.0 vs median 4.0, IQR 3.0-4.0; P=.02), patient notes (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 4.0-4.0; P=.002), clinical reasoning (median 5.0, IQR 5.0-5.0 vs median 4.0, IQR 3.0-4.0; P<.001), and management (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 2.5-4.5; P=.002), there were no significant differences in the scores of patient care and communication (median 5.0, IQR 5.0-5.0 vs median 5.0, IQR 4.0-5.0; P=.06) and history taking (median 5.0, IQR 4.0-5.0 vs median 5.0, IQR 4.0-5.0; P>.99), respectively. ICC values were low in all domains, except for history taking, where the agreement was still poor (ICC=0.36, 95% CI –0.32 to 0.78). Conclusions: ChatGPT-4 produced higher evaluation scores than physicians in several OSCE domains, though the agreement between them was poor. Although these preliminary results suggest that generative AI may be able to support assessment in some domains of OSCE, further research is needed to establish its reproducibility and validity. Generative AI like ChatGPT-4 shows potential as a complementary assessor for OSCE. Trial Registration: University Hospital Medical Information Network Clinical Trials Registry UMIN000050489; https://center6.umin.ac.jp/cgi-open-bin/ctr/ctr_his_list.cgi?recptno=R000057513
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“In recent years, generative artificial intelligence (AI), particularly large language models, has shown potential in medicine [10].”
Assessment of clinical competence using an objective structured clinical examination (OSCE)
HARDEN, R. M., & GLEESON, F. A. (1979) · Medical Education · doi:10.1111/j.1365-2923.1979.tb00918.x · Cited by 1,509
Preiksaitis, C., & Rose, C. (2023) · JMIR Medical Education · doi:10.2196/48785 · Cited by 552 · cited 1×
“In the context of medical education, such models may offer benefits including personalized feedback, enhanced simulation, and support for self-directed learning [11].”
Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.818634 · Cited by 490
Techniques for measuring clinical competence: objective structured clinical examinations
Newble, D. (2004) · Medical Education · doi:10.1111/j.1365-2923.2004.01755.x · Cited by 483
Proof of Concept: Using ChatGPT to Teach Emergency Physicians How to Break Bad News
Webb, J. J. (2023) · Cureus · doi:10.7759/cureus.38755 · Cited by 80
Misra, S. M., & Suresh, S. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241263475 · Cited by 73
Designing and building OSCEBot ® for virtual OSCE – Performance evaluation
Pereira, D. S. M., Falcão, F., Nunes, A., Santos, N., Costa, P., & Pêgo, J. M. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2228550 · Cited by 44
Ataro, G., Worku, S., & Asaminew, T. (2020) · Ethiopian Journal of Health Sciences · doi:10.4314/ejhs.v30i3.13 · Cited by 37 · cited 1×
“These challenges have led to implementation difficulties in some settings [9].”
Research works printed but not linked: 2
- Lee, Goldman, 2023, Goldman-Cecil Medicine, 2-Volume Set, 27th Edition
- Le, T, 2012, First Aid for the USMLE Step 2 CS, Fourth Edition
Cited by
Hirosawa, T., Yokose, M., Sakamoto, T., Hayashi, A., Harada, Y., Tokumasu, K., Mizuta, K., & Shimizu, T. (2026) · JMIR Formative Research · doi:10.2196/92016 · Cited by 3
Self-citationGenerative AI in healthcare: foundations, applications, challenges and future directions
Zheng, J., Li, B., Li, H., & Lu, Y. (2026) · Journal of Management Analytics · doi:10.1080/23270012.2026.2647940 · Cited by 6
Ndacyayisenga, J. C. Y., Kidega, C., & Aciro Can, L. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09303-7 · Cited by 1
“1 conceptualizes generative artificial intelligence (GenAI) as a validity moderator [13].”
Hirosawa, T., Yokose, M., Sakamoto, T., Hayashi, A., Harada, Y., Tokumasu, K., Mizuta, K., & Shimizu, T. (2026) · doi:10.2196/preprints.92016
Self-citationTorres, A., González, P., Fors, M. M., & Trujillo, G. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1798475
Miya, M., Akaishi, Y., Anzai, T., Suzuki, K., & Yamawaki, M. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09175-x
Implementation of Generative AI in Biomedical Research and Healthcare
Nikolopoulos, A., & Karalis, V. D. (2026) · Applied Biosciences · doi:10.3390/applbiosci5020034
The Role of Artificial Intelligence in Medical Education and Training
Scielzo, S. A., & Kazi, S. (2026) · Rheumatic Disease Clinics of North America · doi:10.1016/j.rdc.2026.04.003
Artificial intelligence in urological education: a systematic review
Liu, G., Xu, L., Gong, J., Yin, G., & Yuan, P. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09687-6
Alhaskir, M., Haven, H., Langner, M., Arasteh, S. T., Heidemeyer, H., Sacic, N., Majeed, R. W., Peters, A., Müller, N., Fries, M., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10282455/v1
Joseph-Delaffon, K., Desgrouas, M., Catanese, S., Lejeune, J., Nait-Kaci, J., Piver, É., Breteau, I., Leducq, S., Gatault, P., Khanna, R. K., et al. (2026) · doi:10.2196/preprints.108828
El papel emergente de la inteligencia artificial en los exámenes clínicos objetivos estructurados
Cahuapaza-Gutierrez, N. L., & Calderon-Hernandez, C. C. (2026) · Medical Education & Clinical Practice · doi:10.21142/mecp.2026.1.3.e032
Danneels, P., Gardic, R., Lebdai, S., Bourgeais, A., Bourreau, L., Brochard, A., Despré, M., Demarquette, A., Dhersin, R., Declerck, C., et al. (2026) · doi:10.2196/preprints.114564
Metadata
- Authors
- Yokose, M., Hirosawa, T., Sakamoto, T., Kawamura, R., Suzuki, Y., Harada, Y., & Shimizu, T.
- Affiliations
- Dokkyo Medical University (JP)
- Yamagata Prefectural Government (JP)
- Corresponding author
- Yokose, Masashi
- ORCID
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-12-04
- Volume
- 9
- Pages
- e79465
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC12715467
- OpenAlex
- W4416618182
- Cited by
- 13
- References
- 12
- Citations by year
- 2026: 13
- FWCI
- 3.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- generative AI, ChatGPT-4, OSCE, medical student assessment, AI-assisted assessment, physician assessment, assessment validity, patient interviews, clinical notes, clinical reasoning, medical history, physical examination, large language models in education, human-AI agreement
- MeSH terms
- Generative Artificial Intelligence, Artificial Intelligence, Clinical Competence, Educational Measurement, Female, Humans, Japan, Male, Students, Medical, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.