CoChat

Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study

Open access

Published version of a preprint

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 20 of the 40 sources its claims cite were checked for support.

Abstract

BACKGROUND: At the beginning of their clinical clerkships (CCs), medical students face multiple challenges related to acquiring clinical and communication skills, building professional relationships, and managing psychological stress. While mentoring and structured feedback are known to provide critical support, existing systems may not offer sufficient and timely guidance owing to the faculty's limited availability. Generative artificial intelligence, particularly large language models, offers new opportunities to support medical education by providing context-sensitive responses. OBJECTIVE: This study aimed to develop a generative artificial intelligence CC mentor (AI-CCM) based on ChatGPT and evaluate its effectiveness in supporting medical students' clinical learning, addressing their concerns, and supplementing human mentoring. The secondary objective was to compare AI-CCM's educational value with responses from senior student mentors. METHODS: We conducted 2 studies. In study 1, we created 5 scenarios based on challenges that students commonly encountered during CCs. For each scenario, 5 senior student mentors and AI-CCM generated written advice. Five medical education experts evaluated these responses using a rubric to assess accuracy, practical utility, educational appropriateness (5-point Likert scale), and safety (binary scale). In study 2, a total of 17 fourth-year medical students used AI-CCM for 1 week during their CCs and completed a questionnaire evaluating its usefulness, clarity, emotional support, and impact on communication and learning (5-point Likert scale) informed by the technology acceptance model. RESULTS: All results indicated that AI-CCM achieved higher mean scores than senior student mentors. AI-CCM responses were rated higher in educational appropriateness (4.2, SD 0.7 vs 3.8, SD 1.0; P=.001). No significant differences with senior student mentors were observed in accuracy (4.4, SD 0.7 vs 4.2, SD 0.9; P=.11) or practical utility (4.1, SD 0.7 vs 4.0, SD 0.9; P=.35). No safety concerns were identified in AI-CCM responses, whereas 2 concerns were noted in student mentors' responses. Scenario-specific analysis revealed that AI-CCM performed substantially better in emotional and psychological stress scenarios. In the student trial, AI-CCM was rated as moderately useful (mean usefulness score 3.9, SD 1.1), with positive evaluations for clarity (4.0, SD 0.9) and emotional support (3.8, SD 1.1). However, aspects related to feedback guidance (2.9, SD 0.9) and anxiety reduction (3.2, SD 1.0) received more neutral ratings. Students primarily consulted AI-CCM regarding learning workload and communication difficulties; few students used it to address emotional stress-related issues. CONCLUSIONS: AI-CCM has the potential to serve as a supplementary educational partner during CCs, offering comparable support to that of senior student mentors in structured scenarios. Despite challenges of response latency and limited depth in clinical content, AI-CCM was received well by and accessible to students who used ChatGPT's free version. With further refinements, including specialty-specific content and improved responsiveness, AI-CCM may serve as a scalable, context-sensitive support system in clinical medical education.

Overview

CoChat Summary

TL;DR

AI-CCM, a ChatGPT-based clinical clerkship mentor, was developed and evaluated in a two-part study comparing its written advice with senior student mentors and testing it with fourth-year medical students. It was rated comparable to student mentors overall and higher in educational appropriateness, but the student trial showed only moderate usefulness and neutral ratings for feedback guidance and anxiety reduction.

Key findings

  • In study 1, across 5 scenarios evaluated by 5 faculty members, AI-CCM scored higher than senior student mentors for educational appropriateness (mean 4.2, SD 0.7 vs 3.8, SD 1.0; P=.001), with no significant differences in accuracy (4.4, SD 0.7 vs 4.2, SD 0.9; P=.11) or practical utility (4.1, SD 0.7 vs 4.0, SD 0.9; P=.35). No safety concerns were flagged for AI-CCM responses, whereas 2 were flagged for student mentor responses.
  • For the emotional and psychological stress scenario, AI-CCM was rated significantly higher in accuracy (4.6, SD 0.5 vs 4.0, SD 1.0; P=.02) and educational appropriateness (4.4, SD 0.8 vs 3.6, SD 1.2; P=.007).
  • In study 2, 17 fourth-year medical students used AI-CCM for 1 week; mean usefulness as a support tool was 3.9 (SD 1.1), clarity 4.0 (SD 0.9), empathy/friendliness 3.8 (SD 1.1), feedback guidance 2.9 (SD 0.9), and anxiety reduction 3.2 (SD 1.0).
  • The most common topic students asked about was educational challenges and workload (14/17, 82%); no students reported using AI-CCM for emotional and psychological stress (0/17).

Methods

  • Study 1 was a comparative evaluation: 5 scenarios based on common clerkship challenges were answered by 5 senior student mentors and AI-CCM; 5 medical education experts rated responses for accuracy, practical utility, and educational appropriateness on 5-point Likert scales plus a binary safety scale, with responses anonymized for blinding. Analysis used the Wilcoxon signed rank test in JMP Pro version 18, with P<.05.
  • Study 2 was a 1-week trial in March 2025 with 17 fourth-year medical students during their clinical clerkships; students used AI-CCM at their discretion with no minimum interactions and completed a structured 5-point Likert questionnaire based on the technology acceptance model, plus open-ended questions.
  • AI-CCM was built as a custom GPT using ChatGPT Plus and GPT-4o, with prompts informed by literature, Japanese medical education guidelines, and the Chiba University clerkship syllabus; constraints instructed it to avoid definitive diagnoses or treatment decisions and to encourage consultation with supervising physicians.

Limitations

  • The paper states that the study included few participants from a single Japanese medical school, used convenience sampling from one cohort, and had a short trial period, limiting generalizability and assessment of long-term effects.
  • The paper states that AI-CCM use logs were not comprehensively reviewed, the questionnaire and rubric were not formally validated for reliability or construct validity, evaluator blinding could not be guaranteed because AI-generated responses may have distinct linguistic patterns, and response variability across users, prompts, or repeated trials was not assessed.
  • Evident from the text: the small samples (5 mentors, 5 evaluators, 17 students) and reliance on self-report limit statistical power and may introduce bias; access via free versus Plus ChatGPT could also affect use, though this was not analyzed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/76702

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Development interface for creating generative pretrained transformers (GPTs) within ChatGPT for the artificial intelligence (A) clinical clerkship (CC) mentor. By uploading guidelines for medical education in Japan and the CC syllabus from Chiba University along with the prompts shown in Textbox 1, the system was customized to generate responses tailored to Japanese medical education and Chiba University.
Figure 1 Development interface for creating generative pretrained transformers (GPTs) within ChatGPT for the artificial intelligence (A) clinical clerkship (CC) mentor. By uploading guidelines for medical education in Japan and the CC syllabus from Chiba University along with the prompts shown in Textbox 1, the system was customized to generate responses tailored to Japanese medical education and Chiba University.CC BY · PubMed Central
Figure 2 Initial screen and sample response of the artificial intelligence clinical clerkship mentor. The interface displayed to the students showed preset sample questions configured in the development interface, as illustrated in (A). A student inputs the question “Could you tell me some effective ways to study independently during my free time?” As shown in the figure, a sample response was generated as shown in (B).
Figure 2 Initial screen and sample response of the artificial intelligence clinical clerkship mentor. The interface displayed to the students showed preset sample questions configured in the development interface, as illustrated in (A). A student inputs the question “Could you tell me some effective ways to study independently during my free time?” As shown in the figure, a sample response was generated as shown in (B).CC BY · PubMed Central

Verification

Claims the paper makes

15 Valid · 2 Not valid · 7 Not checked · 82 Own results and interpretations

60 more claims · 2 with their cited sources checked

Create a free account to see all 106 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 32
  • It may state two separate claims as one. 14
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Cited by

See all 10

Metadata

Authors
Ebihara, H., Kasai, H., Shimizu, I., Shikino, K., Tajima, H., Kimura, Y., & Ito, S.
Affiliations
  • Chiba University (JP)
  • Chiba University Hospital (JP)
Corresponding author
Kasai, Hajime
Venue
JMIR Medical Education
Published
2025-09-04
Volume
11
Pages
e76702
ISSN
2369-3762 (ISSN-L)
OpenAlex
W4414005603
Cited by
10
References
44
Citations by year
2026: 10
FWCI
2.39
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Innovations in Medical Education
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
clinical rotations, generative AI, large language models, ChatGPT, medical students, student mentors, clinical mentorship, emotional support, psychological stress, communication skills, technology acceptance, clinical learning, expert evaluation of AI responses, medical education technology
MeSH terms
Mentoring, Artificial Intelligence, Clinical Clerkship, Clinical Competence, Education, Medical, Undergraduate, Female, Humans, Male, Mentors, Students, Medical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.