Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 20 of the 40 sources its claims cite were checked for support.
Abstract
BACKGROUND: At the beginning of their clinical clerkships (CCs), medical students face multiple challenges related to acquiring clinical and communication skills, building professional relationships, and managing psychological stress. While mentoring and structured feedback are known to provide critical support, existing systems may not offer sufficient and timely guidance owing to the faculty's limited availability. Generative artificial intelligence, particularly large language models, offers new opportunities to support medical education by providing context-sensitive responses. OBJECTIVE: This study aimed to develop a generative artificial intelligence CC mentor (AI-CCM) based on ChatGPT and evaluate its effectiveness in supporting medical students' clinical learning, addressing their concerns, and supplementing human mentoring. The secondary objective was to compare AI-CCM's educational value with responses from senior student mentors. METHODS: We conducted 2 studies. In study 1, we created 5 scenarios based on challenges that students commonly encountered during CCs. For each scenario, 5 senior student mentors and AI-CCM generated written advice. Five medical education experts evaluated these responses using a rubric to assess accuracy, practical utility, educational appropriateness (5-point Likert scale), and safety (binary scale). In study 2, a total of 17 fourth-year medical students used AI-CCM for 1 week during their CCs and completed a questionnaire evaluating its usefulness, clarity, emotional support, and impact on communication and learning (5-point Likert scale) informed by the technology acceptance model. RESULTS: All results indicated that AI-CCM achieved higher mean scores than senior student mentors. AI-CCM responses were rated higher in educational appropriateness (4.2, SD 0.7 vs 3.8, SD 1.0; P=.001). No significant differences with senior student mentors were observed in accuracy (4.4, SD 0.7 vs 4.2, SD 0.9; P=.11) or practical utility (4.1, SD 0.7 vs 4.0, SD 0.9; P=.35). No safety concerns were identified in AI-CCM responses, whereas 2 concerns were noted in student mentors' responses. Scenario-specific analysis revealed that AI-CCM performed substantially better in emotional and psychological stress scenarios. In the student trial, AI-CCM was rated as moderately useful (mean usefulness score 3.9, SD 1.1), with positive evaluations for clarity (4.0, SD 0.9) and emotional support (3.8, SD 1.1). However, aspects related to feedback guidance (2.9, SD 0.9) and anxiety reduction (3.2, SD 1.0) received more neutral ratings. Students primarily consulted AI-CCM regarding learning workload and communication difficulties; few students used it to address emotional stress-related issues. CONCLUSIONS: AI-CCM has the potential to serve as a supplementary educational partner during CCs, offering comparable support to that of senior student mentors in structured scenarios. Despite challenges of response latency and limited depth in clinical content, AI-CCM was received well by and accessible to students who used ChatGPT's free version. With further refinements, including specialty-specific content and improved responsiveness, AI-CCM may serve as a scalable, context-sensitive support system in clinical medical education.
Overview
CoChat Summary
TL;DR
AI-CCM, a ChatGPT-based clinical clerkship mentor, was developed and evaluated in a two-part study comparing its written advice with senior student mentors and testing it with fourth-year medical students. It was rated comparable to student mentors overall and higher in educational appropriateness, but the student trial showed only moderate usefulness and neutral ratings for feedback guidance and anxiety reduction.
Key findings
- In study 1, across 5 scenarios evaluated by 5 faculty members, AI-CCM scored higher than senior student mentors for educational appropriateness (mean 4.2, SD 0.7 vs 3.8, SD 1.0; P=.001), with no significant differences in accuracy (4.4, SD 0.7 vs 4.2, SD 0.9; P=.11) or practical utility (4.1, SD 0.7 vs 4.0, SD 0.9; P=.35). No safety concerns were flagged for AI-CCM responses, whereas 2 were flagged for student mentor responses.
- For the emotional and psychological stress scenario, AI-CCM was rated significantly higher in accuracy (4.6, SD 0.5 vs 4.0, SD 1.0; P=.02) and educational appropriateness (4.4, SD 0.8 vs 3.6, SD 1.2; P=.007).
- In study 2, 17 fourth-year medical students used AI-CCM for 1 week; mean usefulness as a support tool was 3.9 (SD 1.1), clarity 4.0 (SD 0.9), empathy/friendliness 3.8 (SD 1.1), feedback guidance 2.9 (SD 0.9), and anxiety reduction 3.2 (SD 1.0).
- The most common topic students asked about was educational challenges and workload (14/17, 82%); no students reported using AI-CCM for emotional and psychological stress (0/17).
Methods
- Study 1 was a comparative evaluation: 5 scenarios based on common clerkship challenges were answered by 5 senior student mentors and AI-CCM; 5 medical education experts rated responses for accuracy, practical utility, and educational appropriateness on 5-point Likert scales plus a binary safety scale, with responses anonymized for blinding. Analysis used the Wilcoxon signed rank test in JMP Pro version 18, with P<.05.
- Study 2 was a 1-week trial in March 2025 with 17 fourth-year medical students during their clinical clerkships; students used AI-CCM at their discretion with no minimum interactions and completed a structured 5-point Likert questionnaire based on the technology acceptance model, plus open-ended questions.
- AI-CCM was built as a custom GPT using ChatGPT Plus and GPT-4o, with prompts informed by literature, Japanese medical education guidelines, and the Chiba University clerkship syllabus; constraints instructed it to avoid definitive diagnoses or treatment decisions and to encourage consultation with supervising physicians.
Limitations
- The paper states that the study included few participants from a single Japanese medical school, used convenience sampling from one cohort, and had a short trial period, limiting generalizability and assessment of long-term effects.
- The paper states that AI-CCM use logs were not comprehensively reviewed, the questionnaire and rubric were not formally validated for reliability or construct validity, evaluator blinding could not be guaranteed because AI-generated responses may have distinct linguistic patterns, and response variability across users, prompts, or repeated trials was not assessed.
- Evident from the text: the small samples (5 mentors, 5 evaluators, 17 students) and reliance on self-report limit statistical power and may introduce bias; access via free versus Plus ChatGPT could also affect use, though this was not analyzed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
15 Valid · 2 Not valid · 7 Not checked · 82 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“mentoring effectiveness can be influenced by the compatibility between mentors and mentees, including differences in communication styles, expectations, and learning needs [22].”
§ Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study > Introduction > Background
What the sources say
- Partly supportedcites [22] sourceExperimental — accuracy not yet measured
The Discussion identifies matching mentors and mentees on shared characteristics, such as clinical experience or demographics, as important to program success. The Introduction also describes benefits of shared language and understanding, but the paper does not specifically link differences in communication styles, expectations, or learning needs between a pair to mentoring effectiveness. Page numbers were not available in our copy. (LLM, conf=0.85)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“peer mentoring typically requires coordination of real-time in-person or online meetings, which may hinder timely access to support in clinical settings [23].”
§ Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study > Introduction > Background
What the sources say
- Partly supportedcites [23] sourceExperimental — accuracy not yet measured
The Reflections and lessons section says busy mentor and mentee schedules can make reaching out for unofficial mentorship challenging, and that official programs require administrative coordination. It does not say peer mentoring typically requires real-time in-person or online meetings, or that scheduling delays timely support in clinical settings. No page numbers are available in our copy. (LLM, conf=0.78)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The trial survey results aligned with 2 TAM factors: perceived usefulness and perceived ease of use [36].”
§ Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study > Discussion > Principal Findings
What the sources say
- Supportedcites [36] source
The theoretical framework identifies perceived usefulness and perceived ease of use as TAM factors. In Results (Assessment of the research model, Table 7), the survey supports both factors’ associations with attitudes toward using LLMs, consistent with the claimed alignment. (LLM, conf=0.88)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Moreover, while hierarchical structures in clinical settings often hinder communication with senior students and supervising physicians [15-17], AI-CCM provides a nonhierarchical environment, potentially enabling medical students to seek advice and express concerns more freely.”
§ Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study > Discussion > Principal Findings
What the sources say
- Partly supportedcites [15] sourceExperimental — accuracy not yet measured
The Results section, “The persistence of hierarchy and exclusivity” (page unavailable), reports unfair treatment of students in lower ward positions and says senior staff’s verbal abuse seriously damaged team relations. This supports the cited point that hierarchy can impede relationships in clinical settings, but the paper does not specifically report hindered communication with senior students and supervising physicians. The statement about AI-CCM is the citing authors’ own proposal, not a finding attributed to this source. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [16] source
- Not checked — no full text of the sourcecites [17] source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, some students may have hesitated to use the system for psychological concerns owing to apprehensions about entering sensitive personal information into an AI tool—a hesitation also noted in previous research [47].”
§ Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study > Discussion > Principal Findings
What the sources say
- Partly supportedcites [47] sourceExperimental — accuracy not yet measured
The Results report that 78.3% of surveyed medical students worried about AI-related privacy issues, and the Discussion links privacy concerns to attitudes toward adopting AI. The paper does not address using an AI system for psychological concerns or reluctance to enter sensitive personal information into one, so it supports only the broader privacy-related hesitation. (LLM, conf=0.83)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
60 more claims · 2 with their cited sources checked
Create a free account to see all 106 claims and every source verdict.
- Probably not a substantial claim. 32
- It may state two separate claims as one. 14
- The quote only partly states this claim. 1
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483 · cited 4×
“To expand adoption, future versions of AI-CCM should follow the core principles of the TAM [34,36].”
InfluentialMedical Student Education in the Time of COVID-19
Rose, S. (2020) · JAMA · doi:10.1001/jama.2020.5227 · Cited by 1,740 · cited 2×
“First, peer mentoring is inherently person dependent, and the quality of mentoring varies widely depending on individual mentors’ experience, interpersonal skills, and availability [20].”
Mentoring in Academic Medicine: A Systematic Review
Sambunjak, D., Straus, S. E., & Marušić, A. (2006) · JAMA · doi:10.1001/jama.296.9.1103 · Cited by 1,689 · cited 3×
“The effectiveness of mentoring in medical education is influenced by various factors, including mentees’characteristics, gender disparities, quality of the mentor-mentee relationship, available support systems, and outcome evaluation methods [24].”
Mentoring programs for medical students - a review of the PubMed literature 2000 - 2008
Frei, E., Stamm, M., & Buddeberg-Fischer, B. (2010) · BMC Medical Education · doi:10.1186/1472-6920-10-32 · Cited by 475 · cited 1×
“The structure and content of the prompts were informed by a narrative review of the literature on effective mentoring in medical education, which highlights key mentor roles such as providing academic guidance, emotional support, and fostering professional identity formation [24,40,41].”
Dahlin, M. E., & Runeson, B. (2007) · BMC Medical Education · doi:10.1186/1472-6920-7-6 · Cited by 459
Students' opinions about their preparation for clinical practice
Prince, K. J. A. H., Boshuizen, H. P. A., Van Der Vleuten, C. P. M., & Scherpbier, A. J. J. A. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02207.x · Cited by 301 · cited 3×
“…students encounter challenges related to knowledge acquisition about medicine, diagnostic and procedural skills, effective communication with patients [1,2], development of professional relationships with senior physicians and other health care professionals, and management of stress [1,3,4].”
Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study
Harries, A. J., Lee, C., Jones, L., Rodriguez, R. M., Davis, J. A., Boysen-Osborn, M., Kashima, K. J., Krane, N. K., Rae, G., Kman, N., et al. (2021) · BMC Medical Education · doi:10.1186/s12909-020-02462-1 · Cited by 286
Lee, A. T., Ramasamy, R. K., & Subbarao, A. (2025) · Healthcare · doi:10.3390/healthcare13030250 · Cited by 241 · cited 1×
“This framework enabled a structured interpretation of students’experiences and the acceptance of the AI-based mentoring tool consistent with previous applications of the TAM in medical education and e-learning contexts [35-37].”
Lee, A. T., Ramasamy, R. K., & Subbarao, A. (2025) · doi:10.20944/preprints202501.0074.v1 · Cited by 241
Medical student mentoring programs: current insights
Nimmons, D., Giny, S., & Rosenthal, J. (2019) · Advances in Medical Education and Practice · doi:10.2147/amep.s154974 · Cited by 194
Cited by
Madwe, M. C. (2026) · International Journal of Sustainability in Business and Economics · doi:10.51137/wrp.ijsbe.575 · Cited by 1
Medina, D., Lavín, C., Leviante, R., Alzolay-Sepúlveda, J., Torres, F. M., Araya, T. A., Reyes, R. V., & Reyes, J. (2026) · doi:10.2196/preprints.91444
Implementation of Generative AI in Biomedical Research and Healthcare
Nikolopoulos, A., & Karalis, V. D. (2026) · Applied Biosciences · doi:10.3390/applbiosci5020034
“…the development of AI mentors for clinical clerkships has shown promise with showing that an AI mentor provided clinical scenario advice that was rated as more educationally appropriate than advice from senior student mentors, particularly in managing emotional and psychological stress [92].”
Kellam, H., Cortés, L. E. P., Dange, A., & Sterling, K. (2026) · doi:10.2196/preprints.100513
Metadata
- Authors
- Ebihara, H., Kasai, H., Shimizu, I., Shikino, K., Tajima, H., Kimura, Y., & Ito, S.
- Affiliations
- Chiba University (JP)
- Chiba University Hospital (JP)
- Corresponding author
- Kasai, Hajime
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2025-09-04
- Volume
- 11
- Pages
- e76702
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC12447005
- OpenAlex
- W4414005603
- Cited by
- 10
- References
- 44
- Citations by year
- 2026: 10
- FWCI
- 2.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- clinical rotations, generative AI, large language models, ChatGPT, medical students, student mentors, clinical mentorship, emotional support, psychological stress, communication skills, technology acceptance, clinical learning, expert evaluation of AI responses, medical education technology
- MeSH terms
- Mentoring, Artificial Intelligence, Clinical Clerkship, Clinical Competence, Education, Medical, Undergraduate, Female, Humans, Male, Mentors, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.