The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 14 of the 39 sources its claims cite were checked for support.
AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study
Open accessAbstract
Background: Feedback is essential for medical students’ learning during clinical clerkships; yet, supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models offer a promising approach to supplement human feedback, but how artificial intelligence (AI)–generated and human feedback differ in authentic clinical settings remains unclear, as most comparisons have been conducted in classroom or simulation contexts. Objective: The aim of the study is to examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students’ clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. Methods: This cross-sectional convergent mixed methods study included 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024). Of 164 eligible logs, 3 were excluded because supervisors entered contact messages rather than substantive feedback. AI feedback was generated using GPT-4o. In total, 10 faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric (criteria-based, clear direction, accuracy, prioritization, and supportive tone), followed by open-ended comments and source identification. Quantitative analyses (paired 2-tailed t tests, cumulative link mixed-effects models; α=.05 with Bonferroni correction) were complemented by qualitative thematic analysis and integrated using joint display analysis. Results: AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters; Cohen d =2.84, 95% CI 2.50‐3.19; P <.001). Cumulative link mixed-effects models showed that AI scored higher on criteria-based (odds ratio [OR] 11.81, 95% CI 7.64‐18.27; P <.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P <.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P >.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P =.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P >.99). AI feedback showed greater consistency (variance ratio 3.9:1; Levene F 1,320 =73.20; P <.001). All 20 evaluators correctly identified feedback sources. Qualitative analysis revealed that AI provided structured, text-anchored feedback addressing rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise. Conclusions: This study extends the comparison of AI-generated and supervisor feedback to an authentic clinical clerkship environment, moving beyond classroom and simulation settings examined in prior work. Through integrated mixed methods analysis, a key distinction emerged between text-anchored AI feedback, which systematically addresses written log content in alignment with rubric criteria, and experience-based supervisor feedback, which draws on clinical observation and professional judgment. AI consistently delivered structured feedback addressing gaps that arise when time-pressured supervisors provide brief comments, while supervisors contributed clinically grounded insights that AI cannot replicate. These complementary strengths suggest that AI feedback should supplement rather than replace supervisor feedback, and that hybrid models leveraging each type’s advantages warrant investigation in clinical education.
Overview
CoChat Summary
TL;DR
AI-generated feedback on medical students’ clinical clerkship logs was longer and scored higher than supervisor feedback on criteria-based and clear-direction rubric items, while supervisors provided clinically grounded, experience-based comments. The authors conclude AI feedback should supplement rather than replace supervisor feedback.
Key findings
- AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 97.87, 95% CI 86.4‐109.33 characters; Cohen d=2.84, 95% CI 2.50‐3.19; P<.001).
- In cumulative link mixed-effects models, AI scored higher on criteria-based (OR 11.81, 95% CI 7.64‐18.27; P<.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P<.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P>.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P=.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P>.99).
- AI feedback showed greater rubric-score consistency (variance ratio 3.9:1; Levene F1,320=73.20; P<.001), and all 20 evaluators correctly identified the feedback source (100% detection accuracy).
- Qualitative themes distinguished text-anchored AI feedback, which addressed written log content and rubric criteria, from experience-based supervisor feedback, which drew on clinical observation, professional judgment, and contextual prioritization.
Methods
- Cross-sectional convergent mixed methods study of 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024); 3 of 164 eligible logs were excluded because supervisors entered contact messages rather than substantive feedback.
- AI feedback was generated with GPT-4o using a prompt that explicitly embedded the five Steiss et al rubric criteria; supervisor feedback was naturalistic, with no instructions on content, length, structure, or format.
- Ten faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric, open-ended comments, and source identification; analyses included paired 2-tailed t tests, cumulative link mixed-effects models, thematic analysis, and joint display integration.
Limitations
- The authors state that the study was conducted at a single Japanese institution, entirely in Japanese, with possible language- and culture-specific effects on GPT-4o performance and evaluator perceptions; the small evaluator pool (10 faculty and 10 students) may also limit robustness.
- The authors state that all evaluators correctly identified the feedback source despite blinding, so implicit source recognition may have influenced ratings; they also note the specific GPT-4o model and tailored prompt, lack of prompt sensitivity analysis, rubric emphasis on structural rather than clinical dimensions, one faculty and one student rater per dataset with poor interrater agreement for AI feedback, and no measurement of learner outcomes.
- Evident from the text: the AI prompt explicitly embedded the rubric criteria while supervisors received no rubric instructions, so the study compares rubric-optimized AI feedback with naturalistic supervisor feedback rather than a neutral comparison of inherent feedback quality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
10 Valid · 0 Not valid · 23 Not checked · 133 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“many studies report that the quality of human feedback varies widely depending on the provider and the situation [29-31]”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Introduction > Rationale and Research Question
What the sources say
- Not checked — no full text of the sourcecites [29] Mooney, Predictors of faculty narrative evaluation quality in medical school clerkships, Med Educ source
- Partly supportedcites [30] Coughlin, Differences in faculty feedback for high, expected, and below-expected clinically performing emergency medicine residents, AEM Educ Train sourceExperimental — accuracy not yet measured
The Results (page unavailable) show that written feedback scores differed by residents’ performance level, and the Methods (page unavailable) say scores from the same evaluator tended to be more alike than scores from different evaluators. This supports variation by recipient and provider, but the paper studies one emergency-medicine residency program; it does not establish that human feedback quality varies widely across situations in general. (LLM, conf=0.83)
- Partly supportedcites [31] Van Ostaeyen, A qualitative textual analysis of feedback comments in ePortfolios: quality and alignment with the CanMEDS roles, Perspect Med Educ sourceExperimental — accuracy not yet measured
The Results (Quality of written feedback comments; page unavailable) classify comments as low, moderate, or high quality, showing variation in this study. The Discussion (page unavailable) cites one study associating quality with provider gender, but says variation by feedback phase is only possible and was not assessed. It does not establish that many studies found wide variation depending on both provider and situation. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“All feedback was generated and evaluated in Japanese, and LLM performance has been shown to vary substantially across languages.”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Comparison With the Literature
What the sources say
- Supportedcites [59] Strasser, Performance evaluation of large language models in multilingual medical multiple-choice questions: mixed methods study, JMIR Med Educ source
The Discussion (Principal Findings) states that performance on multilingual medical licensing questions varied considerably by language, as well as by model and question type. The Abstract (Results) also reports a significant effect of question language. The study tested German, French, and Italian, not Japanese; the citing sentence does not attribute a Japanese-specific finding to this source. (LLM, conf=0.93)
- Supportedcites [60] Harigai, Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan, Jpn J Radiol source
The Results section (‘Comparison of performance across languages’) reports significant differences in GPT-4’s scores on the same diagnostic radiology questions in Japanese, English, Chinese, and German. This supports the cited background point that LLM performance can vary substantially by language. (LLM, conf=0.91)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Clinical reasoning, situational awareness, and tacit professional knowledge, which are central to clerkship learning [55], are difficult to articulate in written form [56] and largely inaccessible to AI systems that operate solely on the written log text.”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Comparison With the Literature
What the sources say
- Supportedcites [56] Yardley, Experiential learning: AMEE guide no. 63, Med Teach source
In “Learning in residency” (page unavailable), the guide says people have difficulty describing complex aspects of their work and expertise, and that much workplace knowledge is tacit rather than codified. This supports the cited point that such knowledge is difficult to articulate in writing; the separate claim about AI access is not attributed to this source. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Strasser et al [59] demonstrated that LLM accuracy on identical medical examination questions differed significantly by language (64%‐87%), with English prompts generally yielding higher performance, although advanced models showed comparable results with language-matched prompts in some languages.”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Comparison With the Literature
What the sources say
- Partly supportedcites [59] Strasser, Performance evaluation of large language models in multilingual medical multiple-choice questions: mixed methods study, JMIR Med Educ sourceExperimental — accuracy not yet measured
The Results section reports a significant language difference for Italian versus German questions, but not for French versus German. It reports 64%–87% as the range of mean accuracies across models, not as the range of accuracies by language. The Discussion’s Principal Findings support the sentence’s point that English prompts generally performed better, while advanced models had comparable results with language-matched prompts in German and French. (LLM, conf=0.88)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI feedback was significantly longer and demonstrated greater adherence to rubric-based criteria, particularly in providing criteria-based guidance and clear directions for improvement.”
- Also stated in: § Comparison With the Literature · boosted
“Our study demonstrated that AI feedback was significantly longer than supervisor feedback, which is consistent with some previous findings [202324].”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Principal Findings
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Comparison With the Literature · boosted
73 more claims
Create a free account to see all 166 claims and to verify the 2 references still open.
- Probably not a substantial claim. 52
- It may state two separate claims as one. 56
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Abstract > Results
“mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Results > Quantitative Analysis
“Mean lengths were 382.02 (95% CI 369.29‐394.76) characters for AI and 97.87 (95% CI 86.4‐109.33) characters for supervisors.”
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051 · cited 2×
“Thematic analysis consists of the following steps: (1) familiarize with the data, (2) generate initial codes, (3) search for themes, (4) review themes, (5) define and name themes, and (6) produce the report [48].”
Achieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 1×
“This study was conducted as a cross-sectional mixed methods study using a convergent design from a pragmatic perspective [37,38].”
Enhancing teaching through constructive alignment
Biggs, J. (1996) · Higher Education · doi:10.1007/bf00138871 · Cited by 4,084 · cited 1×
“This is analogous to constructive alignment in instructional design [64], where intended outcomes are aligned with the activities producing them.”
Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D. M., Josselson, R., & Suárez-Orozco, C. (2018) · American Psychologist · doi:10.1037/amp0000151 · Cited by 2,286 · cited 1×
“Methodological integrity was pursued along the 2 dimensions articulated in the APA Journal Article Reporting Standards for Qualitative Research [49]: fidelity to the subject matter and utility in achieving the research goals.”
Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015) · The Annals of Family Medicine · doi:10.1370/afm.1865 · Cited by 1,740 · cited 2×
“Finally, the results of the quantitative and qualitative analyses were integrated using joint display analysis (JDA) [39,40], in which each quantitative finding was placed alongside its corresponding qualitative theme in a structured display, and meta-inferen-ces about how the 2 sources complement…”
Feedback in Clinical Medical Education
Ende, J. (1983) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1983.03340060055026 · Cited by 1,329
Workplace-based assessment as an educational tool: AMEE Guide No. 31
Norcini, J., & Burch, V. (2007) · Medical Teacher · doi:10.1080/01421590701775453 · Cited by 984
Using thematic analysis in qualitative research
Ahmed, S. K., Mohammed, R. A., Nashwan, A. J., Ibrahim, R. H., Abdalla, A. Q., M. Ameen, B. M., & Khdhir, R. M. (2025) · Journal of Medicine, Surgery, and Public Health · doi:10.1016/j.glmedi.2025.100198 · Cited by 769 · cited 1×
“Following the thematic-map guidance of Ahmed et al [50], the thematic map in Figure 5 visualizes the research question, the 5 themes, and their subthemes arranged by feedback provider.”
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · cited 1×
“LLMs are AI models that can handle various tasks by pretraining on massive amounts of text data in advance [14].”
Cited by
Documentation Quality and Educational Value in AI-Assisted Feedback
Endo, A., Kimura, T., & Kataoka, Y. (2026) · JMIR Medical Education · doi:10.2196/105029
Making AI-Generated Feedback Matter: A Large-Scale Study of Feedback Workflows and Student Enactment
Alsaiari, O., Baghaei, N., Lodge, J. M., Gaševi'c, D., Winstone, N., & Khosravi, H. (2026) · arXiv · doi:10.48550/arxiv.2608.11625
“When guided by explicit prompts and assessment criteria, they can generate structured, criteria-based comments with less variability across submissions [45].”
Metadata
- Authors
- Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S.
- Affiliations
- Maastricht University (NL)
- Nagoya University (JP)
- Corresponding author
- Kondo, Takeshi
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2026-06-16
- Volume
- 12
- Pages
- e90064-e90064
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC13271589
- OpenAlex
- W7160848656
- Cited by
- 1
- References
- 53
- Citations by year
- 2026: 1
- FWCI
- 1.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Japan Society for the Promotion of Science (25K06542)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- AI-generated feedback, medical students, large language models, GPT-4o, feedback quality, medical education, clinical supervision, student self-assessment, convergent validity
- MeSH terms
- Generative Artificial Intelligence, Adult, Clinical Clerkship, Clinical Competence, Cross-Sectional Studies, Female, Humans, Japan, Male, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study. https://app.cochat.ai/papers/10.2196/90064
- The paper
- Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S. (2026). AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study. JMIR Medical Education. https://doi.org/10.2196/90064
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.