CoChat

AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study

Open access

The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 14 of the 39 sources its claims cite were checked for support.

Abstract

Background: Feedback is essential for medical students’ learning during clinical clerkships; yet, supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models offer a promising approach to supplement human feedback, but how artificial intelligence (AI)–generated and human feedback differ in authentic clinical settings remains unclear, as most comparisons have been conducted in classroom or simulation contexts. Objective: The aim of the study is to examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students’ clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. Methods: This cross-sectional convergent mixed methods study included 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024). Of 164 eligible logs, 3 were excluded because supervisors entered contact messages rather than substantive feedback. AI feedback was generated using GPT-4o. In total, 10 faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric (criteria-based, clear direction, accuracy, prioritization, and supportive tone), followed by open-ended comments and source identification. Quantitative analyses (paired 2-tailed t tests, cumulative link mixed-effects models; α=.05 with Bonferroni correction) were complemented by qualitative thematic analysis and integrated using joint display analysis. Results: AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters; Cohen d =2.84, 95% CI 2.50‐3.19; P <.001). Cumulative link mixed-effects models showed that AI scored higher on criteria-based (odds ratio [OR] 11.81, 95% CI 7.64‐18.27; P <.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P <.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P >.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P =.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P >.99). AI feedback showed greater consistency (variance ratio 3.9:1; Levene F 1,320 =73.20; P <.001). All 20 evaluators correctly identified feedback sources. Qualitative analysis revealed that AI provided structured, text-anchored feedback addressing rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise. Conclusions: This study extends the comparison of AI-generated and supervisor feedback to an authentic clinical clerkship environment, moving beyond classroom and simulation settings examined in prior work. Through integrated mixed methods analysis, a key distinction emerged between text-anchored AI feedback, which systematically addresses written log content in alignment with rubric criteria, and experience-based supervisor feedback, which draws on clinical observation and professional judgment. AI consistently delivered structured feedback addressing gaps that arise when time-pressured supervisors provide brief comments, while supervisors contributed clinically grounded insights that AI cannot replicate. These complementary strengths suggest that AI feedback should supplement rather than replace supervisor feedback, and that hybrid models leveraging each type’s advantages warrant investigation in clinical education.

Overview

CoChat Summary

TL;DR

AI-generated feedback on medical students’ clinical clerkship logs was longer and scored higher than supervisor feedback on criteria-based and clear-direction rubric items, while supervisors provided clinically grounded, experience-based comments. The authors conclude AI feedback should supplement rather than replace supervisor feedback.

Key findings

  • AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 97.87, 95% CI 86.4‐109.33 characters; Cohen d=2.84, 95% CI 2.50‐3.19; P<.001).
  • In cumulative link mixed-effects models, AI scored higher on criteria-based (OR 11.81, 95% CI 7.64‐18.27; P<.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P<.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P>.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P=.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P>.99).
  • AI feedback showed greater rubric-score consistency (variance ratio 3.9:1; Levene F1,320=73.20; P<.001), and all 20 evaluators correctly identified the feedback source (100% detection accuracy).
  • Qualitative themes distinguished text-anchored AI feedback, which addressed written log content and rubric criteria, from experience-based supervisor feedback, which drew on clinical observation, professional judgment, and contextual prioritization.

Methods

  • Cross-sectional convergent mixed methods study of 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024); 3 of 164 eligible logs were excluded because supervisors entered contact messages rather than substantive feedback.
  • AI feedback was generated with GPT-4o using a prompt that explicitly embedded the five Steiss et al rubric criteria; supervisor feedback was naturalistic, with no instructions on content, length, structure, or format.
  • Ten faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric, open-ended comments, and source identification; analyses included paired 2-tailed t tests, cumulative link mixed-effects models, thematic analysis, and joint display integration.

Limitations

  • The authors state that the study was conducted at a single Japanese institution, entirely in Japanese, with possible language- and culture-specific effects on GPT-4o performance and evaluator perceptions; the small evaluator pool (10 faculty and 10 students) may also limit robustness.
  • The authors state that all evaluators correctly identified the feedback source despite blinding, so implicit source recognition may have influenced ratings; they also note the specific GPT-4o model and tailored prompt, lack of prompt sensitivity analysis, rubric emphasis on structural rather than clinical dimensions, one faculty and one student rater per dataset with poor interrater agreement for AI feedback, and no measurement of learner outcomes.
  • Evident from the text: the AI prompt explicitly embedded the rubric criteria while supervisors received no rubric instructions, so the study compares rubric-optimized AI feedback with naturalistic supervisor feedback rather than a neutral comparison of inherent feedback quality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/90064

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Overview of the feedback preparation and evaluation process in a mixed methods study comparing AI-generated and supervisor feedback on clinical clerkship logs at Nagoya University, Japan. Student logs from the e-portfolio were used to generate AI feedback (GPT-4o) and collect supervisor feedback, which were then tabulated, shuffled, and evaluated by blinded medical students and faculty evaluators. AI: artificial intelligence.
Figure 1. Overview of the feedback preparation and evaluation process in a mixed methods study comparing AI-generated and supervisor feedback on clinical clerkship logs at Nagoya University, Japan. Student logs from the e-portfolio were used to generate AI feedback (GPT-4o) and collect supervisor feedback, which were then tabulated, shuffled, and evaluated by blinded medical students and faculty evaluators. AI: artificial intelligence.CC BY · PubMed Central
Figure 2. Participant flowchart for a mixed methods study comparing AI-generated and supervisor feedback on clinical clerkship logs at Nagoya University, Japan (January 2024-January 2025). The diagram shows participant flow from enrollment (164 logs from 47 medical students across 12 departments) through exclusion (n=3), feedback generation (within-subjects design: each log received both AI and supervisor feedback), blinded evaluation by 10 faculty and 10 student evaluators, to quantitative (3220 observations) and qualitative analyses. AI: artificial intelligence.
Figure 2. Participant flowchart for a mixed methods study comparing AI-generated and supervisor feedback on clinical clerkship logs at Nagoya University, Japan (January 2024-January 2025). The diagram shows participant flow from enrollment (164 logs from 47 medical students across 12 departments) through exclusion (n=3), feedback generation (within-subjects design: each log received both AI and supervisor feedback), blinded evaluation by 10 faculty and 10 student evaluators, to quantitative (3220 observations) and qualitative analyses. AI: artificial intelligence.CC BY · PubMed Central
Figure 3. Distribution of AI-generated and supervisor feedback lengths by character count for 161 clinical clerkship logs from 47 medical students, Nagoya University, Japan, 2024. AI: artificial intelligence.
Figure 3. Distribution of AI-generated and supervisor feedback lengths by character count for 161 clinical clerkship logs from 47 medical students, Nagoya University, Japan, 2024. AI: artificial intelligence.CC BY · PubMed Central
Figure 4. Distribution of per-record mean rubric scores for AI-generated and supervisor feedback on 161 clinical clerkship logs, Nagoya University, Japan, 2024. AI feedback shows smaller dispersion, indicating higher consistency. AI: artificial intelligence.
Figure 4. Distribution of per-record mean rubric scores for AI-generated and supervisor feedback on 161 clinical clerkship logs, Nagoya University, Japan, 2024. AI feedback shows smaller dispersion, indicating higher consistency. AI: artificial intelligence.CC BY · PubMed Central
Figure 5. Thematic map of differences between AI-generated and supervisor feedback on medical students’ clinical clerkship logs. The research question is presented at the top, the 2 feedback modes are shown as contrasting poles, and each of the 5 themes is displayed with its AI and supervisor subthemes arranged on the left and right, respectively. Convergent mixed methods study, 161 clinical clerkship logs, Nagoya University, Japan, 2024. AI: artificial intelligence.
Figure 5. Thematic map of differences between AI-generated and supervisor feedback on medical students’ clinical clerkship logs. The research question is presented at the top, the 2 feedback modes are shown as contrasting poles, and each of the 5 themes is displayed with its AI and supervisor subthemes arranged on the left and right, respectively. Convergent mixed methods study, 161 clinical clerkship logs, Nagoya University, Japan, 2024. AI: artificial intelligence.CC BY · PubMed Central

Verification

Claims the paper makes

10 Valid · 0 Not valid · 23 Not checked · 133 Own results and interpretations

73 more claims

Create a free account to see all 166 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 52
  • It may state two separate claims as one. 56

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 71 references

Cited by

See all 2

Metadata

Authors
Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S.
Affiliations
  • Maastricht University (NL)
  • Nagoya University (JP)
Corresponding author
Kondo, Takeshi
Venue
JMIR Medical Education
Published
2026-06-16
Volume
12
Pages
e90064-e90064
ISSN
2369-3762 (ISSN-L)
OpenAlex
W7160848656
Cited by
1
References
53
Citations by year
2026: 1
FWCI
1.95
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Japan Society for the Promotion of Science (25K06542)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Innovations in Medical Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
AI-generated feedback, medical students, large language models, GPT-4o, feedback quality, medical education, clinical supervision, student self-assessment, convergent validity
MeSH terms
Generative Artificial Intelligence, Adult, Clinical Clerkship, Clinical Competence, Cross-Sectional Studies, Female, Humans, Japan, Male, Students, Medical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.