AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 14 of the 39 sources its claims cite were checked for support.
Abstract
Background: Feedback is essential for medical students’ learning during clinical clerkships; yet, supervising physicians often struggle to provide meaningful written feedback due to time constraints. Large language models offer a promising approach to supplement human feedback, but how artificial intelligence (AI)–generated and human feedback differ in authentic clinical settings remains unclear, as most comparisons have been conducted in classroom or simulation contexts. Objective: The aim of the study is to examine how AI-generated feedback and supervisor-provided feedback differ when applied to medical students’ clinical clerkship logs, by identifying the distinct characteristics and complementary strengths of each feedback type. Methods: This cross-sectional convergent mixed methods study included 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024). Of 164 eligible logs, 3 were excluded because supervisors entered contact messages rather than substantive feedback. AI feedback was generated using GPT-4o. In total, 10 faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric (criteria-based, clear direction, accuracy, prioritization, and supportive tone), followed by open-ended comments and source identification. Quantitative analyses (paired 2-tailed t tests, cumulative link mixed-effects models; α=.05 with Bonferroni correction) were complemented by qualitative thematic analysis and integrated using joint display analysis. Results: AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters; Cohen d =2.84, 95% CI 2.50‐3.19; P <.001). Cumulative link mixed-effects models showed that AI scored higher on criteria-based (odds ratio [OR] 11.81, 95% CI 7.64‐18.27; P <.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P <.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P >.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P =.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P >.99). AI feedback showed greater consistency (variance ratio 3.9:1; Levene F 1,320 =73.20; P <.001). All 20 evaluators correctly identified feedback sources. Qualitative analysis revealed that AI provided structured, text-anchored feedback addressing rubric criteria, while supervisors offered experience-based feedback grounded in clinical context and professional expertise. Conclusions: This study extends the comparison of AI-generated and supervisor feedback to an authentic clinical clerkship environment, moving beyond classroom and simulation settings examined in prior work. Through integrated mixed methods analysis, a key distinction emerged between text-anchored AI feedback, which systematically addresses written log content in alignment with rubric criteria, and experience-based supervisor feedback, which draws on clinical observation and professional judgment. AI consistently delivered structured feedback addressing gaps that arise when time-pressured supervisors provide brief comments, while supervisors contributed clinically grounded insights that AI cannot replicate. These complementary strengths suggest that AI feedback should supplement rather than replace supervisor feedback, and that hybrid models leveraging each type’s advantages warrant investigation in clinical education.
Overview
CoChat Summary
TL;DR
AI-generated feedback on medical students’ clinical clerkship logs was longer and scored higher than supervisor feedback on criteria-based and clear-direction rubric items, while supervisors provided clinically grounded, experience-based comments. The authors conclude AI feedback should supplement rather than replace supervisor feedback.
Key findings
- AI feedback was significantly longer than supervisor feedback (mean 382.02, SD 81.82 vs mean 97.87, 95% CI 86.4‐109.33 characters; Cohen d=2.84, 95% CI 2.50‐3.19; P<.001).
- In cumulative link mixed-effects models, AI scored higher on criteria-based (OR 11.81, 95% CI 7.64‐18.27; P<.001) and clear direction (OR 6.61, 95% CI 4.35‐10.06; P<.001), with no significant differences on accuracy (OR 1.35, 95% CI 0.91‐2.00; P>.99), prioritization (OR 1.70, 95% CI 1.16‐2.50; P=.10), or supportive tone (OR 1.34, 95% CI 0.87‐2.06; P>.99).
- AI feedback showed greater rubric-score consistency (variance ratio 3.9:1; Levene F1,320=73.20; P<.001), and all 20 evaluators correctly identified the feedback source (100% detection accuracy).
- Qualitative themes distinguished text-anchored AI feedback, which addressed written log content and rubric criteria, from experience-based supervisor feedback, which drew on clinical observation, professional judgment, and contextual prioritization.
Methods
- Cross-sectional convergent mixed methods study of 161 weekly clinical clerkship logs from 47 fifth- and sixth-year medical students across 12 clinical departments at Nagoya University, Japan (January-May 2024); 3 of 164 eligible logs were excluded because supervisors entered contact messages rather than substantive feedback.
- AI feedback was generated with GPT-4o using a prompt that explicitly embedded the five Steiss et al rubric criteria; supervisor feedback was naturalistic, with no instructions on content, length, structure, or format.
- Ten faculty physicians and 10 medical students evaluated both feedback types in blinded, randomized order using a validated 5-category rubric, open-ended comments, and source identification; analyses included paired 2-tailed t tests, cumulative link mixed-effects models, thematic analysis, and joint display integration.
Limitations
- The authors state that the study was conducted at a single Japanese institution, entirely in Japanese, with possible language- and culture-specific effects on GPT-4o performance and evaluator perceptions; the small evaluator pool (10 faculty and 10 students) may also limit robustness.
- The authors state that all evaluators correctly identified the feedback source despite blinding, so implicit source recognition may have influenced ratings; they also note the specific GPT-4o model and tailored prompt, lack of prompt sensitivity analysis, rubric emphasis on structural rather than clinical dimensions, one faculty and one student rater per dataset with poor interrater agreement for AI feedback, and no measurement of learner outcomes.
- Evident from the text: the AI prompt explicitly embedded the rubric criteria while supervisors received no rubric instructions, so the study compares rubric-optimized AI feedback with naturalistic supervisor feedback rather than a neutral comparison of inherent feedback quality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
10 Valid · 0 Not valid · 23 Not checked · 133 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“many studies report that the quality of human feedback varies widely depending on the provider and the situation [29-31]”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Introduction > Rationale and Research Question
What the sources say
- Not checked — no full text of the sourcecites [29] Mooney, Predictors of faculty narrative evaluation quality in medical school clerkships, Med Educ source
- Partly supportedcites [30] Coughlin, Differences in faculty feedback for high, expected, and below-expected clinically performing emergency medicine residents, AEM Educ Train sourceExperimental — accuracy not yet measured
The Results (page unavailable) show that written feedback scores differed by residents’ performance level, and the Methods (page unavailable) say scores from the same evaluator tended to be more alike than scores from different evaluators. This supports variation by recipient and provider, but the paper studies one emergency-medicine residency program; it does not establish that human feedback quality varies widely across situations in general. (LLM, conf=0.83)
- Partly supportedcites [31] Van Ostaeyen, A qualitative textual analysis of feedback comments in ePortfolios: quality and alignment with the CanMEDS roles, Perspect Med Educ sourceExperimental — accuracy not yet measured
The Results (Quality of written feedback comments; page unavailable) classify comments as low, moderate, or high quality, showing variation in this study. The Discussion (page unavailable) cites one study associating quality with provider gender, but says variation by feedback phase is only possible and was not assessed. It does not establish that many studies found wide variation depending on both provider and situation. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“All feedback was generated and evaluated in Japanese, and LLM performance has been shown to vary substantially across languages.”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Comparison With the Literature
What the sources say
- Supportedcites [59] Strasser, Performance evaluation of large language models in multilingual medical multiple-choice questions: mixed methods study, JMIR Med Educ source
The Discussion (Principal Findings) states that performance on multilingual medical licensing questions varied considerably by language, as well as by model and question type. The Abstract (Results) also reports a significant effect of question language. The study tested German, French, and Italian, not Japanese; the citing sentence does not attribute a Japanese-specific finding to this source. (LLM, conf=0.93)
- Supportedcites [60] Harigai, Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan, Jpn J Radiol source
The Results section (‘Comparison of performance across languages’) reports significant differences in GPT-4’s scores on the same diagnostic radiology questions in Japanese, English, Chinese, and German. This supports the cited background point that LLM performance can vary substantially by language. (LLM, conf=0.91)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Clinical reasoning, situational awareness, and tacit professional knowledge, which are central to clerkship learning [55], are difficult to articulate in written form [56] and largely inaccessible to AI systems that operate solely on the written log text.”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Comparison With the Literature
What the sources say
- Supportedcites [56] Yardley, Experiential learning: AMEE guide no. 63, Med Teach source
In “Learning in residency” (page unavailable), the guide says people have difficulty describing complex aspects of their work and expertise, and that much workplace knowledge is tacit rather than codified. This supports the cited point that such knowledge is difficult to articulate in writing; the separate claim about AI access is not attributed to this source. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Strasser et al [59] demonstrated that LLM accuracy on identical medical examination questions differed significantly by language (64%‐87%), with English prompts generally yielding higher performance, although advanced models showed comparable results with language-matched prompts in some languages.”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Comparison With the Literature
What the sources say
- Partly supportedcites [59] Strasser, Performance evaluation of large language models in multilingual medical multiple-choice questions: mixed methods study, JMIR Med Educ sourceExperimental — accuracy not yet measured
The Results section reports a significant language difference for Italian versus German questions, but not for French versus German. It reports 64%–87% as the range of mean accuracies across models, not as the range of accuracies by language. The Discussion’s Principal Findings support the sentence’s point that English prompts generally performed better, while advanced models had comparable results with language-matched prompts in German and French. (LLM, conf=0.88)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI feedback was significantly longer and demonstrated greater adherence to rubric-based criteria, particularly in providing criteria-based guidance and clear directions for improvement.”
- Also stated in: § Comparison With the Literature · boosted
“Our study demonstrated that AI feedback was significantly longer than supervisor feedback, which is consistent with some previous findings [202324].”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Discussion > Principal Findings
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Comparison With the Literature · boosted
73 more claims
Create a free account to see all 166 claims and to verify the 2 references still open.
- Probably not a substantial claim. 52
- It may state two separate claims as one. 56
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Abstract > Results
“mean 382.02, SD 81.82 vs mean 98.87, SD 73.66 characters”
§ AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study > Results > Quantitative Analysis
“Mean lengths were 382.02 (95% CI 369.29‐394.76) characters for AI and 97.87 (95% CI 86.4‐109.33) characters for supervisors.”
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051 · cited 2×
“Thematic analysis consists of the following steps: (1) familiarize with the data, (2) generate initial codes, (3) search for themes, (4) review themes, (5) define and name themes, and (6) produce the report [48].”
Achieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 1×
“This study was conducted as a cross-sectional mixed methods study using a convergent design from a pragmatic perspective [37,38].”
Enhancing teaching through constructive alignment
Biggs, J. (1996) · Higher Education · doi:10.1007/bf00138871 · Cited by 4,084 · cited 1×
“This is analogous to constructive alignment in instructional design [64], where intended outcomes are aligned with the activities producing them.”
Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D. M., Josselson, R., & Suárez-Orozco, C. (2018) · American Psychologist · doi:10.1037/amp0000151 · Cited by 2,286 · cited 1×
“Methodological integrity was pursued along the 2 dimensions articulated in the APA Journal Article Reporting Standards for Qualitative Research [49]: fidelity to the subject matter and utility in achieving the research goals.”
Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015) · The Annals of Family Medicine · doi:10.1370/afm.1865 · Cited by 1,740 · cited 2×
“Finally, the results of the quantitative and qualitative analyses were integrated using joint display analysis (JDA) [39,40], in which each quantitative finding was placed alongside its corresponding qualitative theme in a structured display, and meta-inferen-ces about how the 2 sources complement…”
Feedback in Clinical Medical Education
Ende, J. (1983) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1983.03340060055026 · Cited by 1,329
Workplace-based assessment as an educational tool: AMEE Guide No. 31
Norcini, J., & Burch, V. (2007) · Medical Teacher · doi:10.1080/01421590701775453 · Cited by 984
Using thematic analysis in qualitative research
Ahmed, S. K., Mohammed, R. A., Nashwan, A. J., Ibrahim, R. H., Abdalla, A. Q., M. Ameen, B. M., & Khdhir, R. M. (2025) · Journal of Medicine, Surgery, and Public Health · doi:10.1016/j.glmedi.2025.100198 · Cited by 769 · cited 1×
“Following the thematic-map guidance of Ahmed et al [50], the thematic map in Figure 5 visualizes the research question, the 5 themes, and their subthemes arranged by feedback provider.”
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · cited 1×
“LLMs are AI models that can handle various tasks by pretraining on massive amounts of text data in advance [14].”
Experiential learning: AMEE Guide No. 63
Yardley, S., Teunissen, P. W., & Dornan, T. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.650741 · Cited by 639 · cited 1×
“Clinical reasoning, situational awareness, and tacit professional knowledge, which are central to clerkship learning [55], are difficult to articulate in written form [56] and largely inaccessible to AI systems that operate solely on the written log text.”
ChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628 · cited 1×
“In contrast, LLMs can adapt to diverse contexts with minimal modifications, making them promising tools for the generation of feedback [16,17].”
The rise of ChatGPT : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596 · cited 1×
“In contrast, LLMs can adapt to diverse contexts with minimal modifications, making them promising tools for the generation of feedback [16,17].”
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539
Comparing the quality of human and ChatGPT feedback of students’ writing
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101894 · Cited by 509 · cited 10×
“Although Steiss et al [18] dealt with different types of feedback, their study also compared human-generated descriptions with feedback produced by ChatGPT in response to written tasks.”
InfluentialCan Large Language Models Provide Feedback to Students? A Case Study on ChatGPT
Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., Gašević, D., & Chen, G. (2023) · 2023 IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt58122.2023.00100 · Cited by 395
Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT
Dai, W., Lin, J., Jin, F., Li, T., Tsai, Y.-S., Gasevic, D., & Chen, G. (2023) · doi:10.35542/osf.io/hcgzj · Cited by 395
A systematic review of large language models and their implications in medical education
Lucas, H. C., Upperman, J. S., & Robinson, J. R. (2024) · Medical Education · doi:10.1111/medu.15402 · Cited by 287 · cited 1×
“In particular, since the advent of large language models (LLMs), the ease of using AI has led to an enthusiastic exploration of its application in feedback [11-13].”
Feedback in the clinical setting
Burgess, A., van Diggele, C., Roberts, C., & Mellis, C. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02280-5 · Cited by 262 · cited 1×
“Feedback serves as a critical educational instrument for medical students and residents to learn from their experiences in the clinical environment [1-4].”
How and what do medical students learn in clerkships? Experience based learning (ExBL)
Dornan, T., Tan, N., Boshuizen, H., Gick, R., Isba, R., Mann, K., Scherpbier, A., Spencer, J., & Timmins, E. (2014) · Advances in Health Sciences Education · doi:10.1007/s10459-014-9501-0 · Cited by 198 · cited 1×
“Clinical reasoning, situational awareness, and tacit professional knowledge, which are central to clerkship learning [55], are difficult to articulate in written form [56] and largely inaccessible to AI systems that operate solely on the written log text.”
Joint displays for mixed methods research in psychology
McCrudden, M. T., Marchand, G., & Schutz, P. A. (2021) · Methods in Psychology · doi:10.1016/j.metip.2021.100067 · Cited by 150 · cited 2×
“Finally, the results of the quantitative and qualitative analyses were integrated using joint display analysis (JDA) [39,40], in which each quantitative finding was placed alongside its corresponding qualitative theme in a structured display, and meta-inferen-ces about how the 2 sources complement…”
Yavuz, F., Çelik, Ö., & Yavaş Çelik, G. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13494 · Cited by 121 · cited 1×
“…for AI feedback is therefore best read not as an inherent property of GPT-4o but as an outcome of this prompt configuration, consistent with experimental evidence that detailed prompts combined with low temperature settings yield near-perfect interreplicate consistency in LLM rubric grading [66].”
Branfield Day, L., Miles, A., Ginsburg, S., & Melvin, L. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003315 · Cited by 120 · cited 3×
“Furthermore, prior comparative studies often allowed humans ample preparation time for crafting feedback [13,18,19], whereas in real clinical settings, supervisors are time-pressured and may produce brief or nonspecific comments [9,10].”
Assessing student perceptions and use of instructor versus AI ‐generated feedback
Er, E., Akçapınar, G., Bayazıt, A., Noroozi, O., & Banihashem, S. K. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13558 · Cited by 114
Maudsley, G. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.542523 · Cited by 99 · cited 1×
“This study was conducted as a cross-sectional mixed methods study using a convergent design from a pragmatic perspective [37,38].”
Çiçek, F. E., Ülker, M., Özer, M., & Kıyak, Y. S. (2024) · Postgraduate Medical Journal · doi:10.1093/postmj/qgae170 · Cited by 95 · cited 3×
“Previous research has noted that AI feedback tends to provide comprehensive feedback closely tied to the target text [24,25], while human experts in specialized fields provide context-informed feedback that is perceived to be of higher quality [22,54].”
Pedagogical framework for hybrid intelligent feedback
Banihashem, S. K., Noroozi, O., Khosravi, H., Schunn, C. D., & Drachsler, H. (2025) · Innovations in Education and Teaching International · doi:10.1080/14703297.2025.2499174 · Cited by 89 · cited 1×
“Various hybrid approaches merit exploration, including AI-drafted feedback refined by educators, role-based distribution where AI and supervisors focus on different aspects of student work, or AI enhancement of educator-authored feedback [34,63].”
The grass is not always greener: Teacher vs. GPT-assisted written corrective feedback
Lin, S., & Crosthwaite, P. (2024) · System · doi:10.1016/j.system.2024.103529 · Cited by 86 · cited 2×
“Regarding the consistency of feedback quality, many studies report that the quality of human feedback varies widely depending on the provider and the situation [29-31], whereas some studies find that, compared with AI, human feedback is consistently higher in quality [25, 32].”
AI or Human? Evaluating Student Feedback Perceptions in Higher Education
Nazaretsky, T., Mejia-Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72315-5_20 · Cited by 76 · cited 2×
“Findings are also inconsistent about whether recipients can tell if the feedback provider is AI or human, ranging from studies reporting that people could distinguish it reasonably well [33] to studies reporting that they could hardly distinguish it [34,35].”
Usher, M. (2025) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2025.2487495 · Cited by 75 · cited 4×
“Our study demonstrated that AI feedback was significantly longer than supervisor feedback, which is consistent with some previous findings [20,23,24].”
InfluentialBushido and Medical Professionalism in Japan
Nishigori, H., Harrison, R., Busari, J., & Dornan, T. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000176 · Cited by 74
Self-citationSuchikova, Y., Tsybuliak, N., Teixeira da Silva, J. A., & Nazarovets, S. (2026) · Accountability in Research · doi:10.1080/08989621.2025.2544331 · Cited by 70
Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance
van de Ridder, J. M. M., Berk, F. C. J., Stokking, K. M., & ten Cate, O. T. J. (2015) · Medical Teacher · doi:10.3109/0142159x.2014.970617 · Cited by 69 · cited 2×
“Prior research has demonstrated that knowledge or perception of the feedback provider can influence how feedback is received and evaluated [45,46], and a recent controlled-content experiment further showed that source attribution altered learner engagement by a large margin (Cohen d =0.88 ‐ 1.56)…”
InfluentialThe TRAPD approach as a method for questionnaire translation
Walde, P., & Völlm, B. A. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1199989 · Cited by 69 · cited 1×
“The translation process was conducted based on the translation, review, adjudication, pretesting, and documentation model [44].”
Canavan, C., Holtman, M. C., Richmond, M., & Katsufrakis, P. J. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181ed4cdb · Cited by 62 · cited 2×
“Furthermore, prior comparative studies often allowed humans ample preparation time for crafting feedback [13,18,19], whereas in real clinical settings, supervisors are time-pressured and may produce brief or nonspecific comments [9,10].”
Jacobsen, L. J., & Weber, K. E. (2023) · doi:10.31219/osf.io/cr257 · Cited by 57
van der Vleuten, C. P. M., Lindemann, I., & Schmidt, L. (2018) · The Medical Journal of Australia · doi:10.5694/mja17.00926 · Cited by 56
Nishigori, H. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2372108 · Cited by 55
Self-citationNeural Machine Translation of Logographic Language Using Sub-character Level Information
Zhang, L., & Komachi, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6303 · Cited by 53 · cited 1×
“Although it varies by domain, Japanese is reported to have approximately 1.7 characters per word [47].”
InfluentialOrtega-Ochoa, E., Quiroga Pérez, J., Arguedas, M., Daradoumis, T., & Marquès Puig, J. M. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101101 · Cited by 50 · cited 3×
“Furthermore, prior comparative studies often allowed humans ample preparation time for crafting feedback [13,18,19], whereas in real clinical settings, supervisors are time-pressured and may produce brief or nonspecific comments [9,10].”
Twelve tips for developing feedback literacy in health professions learners
Tripodi, N., Feehan, J., Wospil, R., & Vaughan, B. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1839035 · Cited by 48 · cited 2×
“Prior research has demonstrated that knowledge or perception of the feedback provider can influence how feedback is received and evaluated [45,46], and a recent controlled-content experiment further showed that source attribution altered learner engagement by a large margin (Cohen d =0.88 ‐ 1.56)…”
InfluentialFeedback on Feedback: Student’s Perceptions for Feedback from Teachers and Few-Shot LLMs
Rüdian, S., Podelo, J., Kužílek, J., & Pinkwart, N. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706479 · Cited by 46
Surgeons vs ChatGPT: Assessment and Feedback Performance Based on Real Surgical Scenarios
Jarry, C., Vela, J., Vivas, G. E., Escobar, E. A. G., Villagrán, I., Tirado, P. A., & Cohen, J. V. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.03.012 · Cited by 37
Oral versus written feedback in medical clinic
Elnicki, D. M., Layne, R. D., Ogden, P. E., & Morris, D. K. (1998) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1998.00049.x · Cited by 34 · cited 1×
“Such feedback can be provided orally or as written feedback [5].”
Ifrah, S. (2024) · Apress eBooks · doi:10.1007/979-8-8688-0599-8_6 · Cited by 29
Dental students’ learning experience: artificial intelligence vs human feedback on assignments
Jayawardena, C. K., Gunathilake, Y., & Ihalagedara, D. (2025) · International Dental Journal · doi:10.1016/j.identj.2024.12.022 · Cited by 26 · cited 8×
“However, even studies demonstrating that AI can generate high-quality feedback argue that AI feedback should complement rather than replace human feedback, citing student preferences and the responsibility that human instructors should maintain [20-22].”
InfluentialKhojasteh, L., Kafipour, R., Pakdel, F., & Mukundan, J. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06753-3 · Cited by 24 · cited 1×
“However, such direct provision of AI feedback requires careful consideration of potential risks, including student overreliance on AI recommendations [62] or conversely, dismissal of AI-generated insights [20].”
Strasser, L. M., Anschuetz, W., Dennstädt, F., & Hastings, J. (2026) · JMIR Medical Education · doi:10.2196/81399 · Cited by 20 · cited 1×
“Strasser et al [59] demonstrated that LLM accuracy on identical medical examination questions differed significantly by language (64% ‐ 87%), with English prompts generally yielding higher performance, although advanced models showed comparable results with language-matched prompts in some…”
Ali, M., Harbieh, I., & Haider, K. H. (2026) · Medical Teacher · doi:10.1080/0142159x.2025.2519639 · Cited by 19 · cited 6×
“Most prior comparisons of AI and human feedback have been conducted in classroom or simulation settings [22,24,25, 54], where the knowledge base is relatively well-defined and the feedback task is circumscribed.”
InfluentialHarvey, P., Radomski, N., & O'Connor, D. J. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.827332 · Cited by 18 · cited 1×
“In the clinical environment, where multiple supervising physicians often take turns in providing guidance, obtaining and documenting written feedback is important for ensuring the continuity of learning and assessment [6-8].”
Leclair, R., Ho, J. S. S., Braund, H., Kouzmina, E., Bruzzese, S., Awad, S., Mann, S., & Zevin, B. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231175734 · Cited by 18 · cited 1×
“Prior research has pointed out that busy physicians tend to provide short feedback, often lacking important elements such as specific examples and future action guidelines [28,52].”
AI or Human? Evaluating Student Feedback Perceptions in Higher EducationPreprint
Nazaretsky, T., Mejia‐Domenzain, P., Swamy, V., et al. (2024) · doi:10.31219/osf.io/6zm83 · Cited by 18
Ostaeyen, S. V., Langhe, L. D., Clercq, O. D., Embo, M., Schellens, T., & Valcke, M. (2023) · Perspectives on Medical Education · doi:10.5334/pme.1056 · Cited by 15
Harigai, A., Toyama, Y., Nagano, M., Abe, M., Kawabata, M., Li, L., Yamamura, J., & Takase, K. (2025) · Japanese Journal of Radiology · doi:10.1007/s11604-024-01673-6 · Cited by 15
Lu, K. J. Q., Meaney, C., Guo, E., & Leung, F.-H. (2023) · JMIR Medical Education · doi:10.2196/41953 · Cited by 11
Renting, N., Jaarsma, D. A. D. C., Borleffs, J. C. C., Slaets, J. P. J., Cohen‐Schotanus, J., & Gans, R. O. B. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076946 · Cited by 11 · cited 1×
“Previous research has identified multiple factors that influence feedback quality variability, including feedback length [29], the subject of feedback [30], and whether supervisors have received feedback training [53].”
Predictors of faculty narrative evaluation quality in medical school clerkships
Mooney, C. J., Pascoe, J. M., Blatt, A. E., Lang, V. J., Kelly, M. S., Braun, M. K., Burch, J. E., & Stone, R. T. (2022) · Medical Education · doi:10.1111/medu.14911 · Cited by 10 · cited 3×
“Regarding the consistency of feedback quality, many studies report that the quality of human feedback varies widely depending on the provider and the situation [29-31], whereas some studies find that, compared with AI, human feedback is consistently higher in quality [25, 32].”
Mendiola, M. S., Anakin, M., Findyartini, A., Levine, R., Silva, A. S. D., & Vakani, F. S. (2025) · MedEdPublish · doi:10.12688/mep.21403.1 · Cited by 10 · cited 1×
“Recent applied guidance in health professions education has similarly recommended grounding generative AI prompts in institutional pedagogical and assessment frameworks [65].”
How Do Learners Receive Feedback on Note Writing? A Scoping Review
Hansen, A., Klute, R. M., Yadav, M., Bansal, S. K., & Bond, W. F. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005653 · Cited by 8 · cited 1×
“Feedback serves as a critical educational instrument for medical students and residents to learn from their experiences in the clinical environment [1-4].”
Evaluating feedback comments in entrustable professional activities: a cross-sectional study
Andreou, V., Peters, S., Eggermont, J., & Schoenmakers, B. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241275810 · Cited by 8 · cited 1×
“Although longer feedback is often associated with higher perceived quality [26-28], the extent to which differences in length between AI and human feedback translate into quality advantages remains unclear [18,20].”
Differentiating between GPT-generated and human-written feedback for radiology residents
Zhou, Z., Rizwan, A. P., Rogoza, N., Chung, A. D., & Kwan, B. Y. M. (2025) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2025.02.002 · Cited by 8 · cited 2×
“Previous research has noted that AI feedback tends to provide comprehensive feedback closely tied to the target text [24,25], while human experts in specialized fields provide context-informed feedback that is perceived to be of higher quality [22,54].”
MD Student Perceptions of ChatGPT for Reflective Writing Feedback in Undergraduate Medical Education
Haider, N., Morjaria, L., Sheth, U., Al-Jabouri, N., & Sibbald, M. (2025) · International Medical Education · doi:10.3390/ime4030027 · Cited by 8
Coughlin, R. F., Tsyrulnik, A., Gottlieb, M., Bod, J., Barnicle, R., Dziura, J., Della‐Giustina, D., & Goldflam, K. (2022) · AEM Education and Training · doi:10.1002/aet2.10788 · Cited by 7 · cited 1×
“Previous research has identified multiple factors that influence feedback quality variability, including feedback length [29], the subject of feedback [30], and whether supervisors have received feedback training [53].”
Curtis, R., Moon, C. C., Hanmore, T., Hopman, W. M., & Baxter, S. (2024) · Canadian Medical Education Journal · doi:10.36834/cmej.76671 · Cited by 7 · cited 1×
“Some reports indicate that feedback length does not contribute to quality improvement beyond a certain threshold [51].”
Evaluating Trust in AI, Human, and Co-produced Feedback Among Undergraduate Students
Zhang, A., Gao, Y., Suraworachet, W., Nazaretsky, T., & Cukurova, M. (2025) · arXiv · doi:10.48550/arxiv.2504.10961 · Cited by 7
Van Ostaeyen, S., Embo, M., Rotsaert, T., De Clercq, O., Schellens, T., & Valcke, M. (2023) · Perspectives on Medical Education · doi:10.5334/pme.1050 · Cited by 6
Poeppelman, R. S., Cho, J., Nachbor, K., Sekhar, T. C., Pruett, J., Baim, A., Strul, S., Barsam, A., Langworthy, B., Waxman, E. L., et al. (2025) · Academic Medicine · doi:10.1097/acm.0000000000005985 · Cited by 4
Greviana, N., Pinasthika, A., & Soemantri, D. (2024) · Medical Science Educator · doi:10.1007/s40670-024-02244-0 · Cited by 2
Same Feedback, Different Source: How AI vs. Human Feedback Shapes Learner Engagement
Morris, C., & Maes, P. (2026) · arXiv · doi:10.48550/arxiv.2602.11311 · Cited by 1
Maher, J., Byszewski, A. M., & Lochnan, H. A. (2026) · Perspectives on Medical Education · doi:10.5334/pme.2265
ncukondo-research/ai-vs-human-feedback-analysis-open-data: v1.0.0: Initial Release
Kondo, T. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.18868578
Self-citation
Cited by
Documentation Quality and Educational Value in AI-Assisted Feedback
Endo, A., Kimura, T., & Kataoka, Y. (2026) · JMIR Medical Education · doi:10.2196/105029
Making AI-Generated Feedback Matter: A Large-Scale Study of Feedback Workflows and Student Enactment
Alsaiari, O., Baghaei, N., Lodge, J. M., Gaševi'c, D., Winstone, N., & Khosravi, H. (2026) · arXiv · doi:10.48550/arxiv.2608.11625
“When guided by explicit prompts and assessment criteria, they can generate structured, criteria-based comments with less variability across submissions [45].”
Metadata
- Authors
- Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S.
- Affiliations
- Maastricht University (NL)
- Nagoya University (JP)
- Corresponding author
- Kondo, Takeshi
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2026-06-16
- Volume
- 12
- Pages
- e90064-e90064
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC13271589
- OpenAlex
- W7160848656
- Cited by
- 1
- References
- 53
- Citations by year
- 2026: 1
- FWCI
- 1.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Japan Society for the Promotion of Science (25K06542)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- AI-generated feedback, medical students, large language models, GPT-4o, feedback quality, medical education, clinical supervision, student self-assessment, convergent validity
- MeSH terms
- Generative Artificial Intelligence, Adult, Clinical Clerkship, Clinical Competence, Cross-Sectional Studies, Female, Humans, Japan, Male, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.