“But why?”: Explanatory feedback is a reliable marker of high-quality narrative assessment of surgical performance
Open accessAbstract
PURPOSE: This study examines the quality of short narrative comments collected using an online workplace-based assessment (WBA) tool. METHOD: The quality of comments collected by a WBA tool at the UPMC Ophthalmology Residency Training Program was evaluated between July 2017-June 2020. A randomized rating exercise involving 10 meta-raters from 6 institutions was performed to evaluate the value of narratives from deidentified WBAs. The tool captured a single-item entrustment competency question with brief comments. Comments were evaluated using a Quality of Assessment of Learning (QuAL) score (range, 0-5; ≥3 considered high quality) and on whether the assessor provided a feedback rationale. RESULTS: A total of 838 unique WBAs were collected from 15 attending evaluators. Comments were brief (median [interquartile range] length, 11 [7-17] words), yet 514 (61.3%) were rated as high quality (QuAL score ≥3). Of all 838 comments, 98 (11.7%) included a specific reason the evidence or suggestion was provided to the learner. Of these 98 comments, 94 (95.9%) met the high-quality feedback threshold. A higher QuAL score was associated with a higher postgraduate year (PGY) level (estimate [SE], 1.603 [0.428], P < .001 for PGY2 [reference]; 1.003 [0.389], P = .01 for PGY3; 1.079 [0.360], P = .003 for PGY4), suggesting more advanced learners receive higher-quality narrative comments. A correlation was found between a higher entrustment rating and a lower QuAL score (estimate [SE], -0.199 [0.053], P < .001). When the PGY level was controlled for, this association got stronger (estimate [SE], -0.310 [0.057], P < .001). CONCLUSIONS: Analysis of WBA comments from attending physicians evaluated using the QuAL score demonstrated that most comments were high quality despite their brevity. Residents in later training years and with lower entrustment ratings received higher-quality comments. High-quality narrative assessments were longer and addressed rationale as part of the comment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Longer feedback is often associated with higher perceived quality.
“Although longer feedback is often associated with higher perceived quality [26-28]”
From AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study (2026) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Feedback in Clinical Medical Education
Ende, J. (1983) · JAMA · doi:10.1001/jama.1983.03340060055026 · Cited by 1,259 · see citing works
The assessment of professional competence: Developments, research and practical implications
Van Der Vleuten, C. P. M. (1996) · Advances in Health Sciences Education · doi:10.1007/bf00596229 · Cited by 1,247 · see citing works
The role of assessment in competency-based medical education
Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., Frank, J. R., & Collaborators, F. T. I. C. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.500704 · Cited by 956 · see citing works
Mohammed, S., & Dumville, B. C. (2001) · Journal of Organizational Behavior · doi:10.1002/job.86 · Cited by 852 · see citing works
Melle, E. V., Frank, J. R., Holmboe, E. S., Dagnone, D., Stockley, D. B., & Sherbino, J. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002743 · Cited by 456 · see citing works
Implementation of competency‐based medical education: are we addressing the concerns and challenges?
Hawkins, R. E., Welcher, C. M., Holmboe, E. S., Kirk, L. M., Norcini, J. J., Simons, K. B., & Skochelak, S. E. (2015) · Medical Education · doi:10.1111/medu.12831 · Cited by 409 · see citing works
Implementing workplace‐based assessment across the medical specialties in the United Kingdom
Wilkinson, J. R., Crossley, J. G. M., Wragg, A., Mills, P., Cowan, G., & Wade, W. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03010.x · Cited by 272 · see citing works
Competency-based medical education: implications for undergraduate programs
Harris, P., Snell, L. S., Talbot, M. D., Harden, R. M., & Collaborators, F. T. I. C. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.500703 · Cited by 241 · see citing works
Bohnen, J. D., George, B. C., Williams, R. G., Schuller, M. C., DaRosa, D. A., Torbeck, L. J., Mullen, J. T., Meyerson, S. L., Auyang, E. D., Chipman, J. G., et al. (2016) · Journal of surgical education · doi:10.1016/j.jsurg.2016.08.010 · Cited by 226 · see citing works
Crossley, J., & Jolly, B. C. (2011) · Medical Education · doi:10.1111/j.1365-2923.2011.04166.x · Cited by 212 · see citing works
Feedback and assessment for clinical placements: achieving the right balance
Burgess, A. W., & Mellis, C. (2015) · Advances in Medical Education and Practice · doi:10.2147/amep.s77890 · Cited by 153 · see citing works
Massie, J., & Ali, J. M. (2016) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9614-0 · Cited by 147 · see citing works
Hanson, J. L., Rosenberg, A. A., & Lane, J. L. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00668 · Cited by 113 · see citing works
Composite reliability of a workplace-based assessment toolbox for postgraduate medical education
van Loon, J., Overeem, K., Donkers, J. H. H. L. M., van der Vleuten, C. P. M., & Driessen, E. W. (2013) · Advances in Health Sciences Education · doi:10.1007/s10459-013-9450-z · Cited by 104 · see citing works
A short questionnaire to evaluate the effectiveness of tutors in PBL: validity and reliability
Dolmans, D. H. J. M., & Ginns, P. (2005) · Medical Teacher · doi:10.1080/01421590500136477 · Cited by 87 · see citing works
Anderson, H. L. K., Kurtz, J. B., & West, D. C. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004366 · Cited by 69 · see citing works
Swanwick, T., & Chana, N. (2009) · British Journal of Hospital Medicine · doi:10.12968/hmed.2009.70.5.42235 · Cited by 65 · see citing works
Chan, T. M., Sebok‐Syer, S. S., Sampson, C. S., & Monteiro, S. (2020) · Teaching and Learning in Medicine · doi:10.1080/10401334.2019.1708365 · Cited by 65 · see citing works
The reliability of encounter cards to assess the CanMEDS roles
Sherbino, J., Kulasegaram, K. M., Worster, A., & Norman, G. R. (2013) · Advances in Health Sciences Education · doi:10.1007/s10459-012-9440-6 · Cited by 46 · see citing works
Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool
Ross, S., Hamza, D. M., Zulla, R. T., Stasiuk, S., & Nichols, D. N. (2022) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-21-00602.1 · Cited by 27 · see citing works
Trends in ophthalmology resident surgical experience from 2009 to 2015
Chadha, N., Liu, J. Y., Maslin, J. S., & Teng, C. C. (2016) · Clinical ophthalmology · doi:10.2147/opth.s106293 · Cited by 26 · see citing works
On the issue of costs in programmatic assessment
van der Vleuten, C. P. M., & Heeneman, S. (2016) · Perspectives on Medical Education · doi:10.1007/s40037-016-0295-z · Cited by 25 · see citing works
Clement, E., Oswald, A., Ghosh, S., & Hamza, D. M. (2024) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-23-00210.1 · Cited by 24 · see citing works
In‐training evaluations: developing an automated screening tool to measure report quality
Bismil, R., Dudek, N. L., & Wood, T. J. (2014) · Medical Education · doi:10.1111/medu.12490 · Cited by 22 · see citing works
Daily Encounter Cards—Evaluating the Quality of Documented Assessments
Cheung, W. J., Dudek, N. L., Wood, T. J., & Frank, J. R. (2016) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-15-00505.1 · Cited by 21 · see citing works
Woods, R. A., Singh, S., Thoma, B., Patocka, C., Cheung, W. J., Monteiro, S., & Chan, T. M. (2022) · Canadian Medical Education Journal · doi:10.36834/cmej.74860 · Cited by 15 · see citing works
Paley, G., Zhao, H., Addis, V. M., Beal, C. J., Davis, G., Espinoza, G. M., Lee, M. S., Pineles, S. L., Rosenberg, J. B., Swan, R. T., et al. (2024) · Journal of Academic Ophthalmology · doi:10.62199/2475-4757.1015 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S. (2026) · JMIR Medical Education · doi:10.2196/90064 · Cited by 1 · see citing works
Kondo, T., Donkers, J. H. H. L. M., Nishigori, H., Rovers, S. F. E., & Heeneman, S. (2025) · doi:10.2196/preprints.90064
Cloonan, M. R., Tieken, K., Maginot, E. R., Schindler, N. M., Fingeret, A. L., Tanner, T. N., & Merani, S. (2026) · Journal of surgical education · doi:10.1016/j.jsurg.2026.103931
Yu, E., Tian, H., Mohamad, F., Schultz, K., McEwen, L., Gauthier, S. R., Braund, H., Cofie, N., Dalgarno, N., Szulewski, A., et al. (2026) · Canadian Medical Education Journal · doi:10.36834/v1anvj69
- Authors
- Poeppelman, R. S., Cho, J., Nachbor, K., Sekhar, T. C., Pruett, J., Baim, A., Strul, S., Barsam, A., Langworthy, B., Waxman, E. L., & Culican, S. M.
- Affiliations
- University of Minnesota (US)
- Vanderbilt University Medical Center (US)
- University of Michigan (US)
- Rush University Medical Center (US)
- Georgetown University (US)
- Northwestern University (US)
- Duke University Hospital (US)
- University of Pittsburgh (US)
- Corresponding author
- Poeppelman, Rachel Stork
- ORCID
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-01-31
- Volume
- 100
- Issue
- 5
- Pages
- 614-620
- ISSN
- 1040-2446 (ISSN-L)
- 1938-808X
- OpenAlex
- W4407012085
- Cited by
- 4
- References
- 27
- Citations by year
- 2026: 3 · 2025: 1
- FWCI
- 3.64
- Type
- journal article
- Language
- en
- Open access
- Locations
- Academic Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (UL1TR002494, UM1TR004405)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Keywords
- workplace-based assessment, narrative feedback, feedback quality, surgical performance assessment, ophthalmology residency, resident training level
- MeSH terms
- Formative Feedback, Clinical Competence, Education, Medical, Graduate, Educational Measurement, Feedback, Humans, Internship and Residency, Narration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
