Use the right words: evaluating the effect of word choice and word count on quality of narrative feedback in ophthalmology competency-based medical education assessments
Open accessAbstract
Background: The purpose of this study was to investigate the effect of word choice on the quality of narrative feedback in ophthalmology resident trainee assessments following the introduction of competency-based medical education at Queen’s University. Methods: Assessment data from July 2017-December 2020 were retrieved from ElentraTM (Integrated Teaching and Learning Platform) and anonymized. Written feedback was assigned a Quality of Assessment for Learning (QuAL) score out of five based on this previously validated rubric. The correlation between QuAL score and specific coaching words was determined using a Spearman’s Rho analysis. Independent samples t-tests were used to compare the QuAL score when a specific word was used, and when it was absent. Results: A total of 1997 individual assessments were used in this analysis. The number of times the identified coaching words were used within a comment was significantly and positively associated with the total QuAL score, with the exception of “next time” (rho=0.039, p=0.082), “read” (rho = 0.036, p = 0.112), “read more” (rho = -0.025, p = 0.256) and “review” (rho = -0.017, p = 0.440). The strongest correlations were for “continue” (rho = 0.182, p < 0.001), “try(ing)” (rho = 0.113, p < 0.001) and “next step” (rho = 0.103, p < 0.001). The mean value of the QuAL score increased when coaching words were used vs. not used with the largest mean difference of 1.44 (p < 0.001) for “reflect”. A clear positive relationship was demonstrated between word count and QuAL score (rho = .556, p < 0.001). Conclusions: The use of certain coaching words in written comments may improve the quality of feedback.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The role of assessment in competency-based medical education
Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., Frank, J. R., & Collaborators, F. T. I. C. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.500704 · Cited by 956 · see citing works
Ginsburg, S., van der Vleuten, C. P. M., & Eva, K. W. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001669 · Cited by 125 · see citing works
Assessing the quality of supervisors’ completed clinical evaluation reports
Dudek, N. L., Marks, M. B., Wood, T. J., & Lee, A. C. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03105.x · Cited by 71 · see citing works
Chan, T. M., Sebok‐Syer, S. S., Sampson, C. S., & Monteiro, S. (2020) · Teaching and Learning in Medicine · doi:10.1080/10401334.2019.1708365 · Cited by 65 · see citing works
Tomiak, A. T., Braund, H., Egan, R. G., Dalgarno, N., Emack, J., Reid, M.-A., & Hammad, N. (2019) · Journal of Cancer Education · doi:10.1007/s13187-018-1456-z · Cited by 53 · see citing works
Stodel, E. J., Wyand, A., Crooks, S., Moffett, S., Chiu, M., & Hudson, C. C. C. (2015) · Anesthesiology Research and Practice · doi:10.1155/2015/713038 · Cited by 43 · see citing works
Competency-Based Medical Education: Developing a Framework for Obstetrics and Gynaecology
Caccia, N. C., Nakajima, A., Scheele, F., & Kent, N. (2015) · Journal of Obstetrics and Gynaecology Canada · doi:10.1016/s1701-2163(16)30076-7 · Cited by 42 · see citing works
Marcotte, L., Egan, R. G., Soleas, E., Dalgarno, N., Norris, M., & Smith, C. A. (2019) · Canadian Medical Education Journal · doi:10.36834/cmej.57323 · Cited by 42 · see citing works
Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool
Ross, S., Hamza, D. M., Zulla, R. T., Stasiuk, S., & Nichols, D. N. (2022) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-21-00602.1 · Cited by 27 · see citing works
Introducing a competency based Fellowship programme for psychiatry in Australia and New Zealand
Jurd, S. M., de Beer, W., Aimer, M., Fletcher, S., Halley, E., Schapper, C., & Orkin, M. (2015) · Australasian Psychiatry · doi:10.1177/1039856215600898 · Cited by 26 · see citing works
A systems approach for institutional CBME adoption at Queen’s University
Stockley, D. B., Egan, R. G., van Wylick, R., et al. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1767768 · Cited by 25 · see citing works
Marcotte, L., Egan, R. G., Soleas, E., Dalgarno, N., Norris, M., & Smith, C. (2019) · PubMed Central · Cited by 23 · see citing works
In‐training evaluations: developing an automated screening tool to measure report quality
Bismil, R., Dudek, N. L., & Wood, T. J. (2014) · Medical Education · doi:10.1111/medu.12490 · Cited by 22 · see citing works
Daily Encounter Cards—Evaluating the Quality of Documented Assessments
Cheung, W. J., Dudek, N. L., Wood, T. J., & Frank, J. R. (2016) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-15-00505.1 · Cited by 21 · see citing works
Tekian, A. S., Borhani, M., Tilton, S. T., et al. (2018) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2018.09.031 · Cited by 16 · see citing works
The effect of gender dyads on the quality of narrative assessments of general surgery trainees
Roshan, A., Farooq, A., Acai, A., Wagner, N., Sonnadara, R., Scott, T. M., & Karimuddin, A. A. (2021) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2021.12.001 · Cited by 14 · see citing works
Beyond “Read More”: An Intervention to Improve Faculty Written Feedback to Learners
Zelenski, A. B., Tischendorf, J. S., Kessler, M. A., Saunders, S., MacDonald, M. M., Vogelman, B., & Zakowski, L. J. (2019) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-19-00058.1 · Cited by 13 · see citing works
Competency assessment form to improve feedback
Hauer, K. E., Nishimura, H. M., Dubon, D., Teherani, A., & Boscardin, C. (2017) · The Clinical Teacher · doi:10.1111/tct.12726 · Cited by 11 · see citing works
Involving ophthalmology departmental stakeholders in developing workplace-based assessment tools
Braund, H., Dalgarno, N., McEwen, L. A., Egan, R. G., Reid, M.-A., & Baxter, S. A. (2019) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2019.01.013 · Cited by 11 · see citing works
Self-citation‘Next steps are…’: An exploration of coaching and feedback language in EPA assessment comments
Day, L. B., Rassos, J., Billick, M. J., & Ginsburg, S. (2022) · Medical Teacher · doi:10.1080/0142159x.2022.2098098 · Cited by 11 · see citing works
Roberts, A., Jellicoe, M., & Fox, K. (2021) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10066-7 · Cited by 10 · see citing works
The Modern Teaching Physician—Responsibilities and Challenges: An APDIM White Paper
Steinmann, A. F., Dy, N. M., Kane, G. C., et al. (2009) · The American Journal of Medicine · doi:10.1016/j.amjmed.2009.03.020 · Cited by 9 · see citing works
Comparing the Quality of Narrative Comments by Rotation Setting
Roshan, A., Wagner, N., Acai, A., Emmerton-Coughlin, H. M. A., Sonnadara, R., Scott, T. M., & Karimuddin, A. A. (2021) · Journal of surgical education · doi:10.1016/j.jsurg.2021.06.012 · Cited by 7 · see citing works
Curtis, R., Moon, C. C., Hanmore, T., Hopman, W. M., & Baxter, S. A. (2023) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2023.01.003 · Cited by 4 · see citing works
Self-citationChan, T. M., Sebok‐Syer, S. S., Yılmaz, Y., et al. (2022) · Cureus · doi:10.7759/cureus.23480 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [18]Ross S, Huie M, Schipper S. Use words that count: a content analysis to identify words and phrases that commonly appear in effective formative feedback. International Conference on Residency Education 2015 Oct 22.
Cited by
Sorted by citations
Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S. (2026) · JMIR Medical Education · doi:10.2196/90064 · Cited by 1 · see citing works
“Some reports indicate that feedback length does not contribute to quality improvement beyond a certain threshold [51].”
- Authors
- Curtis, R., Moon, C. C., Hanmore, T., Hopman, W. M., & Baxter, S.
- Affiliations
- Queen's University (CA)
- Corresponding author
- Curtis, Rachel
- Venue
- Canadian Medical Education Journal
- Published
- 2024-04-29
- Volume
- 15
- Issue
- 6
- Pages
- 58-63
- ISSN
- 1923-1202 (ISSN-L)
- PMC
- PMC11725001
- OpenAlex
- W4396515249
- Cited by
- 7
- References
- 25
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 2
- FWCI
- 2.89
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Canadian Medical Education Journal (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Érudit (Université de Montréal) (submittedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Education › Student Assessment and Feedback
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- narrative feedback, word count, feedback quality, competency-based medical education, ophthalmology residency, reflective feedback
- MeSH terms
- Formative Feedback, Clinical Competence, Competency-Based Education, Educational Measurement, Feedback, Humans, Internship and Residency, Ophthalmology, Narration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
