Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Purpose: Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback. This study evaluated which NLP techniques best classify the quality of surgical trainee formative feedback recorded as part of a workplace assessment. Method: During the 2016–2017 academic year, the SIMPL (Society for Improving Medical Professional Learning) app was used to record operative performance narrative feedback for residents at 3 university-based general surgery residency training programs. Feedback comments were collected for a sample of residents representing all 5 postgraduate year levels and coded for quality. In May 2019, the coded comments were then used to train NLP models to automatically classify the quality of feedback across 4 categories (effective, mediocre, ineffective, or other). Models included support vector machines (SVM), logistic regression, gradient boosted trees, naive Bayes, and random forests. The primary outcome was mean classification accuracy. Results: The authors manually coded the quality of 600 recorded feedback comments. Those data were used to train NLP models to automatically classify the quality of feedback across 4 categories. The NLP model using an SVM algorithm yielded a maximum mean accuracy of 0.64 (standard deviation, 0.01). When the classification task was modified to distinguish only high-quality vs low-quality feedback, maximum mean accuracy was 0.83, again with SVM. Conclusions: To the authors’ knowledge, this is the first study to examine the use of NLP for classifying feedback quality. SVM NLP models demonstrated the ability to automatically classify the quality of surgical trainee evaluations. Larger training datasets would likely further increase accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Taylor, D. C. M., & Hamdy, H. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.828153 · Cited by 1,176
Mattar, S. G., Alseidi, A. A., Jones, D. B., Jeyarajah, D. R., Swanstrom, L. L., Aye, R. W., Wexner, S. D., Martinez, J. M., Ross, S. B., Awad, M. M., et al. (2013) · Annals of Surgery · doi:10.1097/sla.0b013e3182a191ca · Cited by 788
What is feedback in clinical education?
Van De Ridder, J. M. M., Stokking, K. M., McGaghie, W. C., & Ten Cate, O. T. J. (2008) · Medical Education · doi:10.1111/j.1365-2923.2007.02973.x · Cited by 686
Sapci, A. H., & Sapci, H. A. (2020) · JMIR Medical Education · doi:10.2196/19285 · Cited by 377
Research works printed but not linked: 17
- Wolverton, 1985, A survey of resident perceptions of effective teaching behaviors., Fam Med, 17, 106
- Heckman-Stone, 2004, Trainee preferences for feedback and evaluation in clinical supervision., Clin Supervisor, 22, 21, 10.1300/J001v22n01_03 · DOI
- Grantcharov, 2007, The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room., Surg Endosc, 21, 2240, 10.1007/s00464-007-9356-z · DOI
- Boyle, 2011, Optimising surgical training: Use of feedback to reduce errors during a simulated surgical procedure., Postgrad Med J, 87, 524, 10.1136/pgmj.2010.109363 · doi:10.1136/pgmj.2010.109363
- Strandbygaard, 2013, Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: A randomized trial., Ann Surg, 257, 839, 10.1097/SLA.0b013e31827eee6e · DOI · Paper page
- Trehan, 2015, The impact of feedback of intraoperative technical performance in surgery: A systematic review., BMJ Open, 5, e006759, 10.1136/bmjopen-2014-006759 · DOI
- George, 2017, Readiness of US general surgery residents for independent practice., Ann Surg, 266, 582, 10.1097/SLA.0000000000002414 · DOI · Paper page
- Williams, 2016, Practice guidelines for operative performance assessments., Ann Surg, 264, 934, 10.1097/SLA.0000000000001685 · DOI
- Ali, 2012, Quality of feedback using workplace based assessments in urological training., British J Med Surg Urology, 5, 39, 10.1016/j.bjmsu.2011.10.001 · DOI
- Ahle, The quality of operative performance narrative feedback: A retrospective data comparison between end of rotation evaluations and workplace-based assessments [published online ahead of print June 4, 2020]., Ann of Surg
- Larson, 2005, Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents., Surgery, 138, 640, 10.1016/j.surg.2005.07.017 · DOI · Paper page
- Wohaibi, 2007, A new web-based operative skills assessment tool effectively tracks progression in surgical resident performance., J Surg Educ, 64, 333, 10.1016/j.jsurg.2007.06.015 · doi:10.1016/j.jsurg.2007.06.015
- Bohnen, 2016, The feasibility of real-time intraoperative performance assessment with SIMPL (System for Improving and Measuring Procedural Learning): Early experience from a multi-institutional trial., J Surg Educ, 73, e118, 10.1016/j.jsurg.2016.08.010 · DOI · Paper page
- Chary, 2019, A review of natural language processing in medical education., West J Emerg Med, 20, 78, 10.5811/westjem.2018.11.39725 · DOI
- Denny, 2009, Tracking medical students’ clinical experiences using natural language processing., J Biomed Inform, 42, 781, 10.1016/j.jbi.2009.02.004 · DOI
- Goldberg, 2017, Neural Network Methods for Natural Language Processing Synthesis Lectures on Human Language Technologies, 1
- George, 2020, Using smartphones for trainee performance assessment: A SIMPL case study., Surgery, 167, 903, 10.1016/j.surg.2019.09.011 · DOI
Cited by
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 679
AI in Medical Education: Global situation, effects and challenges
Zhang, W., Cai, M., Lee, H. J., Evans, R., Zhu, C., & Ming, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12009-8 · Cited by 148
Shaw, K., Henning, M. A., & Webster, C. S. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02373-0 · Cited by 58
Exploring how feedback reflects entrustment decisions using artificial intelligence
Gin, B. C., ten Cate, O., O'Sullivan, P. S., Hauer, K. E., & Boscardin, C. (2022) · Medical Education · doi:10.1111/medu.14696 · Cited by 37
Van Ostaeyen, S., Embo, M., Rotsaert, T., De Clercq, O., Schellens, T., & Valcke, M. (2023) · Perspectives on Medical Education · doi:10.5334/pme.1050 · Cited by 6
Metadata
- Authors
- Ötleş, E., Kendrick, D. E., Solano, Q. P., Schuller, M., Ahle, S. L., Eskender, M. H., Carnes, E., & George, B. C.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-05-04
- Volume
- 96
- Issue
- 10
- Pages
- 1457-1460
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W3157626300
- Cited by
- 48
- References
- 5
- FWCI
- 5.86
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies. https://app.cochat.ai/papers/10.1097/acm.0000000000004153
- The paper
- Ötleş, E., Kendrick, D. E., Solano, Q. P., Schuller, M., Ahle, S. L., Eskender, M. H., Carnes, E., & George, B. C. (2021). Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies. Academic Medicine. https://doi.org/10.1097/acm.0000000000004153
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.