CoChat

Natural Language Processing and Assessment of Resident Feedback Quality

Solano, Quintin P., Hayward, Laura, Chopra, Zoey, Quanstrom, Kathryn, Kendrick, Daniel, Abbott, Kenneth L., Kunzmann, Marcus, Ahle, Samantha, Schuller, Mary, Ötleş, Erkin, George, Brian C. · 2021 · Journal of Surgical Education
Cited by 48

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

OBJECTIVE To validate the performance of a natural language processing (NLP) model in characterizing the quality of feedback provided to surgical trainees. DESIGN Narrative surgical resident feedback transcripts were collected from a large academic institution and classified for quality by trained coders. 75% of classified transcripts were used to train a logistic regression NLP model and 25% were used for testing the model. The NLP model was trained by uploading classified transcripts and tested using unclassified transcripts. The model then classified those transcripts into dichotomized high- and low- quality ratings. Model performance was primarily assessed in terms of accuracy and secondary performance measures including sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). SETTING A surgical residency program based in a large academic medical center. PARTICIPANTS All surgical residents who received feedback via the Society for Improving Medical Professional Learning smartphone application (SIMPL, Boston, MA) in August 2019. RESULTS The model classified the quality (high vs. low) of 2,416 narrative feedback transcripts with an accuracy of 0.83 (95% confidence interval: 0.80, 0.86), sensitivity of 0.37 (0.33, 0.45), specificity of 0.97 (0.96, 0.98), and an area under the receiver operating characteristic curve of 0.86 (0.83, 0.87). CONCLUSIONS The NLP model classified the quality of operative performance feedback with high accuracy and specificity. NLP offers residency programs the opportunity to efficiently measure feedback quality. This information can be used for feedback improvement efforts and ultimately, the education of surgical trainees.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jsurg.2021.05.012

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 21

  1. Bjerrum, 2015, Effect of Instructor feedback on skills retention after laparoscopic simulator training: Follow-up of a randomized trial, Journal of Surgical Education, 72, 53, 10.1016/j.jsurg.2014.06.013 · DOI
  2. Boyle, 2011, Optimising surgical training: Use of feedback to reduce errors during a simulated surgical procedure, Postgraduate Medical Journal, 87, 524, 10.1136/pgmj.2010.109363 · doi:10.1136/pgmj.2010.109363
  3. Boyle, 2011, The importance of expert feedback during endovascular simulator training, Journal of Vascular Surgery, 54, 10.1016/j.jvs.2011.01.058 · DOI
  4. Trehan, 2015, The impact of feedback of intraoperative technical performance in surgery: a systematic review, BMJ Open, 5, 10.1136/bmjopen-2014-006759 · DOI
  5. Porte, 2007, Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills, American Journal of Surgery, 193, 105, 10.1016/j.amjsurg.2006.03.016 · DOI · Paper page
  6. Wigton, 1986, The effect of feedback in learning clinical diagnosis, Journal of Medical Education, 61, 816
  7. Bohnen, 2016, The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial, Journal of Surgical Education, 73, 10.1016/j.jsurg.2016.08.010 · DOI · Paper page
  8. Ahmed, 2013, Operation debrief: a SHARP improvement in performance feedback in the operating room, Annals of surgery, 258, 958, 10.1097/SLA.0b013e31828c88fc · DOI
  9. Shaughness, 2017, Assessment of clinical feedback given to medical students via an electronic feedback system, Journal of Surgical Research, 218, 174, 10.1016/j.jss.2017.05.055 · DOI
  10. George, 2017, Readiness of US General Surgery Residents for Independent Practice, Annals of Surgery, 266, 582, 10.1097/SLA.0000000000002414 · DOI · Paper page
  11. Ende, 1983, Feedback in Clinical Medical Education, JAMA, 250, 777, 10.1001/jama.1983.03340060055026 · DOI · Paper page
  12. Grantcharov T.P., Schulze S., Kristiansen V.B.. The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room. Surgical Endoscopy and Other Interventional Techniques. 2007;21:2240-2243. http://doi:10.1007/s00464-007-9356-z. · DOI
  13. Ahle, 2020, The Quality of Operative Performance Narrative Feedback, Annals of Surgery
  14. Pedregosa, 2011, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research, 12, 2825
  15. McKinney W, Data Structures for Statistical Computing in Python. In: van der Walt S, Millman J, eds.; 2010. doi:10.25080/Majora-92bf1922-00a · DOI · Paper page
  16. Harris, 2020, Array programming with NumPy, Nature, 585, 357, 10.1038/s41586-020-2649-2 · DOI · Paper page
  17. Ramachandran, 2017, Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques, International Journal of Artificial Intelligence in Education, 27, 534, 10.1007/s40593-016-0132-x · DOI · Paper page
  18. Cho, 2008, Machine Classification of Peer Comments in Physics, Educational Data Mining, 192
  19. Ali, 2012, Quality of feedback using workplace based assessments in urological training, Journal of Clinical Urology, 5, 39
  20. Domson, 2019, Instituting a Postoperative Feedback Process for Orthopedic Surgery Residents, Journal of Surgical Education, 76, 1200, 10.1016/j.jsurg.2019.03.007 · DOI
  21. Shaughnessy, 2019, Improving Satisfaction With Operating Room Feedback: An Effective, Low-Profile, No-Cost Intervention, Journal of Surgical Education, 76, 10.1016/j.jsurg.2019.10.002 · DOI

Cited by

See all 48

Metadata

Authors
Solano, Q. P., Hayward, L., Chopra, Z., Quanstrom, K., Kendrick, D., Abbott, K. L., Kunzmann, M., Ahle, S., Schuller, M., Ötleş, E., & George, B. C.
Venue
Journal of Surgical Education
Publisher
Elsevier BV
Published
2021-11
Volume
78
Issue
6
Pages
e72-e77
ISSN
  • 1878-7452 (ISSN-L)
  • 1931-7204
OpenAlex
W3177350052
Cited by
48
References
2
FWCI
6.64
Type
journal article
Language
en
Open access
closed
Licence
  • www.elsevier.com/tdm/userlicense/1.0/ (tdm)
  • www.elsevier.com/legal/tdmrep-license (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Natural Language Processing and Assessment of Resident Feedback Quality. https://app.cochat.ai/papers/10.1016/j.jsurg.2021.05.012
The paper
Solano, Q. P., Hayward, L., Chopra, Z., Quanstrom, K., Kendrick, D., Abbott, K. L., Kunzmann, M., Ahle, S., Schuller, M., Ötleş, E., & George, B. C. (2021). Natural Language Processing and Assessment of Resident Feedback Quality. Journal of Surgical Education. https://doi.org/10.1016/j.jsurg.2021.05.012

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.