Natural Language Processing and Assessment of Resident Feedback Quality
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
OBJECTIVE To validate the performance of a natural language processing (NLP) model in characterizing the quality of feedback provided to surgical trainees. DESIGN Narrative surgical resident feedback transcripts were collected from a large academic institution and classified for quality by trained coders. 75% of classified transcripts were used to train a logistic regression NLP model and 25% were used for testing the model. The NLP model was trained by uploading classified transcripts and tested using unclassified transcripts. The model then classified those transcripts into dichotomized high- and low- quality ratings. Model performance was primarily assessed in terms of accuracy and secondary performance measures including sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). SETTING A surgical residency program based in a large academic medical center. PARTICIPANTS All surgical residents who received feedback via the Society for Improving Medical Professional Learning smartphone application (SIMPL, Boston, MA) in August 2019. RESULTS The model classified the quality (high vs. low) of 2,416 narrative feedback transcripts with an accuracy of 0.83 (95% confidence interval: 0.80, 0.86), sensitivity of 0.37 (0.33, 0.45), specificity of 0.97 (0.96, 0.98), and an area under the receiver operating characteristic curve of 0.86 (0.83, 0.87). CONCLUSIONS The NLP model classified the quality of operative performance feedback with high accuracy and specificity. NLP offers residency programs the opportunity to efficiently measure feedback quality. This information can be used for feedback improvement efforts and ultimately, the education of surgical trainees.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Natural language processing techniques for researching and improving peer feedback
Xiong, W., Litman, D., & Schunn, C. (2012) · Journal of Writing Research · doi:10.17239/jowr-2012.04.02.3 · Cited by 36
Research works printed but not linked: 21
- Bjerrum, 2015, Effect of Instructor feedback on skills retention after laparoscopic simulator training: Follow-up of a randomized trial, Journal of Surgical Education, 72, 53, 10.1016/j.jsurg.2014.06.013 · DOI
- Boyle, 2011, Optimising surgical training: Use of feedback to reduce errors during a simulated surgical procedure, Postgraduate Medical Journal, 87, 524, 10.1136/pgmj.2010.109363 · doi:10.1136/pgmj.2010.109363
- Boyle, 2011, The importance of expert feedback during endovascular simulator training, Journal of Vascular Surgery, 54, 10.1016/j.jvs.2011.01.058 · DOI
- Trehan, 2015, The impact of feedback of intraoperative technical performance in surgery: a systematic review, BMJ Open, 5, 10.1136/bmjopen-2014-006759 · DOI
- Porte, 2007, Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills, American Journal of Surgery, 193, 105, 10.1016/j.amjsurg.2006.03.016 · DOI · Paper page
- Wigton, 1986, The effect of feedback in learning clinical diagnosis, Journal of Medical Education, 61, 816
- Bohnen, 2016, The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial, Journal of Surgical Education, 73, 10.1016/j.jsurg.2016.08.010 · DOI · Paper page
- Ahmed, 2013, Operation debrief: a SHARP improvement in performance feedback in the operating room, Annals of surgery, 258, 958, 10.1097/SLA.0b013e31828c88fc · DOI
- Shaughness, 2017, Assessment of clinical feedback given to medical students via an electronic feedback system, Journal of Surgical Research, 218, 174, 10.1016/j.jss.2017.05.055 · DOI
- George, 2017, Readiness of US General Surgery Residents for Independent Practice, Annals of Surgery, 266, 582, 10.1097/SLA.0000000000002414 · DOI · Paper page
- Ende, 1983, Feedback in Clinical Medical Education, JAMA, 250, 777, 10.1001/jama.1983.03340060055026 · DOI · Paper page
- Grantcharov T.P., Schulze S., Kristiansen V.B.. The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room. Surgical Endoscopy and Other Interventional Techniques. 2007;21:2240-2243. http://doi:10.1007/s00464-007-9356-z. · DOI
- Ahle, 2020, The Quality of Operative Performance Narrative Feedback, Annals of Surgery
- Pedregosa, 2011, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research, 12, 2825
- McKinney W, Data Structures for Statistical Computing in Python. In: van der Walt S, Millman J, eds.; 2010. doi:10.25080/Majora-92bf1922-00a · DOI · Paper page
- Harris, 2020, Array programming with NumPy, Nature, 585, 357, 10.1038/s41586-020-2649-2 · DOI · Paper page
- Ramachandran, 2017, Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques, International Journal of Artificial Intelligence in Education, 27, 534, 10.1007/s40593-016-0132-x · DOI · Paper page
- Cho, 2008, Machine Classification of Peer Comments in Physics, Educational Data Mining, 192
- Ali, 2012, Quality of feedback using workplace based assessments in urological training, Journal of Clinical Urology, 5, 39
- Domson, 2019, Instituting a Postoperative Feedback Process for Orthopedic Surgery Residents, Journal of Surgical Education, 76, 1200, 10.1016/j.jsurg.2019.03.007 · DOI
- Shaughnessy, 2019, Improving Satisfaction With Operating Room Feedback: An Effective, Low-Profile, No-Cost Intervention, Journal of Surgical Education, 76, 10.1016/j.jsurg.2019.10.002 · DOI
Cited by
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 679
AI in Medical Education: Global situation, effects and challenges
Zhang, W., Cai, M., Lee, H. J., Evans, R., Zhu, C., & Ming, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12009-8 · Cited by 148
The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156.
Tolsgaard, M. G., Pusic, M. V., Sebok-Syer, S. S., Gin, B., Svendsen, M. B., Syer, M. D., Brydges, R., Cuddy, M. M., & Boscardin, C. K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2180340 · Cited by 137
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023) · arXiv · doi:10.48550/arxiv.2306.13063 · Cited by 618
Metadata
- Authors
- Solano, Q. P., Hayward, L., Chopra, Z., Quanstrom, K., Kendrick, D., Abbott, K. L., Kunzmann, M., Ahle, S., Schuller, M., Ötleş, E., & George, B. C.
- Venue
- Journal of Surgical Education
- Publisher
- Elsevier BV
- Published
- 2021-11
- Volume
- 78
- Issue
- 6
- Pages
- e72-e77
- ISSN
- 1878-7452 (ISSN-L)
- 1931-7204
- OpenAlex
- W3177350052
- Cited by
- 48
- References
- 2
- FWCI
- 6.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Natural Language Processing and Assessment of Resident Feedback Quality. https://app.cochat.ai/papers/10.1016/j.jsurg.2021.05.012
- The paper
- Solano, Q. P., Hayward, L., Chopra, Z., Quanstrom, K., Kendrick, D., Abbott, K. L., Kunzmann, M., Ahle, S., Schuller, M., Ötleş, E., & George, B. C. (2021). Natural Language Processing and Assessment of Resident Feedback Quality. Journal of Surgical Education. https://doi.org/10.1016/j.jsurg.2021.05.012
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.