Using Machine Learning to Evaluate Attending Feedback on Resident Performance
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
BACKGROUND: High-quality and high-utility feedback allows for the development of improvement plans for trainees. The current manual assessment of the quality of this feedback is time consuming and subjective. We propose the use of machine learning to rapidly distinguish the quality of attending feedback on resident performance. METHODS: Using a preexisting databank of 1925 manually reviewed feedback comments from 4 anesthesiology residency programs, we trained machine learning models to predict whether comments contained 6 predefined feedback traits (actionable, behavior focused, detailed, negative feedback, professionalism/communication, and specific) and predict the utility score of the comment on a scale of 1–5. Comments with ≥4 feedback traits were classified as high-quality and comments with ≥4 utility scores were classified as high-utility; otherwise comments were considered low-quality or low-utility, respectively. We used RapidMiner Studio (RapidMiner, Inc, Boston, MA), a data science platform, to train, validate, and score performance of models. RESULTS: Models for predicting the presence of feedback traits had accuracies of 74.4%–82.2%. Predictions on utility category were 82.1% accurate, with 89.2% sensitivity, and 89.8% class precision for low-utility predictions. Predictions on quality category were 78.5% accurate, with 86.1% sensitivity, and 85.0% class precision for low-quality predictions. Fifteen to 20 hours were spent by a research assistant with no prior experience in machine learning to become familiar with software, create models, and review performance on predictions made. The program read data, applied models, and generated predictions within minutes. In contrast, a recent manual feedback scoring effort by an author took 15 hours to manually collate and score 200 comments during the course of 2 weeks. CONCLUSIONS: Harnessing the potential of machine learning allows for rapid assessment of attending feedback on resident performance. Using predictive models to rapidly screen for low-quality and low-utility feedback can aid programs in improving feedback provision, both globally and by individual faculty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Machine learning for medical imaging
Erickson, B. J., Korfiatis, P., Akkus, Z., & Kline, T. L. (2017) · RadioGraphics · doi:10.1148/rg.2017160130 · Cited by 1,672
Current Applications and Future Impact of Machine Learning in Radiology
Choy, G., Khalilzadeh, O., Michalski, M., Do, S., Samir, A. E., Pianykh, O. S., Geis, J. R., Pandharipande, P. V., Brink, J. A., & Dreyer, K. J. (2018) · Radiology · doi:10.1148/radiol.2018171820 · Cited by 800
On the ethics of algorithmic decision-making in healthcare
Grote, T., & Berens, P. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105586 · Cited by 551
Educational feedback in the operating room: A gap between resident and faculty perceptions.
Jensen, A. R., Wright, A. S., Kim, S., Horvath, K. D., & Calhoun, K. E. (2012) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2011.08.019 · Cited by 174
“Directed” self-assessment: Practice and feedback within a social context
Sargeant, J., Mann, K., van der Vleuten, C., & Metsemakers, J. (2008) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.155 · Cited by 158
Research works printed but not linked: 10
- Ende, 1983, Feedback in clinical medical education., JAMA, 250, 777, 10.1001/jama.1983.03340060055026 · DOI · Paper page
- Angus, 2015, Internal medicine residents’ perspectives on receiving feedback in milestone format., J Grad Med Educ, 7, 220, 10.4300/JGME-D-14-00446.1 · DOI
- Myerholtz, 2019, Residency faculty teaching evaluation: what do faculty, residents, and program directors want?, Fam Med, 51, 509, 10.22454/FamMed.2019.168353 · DOI
- Mitchell, 2016, Faculty development in feedback provision., Int Anesthesiol Clin, 54, 54, 10.1097/AIA.0000000000000109 · DOI
- Mitchell, 2017, Enhancing feedback on professionalism and communication skills in anesthesia residency programs., Anesth Analg, 125, 620, 10.1213/ANE.0000000000002143 · DOI
- Awan, 2018, Machine learning in heart failure: ready for prime time., Curr Opin Cardiol, 33, 190, 10.1097/HCO.0000000000000491 · DOI
- Arora, 2018, Harnessing the power of big data to improve graduate medical education: big idea or bust?, Acad Med, 93, 833, 10.1097/ACM.0000000000002209 · DOI · Paper page
- Thanawala, 2018, Novel educational information management platform improves the surgical skill evaluation process of surgical residents., J Surg Educ, 75, e204, 10.1016/j.jsurg.2018.06.004 · DOI
- Amirhajlou, 2019, Application of data mining techniques for predicting residents’ performance on pre-board examinations: a case study., J Educ Health Promot, 8, 108, 10.4103/jehp.jehp_394_18 · doi:10.4103/jehp.jehp_394_18
- Jin, 2020, Is BERT really robust? A strong baseline for natural language attack on text classification and entailment., Proceedings of the AAAI Conference on Artificial Intelligence, 34, 8018, 10.1609/aaai.v34i05.6311 · DOI · Paper page
Cited by
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 679
AI in Medical Education: Global situation, effects and challenges
Zhang, W., Cai, M., Lee, H. J., Evans, R., Zhu, C., & Ming, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12009-8 · Cited by 148
The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156.
Tolsgaard, M. G., Pusic, M. V., Sebok-Syer, S. S., Gin, B., Svendsen, M. B., Syer, M. D., Brydges, R., Cuddy, M. M., & Boscardin, C. K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2180340 · Cited by 137
Shaw, K., Henning, M. A., & Webster, C. S. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02373-0 · Cited by 58
Metadata
- Authors
- Neves, S. E., Chen, M. J., Ku, C. M., Karan, S., DiLorenzo, A. N., Schell, R. M., Lee, D. E., Diachun, C. A. B., Jones, S. B., & Mitchell, J. D.
- Venue
- Anesthesia & Analgesia
- Published
- 2020-12-09
- Volume
- 132
- Issue
- 2
- Pages
- 545-555
- ISSN
- 0003-2999 (ISSN-L)
- OpenAlex
- W3111931458
- Cited by
- 28
- References
- 7
- FWCI
- 1.72
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using Machine Learning to Evaluate Attending Feedback on Resident Performance. https://app.cochat.ai/papers/10.1213/ane.0000000000005265
- The paper
- Neves, S. E., Chen, M. J., Ku, C. M., Karan, S., DiLorenzo, A. N., Schell, R. M., Lee, D. E., Diachun, C. A. B., Jones, S. B., & Mitchell, J. D. (2021). Using Machine Learning to Evaluate Attending Feedback on Resident Performance. Anesthesia & Analgesia. https://doi.org/10.1213/ane.0000000000005265
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.