CoChat

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

Booth, Gregory J, Ross, Benjamin, Cronin, William A, McElrath, Angela, Cyr, Kyle L, Hodgson, John A, Sibley, Charles, Ismawan, J Martin, Zuehl, Alyssa, Slotto, James G, Higgs, Maureen, Haldeman, Matthew, Geiger, Phillip, Jardine, Dink · 2023 · Academic Medicine
Cited by 30

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

Purpose: Faculty feedback on trainees is critical to guiding trainee progress in a competency-based medical education framework. The authors aimed to develop and evaluate a Natural Language Processing (NLP) algorithm that automatically categorizes narrative feedback into corresponding Accreditation Council for Graduate Medical Education Milestone 2.0 subcompetencies. Method: Ten academic anesthesiologists analyzed 5,935 narrative evaluations on anesthesiology trainees at 4 graduate medical education (GME) programs between July 1, 2019, and June 30, 2021. Each sentence (n = 25,714) was labeled with the Milestone 2.0 subcompetency that best captured its content or was labeled as demographic or not useful. Inter-rater agreement was assessed by Fleiss’ Kappa. The authors trained an NLP model to predict feedback subcompetencies using data from 3 sites and evaluated its performance at a fourth site. Performance metrics included area under the receiver operating characteristic curve (AUC), positive predictive value, sensitivity, F1, and calibration curves. The model was implemented at 1 site in a self-assessment exercise. Results: Fleiss’ Kappa for subcompetency agreement was moderate (0.44). Model performance was good for professionalism, interpersonal and communication skills, and practice-based learning and improvement (AUC 0.79, 0.79, and 0.75, respectively). Subcompetencies within medical knowledge and patient care ranged from fair to excellent (AUC 0.66–0.84 and 0.63–0.88, respectively). Performance for systems-based practice was poor (AUC 0.59). Performances for demographic and not useful categories were excellent (AUC 0.87 for both). In approximately 1 minute, the model interpreted several hundred evaluations and produced individual trainee reports with organized feedback to guide a self-assessment exercise. The model was built into a web-based application. Conclusions: The authors developed an NLP model that recognized the feedback language of anesthesiologists across multiple GME programs. The model was operationalized in a self-assessment exercise. It is a powerful tool which rapidly organizes large amounts of narrative feedback.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/acm.0000000000005115

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 18

  1. Powell, 2018, Toward competency-based medical education., N Engl J Med, 378, 3, 10.1056/NEJMp1712900 · DOI
  2. Kogan, 2017, The do’s, don’ts and don’t knows of direct observation of clinical skills in medical education., Perspect Med Educ, 6, 286, 10.1007/S40037-017-0376-7 · DOI · Paper page
  3. Edgar, 2020, Accreditation Council for Graduate Medical Education (ACGME): The Milestones Guidebook
  4. Hanson, 2013, Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States., Front Psychol, 4, 668, 10.3389/fpsyg.2013.00668 · DOI
  5. Chary, 2019, A review of natural language processing in medical education., West J Emerg Med, 20, 78, 10.5811/westjem.2018.11.39725 · DOI · Paper page
  6. Neves, 2021, Using machine learning to evaluate attending feedback on resident performance., Anesth Analg, 132, 545, 10.1213/ANE.0000000000005265 · DOI · Paper page
  7. Zhang, 2012, Automated assessment of medical training evaluation text., AMIA Annu Symp Proc, 1459
  8. Heath, 2019, Assessment of gender-based linguistic differences in physician trainee evaluations of medical faculty using automated text mining., JAMA Netw Open, 2, e193520, 10.1001/jamanetworkopen.2019.3520 · DOI
  9. Luo, 2016, Guidelines for developing and reporting machine learning predictive models in biomedical research: A multidisciplinary view., J Med Internet Res, 18, e323, 10.2196/jmir.5870 · DOI · Paper page
  10. Bird, 2009, Natural Language Processing with Python (version 3.6)
  11. Meskó, 2020, A short guide for medical professionals in the era of artificial intelligence., NPJ Digit Med, 3, 126, 10.1038/s41746-020-00333-z · DOI · Paper page
  12. Otles, 2021, Using natural language processing to automatically assess feedback quality: Findings from three surgical residencies., Acad Med, 96, 1457, 10.1097/ACM.0000000000004153 · DOI · Paper page
  13. Solano, 2021, Natural language processing and assessment of resident feedback quality., J Surg Educ, 78, e72, 10.1016/j.jsurg.2021.05.012 · DOI · Paper page
  14. Holmboe, 2011, Faculty development in assessment: The missing link in competency-based medical education., Acad Med, 86, 460, 10.1097/ACM.0b013e31820cb2a7 · DOI · Paper page
  15. Swing, 2009, Advancing resident assessment in graduate medical education., J Grad Med Educ, 1, 278, 10.4300/JGME-D-09-00010.1 · DOI · Paper page
  16. Kostick-Quenet, 2022, Mitigating racial bias in machine learning., J Law Med Ethics, 50, 92, 10.1017/jme.2022.13 · DOI
  17. Thompson, 2021, Bias and fairness assessment of a natural language processing opioid misuse classifier: Detection and mitigation of electronic health record data disadvantages across racial subgroups., J Am Med Inform Assoc, 28, 2393, 10.1093/jamia/ocab148 · DOI · Paper page
  18. Andrews, 2021, Gender bias in resident evaluations: Natural language processing and competency evaluation., Med Educ, 55, 1383, 10.1111/medu.14593 · DOI

Cited by

See all 30

Metadata

Authors
Booth, G. J., Ross, B., Cronin, W. A., McElrath, A., Cyr, K. L., Hodgson, J. A., Sibley, C., Ismawan, J. M., Zuehl, A., Slotto, J. G., Higgs, M., Haldeman, M., Geiger, P., & Jardine, D.
Venue
Academic Medicine
Published
2022-12-05
Volume
98
Issue
4
Pages
497-504
ISSN
  • 1040-2446 (ISSN-L)
  • 1938-808X
OpenAlex
W4310822639
Cited by
30
References
5
FWCI
4.60
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content. https://app.cochat.ai/papers/10.1097/acm.0000000000005115
The paper
Booth, G. J., Ross, B., Cronin, W. A., McElrath, A., Cyr, K. L., Hodgson, J. A., Sibley, C., Ismawan, J. M., Zuehl, A., Slotto, J. G., Higgs, M., Haldeman, M., Geiger, P., & Jardine, D. (2023). Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content. Academic Medicine. https://doi.org/10.1097/acm.0000000000005115

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.