Using Machine Learning to Assess Physician Competence: A Systematic Review
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Purpose: To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different competence domains in several medical specialties. Method: In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students’, residents’, fellows’, or attending physicians’ competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted. Results: Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24; 34.8%) and radiology (15; 21.7%) were the most studied specialties; natural language processing (24; 34.8%), support vector machine (15; 21.7%), and hidden Markov models (14; 20.3%) were the ML techniques most often applied; and patient care (63; 91.3%) and medical knowledge (45; 65.2%) were the most assessed competence domains. Conclusions: A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Competency-based medical education: Theory to practice.
Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., Harris, P., Glasgow, N. J., Campbell, C., Dath, D., et al. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.501190 · Cited by 2,883
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
Englander, R., Cameron, T., Ballard, A. J., Dodge, J., Bull, J., & Aschenbrener, C. A. (2013) · Academic Medicine · doi:10.1097/acm.0b013e31829a3b2b · Cited by 565
Implementation of competency‐based medical education: are we addressing the concerns and challenges?
Hawkins, R. E., Welcher, C. M., Holmboe, E. S., Kirk, L. M., Norcini, J. J., Simons, K. B., & Skochelak, S. E. (2015) · Medical Education · doi:10.1111/medu.12831 · Cited by 409
Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions
Lin, H. C., Shafran, I., Yuh, D., & Hager, G. D. (2006) · Computer Aided Surgery · doi:10.3109/10929080600989189 · Cited by 254
El Saadawi, G. M., Tseytlin, E., Legowski, E., Jukic, D., Castine, M., Fine, J., Gormley, R., & Crowley, R. S. (2008) · Advances in Health Sciences Education · doi:10.1007/s10459-007-9081-3 · Cited by 48
Cited by
Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025) · Journal of Medical Internet Research · doi:10.2196/68486 · Cited by 96
“Artificial intelligence (AI) has long been used to rate narrative text [33-37], but this typically requires supervised machine learning—using human-graded texts to train the AI system.”
Demystifying AI: Current State and Future Role in Medical Education Assessment
Turner, L., Hashimoto, D. A., Vasisht, S., & Schaye, V. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005598 · Cited by 70
“23 For instance, models like Schaye et al ’ s ML model for automated assessment of resident clinical reasoning documentation are examples of supervised ML that use text-based labeled datasets.”
MethodMachine learning to extract communication and history‐taking skills in OSCE transcripts
Jani, K. H., Jones, K. A., Jones, G. W., Amiel, J., Barron, B., & Elhadad, N. (2020) · Medical Education · doi:10.1111/medu.14347 · Cited by 39
(2022) · doi:10.2196/30537 · Cited by 34
Machine Scoring of Medical Students’ Written Clinical Reasoning: Initial Validity Evidence
Cianciolo, A. T., LaVoie, N., & Parker, J. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004010 · Cited by 28
“27 Re-rating the diagnostic justification essays using a refined rubric was somewhat unusual for machine scoring projects and would not have been feasible outside of a funded research project.”
Background
Metadata
- Authors
- Dias, R. D., Gupta, A., & Yule, S. J.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-03
- Volume
- 94
- Issue
- 3
- Pages
- 427-439
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W2885740121
- Cited by
- 70
- References
- 8
- FWCI
- 1.38
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using Machine Learning to Assess Physician Competence: A Systematic Review. https://app.cochat.ai/papers/10.1097/acm.0000000000002414
- The paper
- Dias, R. D., Gupta, A., & Yule, S. J. (2019). Using Machine Learning to Assess Physician Competence: A Systematic Review. Academic Medicine. https://doi.org/10.1097/acm.0000000000002414
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.