Cited by 37Innovations in Medical EducationArtificial Intelligence in Healthcare and EducationClinical Reasoning and Diagnostic Skills

Exploring how feedback reflects entrustment decisions using artificial intelligence

Open access
Gin, Brian C., ten Cate, Olle, O'Sullivan, Patricia S., Hauer, Karen E., Boscardin, Christy·2022·Medical Education

Abstract

Context: Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs. Methods: In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment‐based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction. Results: We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently. Conclusions: Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal‐setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.

Integrity signals

Registry

Crossref · 10.1111/medu.14696

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Gin, B. C., ten Cate, O., O'Sullivan, P. S., Hauer, K. E., & Boscardin, C.
Affiliations
  • University of California, San Francisco (US)
  • Utrecht Center for Research and Development of Health Professions Education University Medical Center Utrecht The Netherlands
  • Department of Medicine University of California San Francisco San Francisco CA USA
Corresponding author
Gin, Brian C.
Venue
Medical Education
Publisher
Wiley
Published
2021-12
Volume
56
Issue
3
Pages
303-311
ISSN
  • 0308-0110 (ISSN-L)
  • 1365-2923
OpenAlex
W3213621237
Cited by
37
References
62
Citations by year
2026: 5 · 2025: 13 · 2024: 12 · 2023: 7
FWCI
4.11
Type
journal article
Language
en
Open access
Locations
  • Medical Education (publishedVersion)
  • PubMed (publishedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • eScholarship (California Digital Library) (submittedVersion)
Topics
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
entrustment, clinical supervision, narrative feedback, medical students, clinical rotations, clinical competence, feedback types, task type, goal setting, explainable AI, natural language processing, deep neural network inference
MeSH terms
Artificial Intelligence, Clinical Competence, Competency-Based Education, Feedback, Humans, Internship and Residency, Learning, Students, Medical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review