CoChat
Sign Up
Cited by 72Simulation-Based Education in HealthcareInnovations in Medical EducationArtificial Intelligence in Healthcare and Education

Using virtual standardized patients to accurately assess information gathering skills in medical students

Maicher, Kellen R., Zimmerman, Laura, Wilcox, Bruce, Liston, Beth, Cronau, Holly, Macerollo, Allison, Jin, Lifeng, Jaffe, Evan +5 more·2019·Medical Teacher
DOI 10.1080/0142159x.2019.1616683
Open this paper in CoChat — free account

Abstract

Introduction: Practicing a medical history using standardized patients is an essential component of medical school curricula. Recent advances in technology now allow for newer approaches for practicing and assessing communication skills. We describe herein a virtual standardized patient (VSP) system that allows students to practice their history taking skills and receive immediate feedback.Methods: Our VSPs consist of artificially intelligent, emotionally responsive 3D characters which communicate with students using natural language. The system categorizes the input questions according to specific domains and summarizes the encounter. Automated assessment by the computer was compared to manual assessment by trained raters to assess accuracy of the grading system.Results: Twenty dialogs chosen randomly from 102 total encounters were analyzed by three human and one computer rater. Overall scores calculated by the computer were not different than those provided by the human raters, and overall accuracy of the computer system was 87%, compared with 90% for human raters. Inter-rater reliability was high across 19 of 21 categories.Conclusions: We have developed a virtual standardized patient system that can understand, respond, categorize, and assess student performance in gathering information during a typical medical history, thus enabling students to practice their history-taking skills and receive immediate feedback.

Integrity signals

Registry

Crossref · 10.1080/0142159x.2019.1616683

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Maicher et al. [45] automatically assess and grade medical students' collection of patient information.

    “The study by Maicher et al. [45] goes along the same lines by automatically assessing and grading medical students' collection of patient information.”

    From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [45]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Maicher, K. R., Zimmerman, L., Wilcox, B., Liston, B., Cronau, H., Macerollo, A., Jin, L., Jaffe, E., White, M., Fosler-Lussier, E., Schuler, W., Way, D. P., & Danforth, D. R.
Affiliations
  • The Ohio State University Comprehensive Cancer Center – Arthur G. James Cancer Hospital and Richard J. Solove Research Institute (US)
  • The Ohio State University (US)
  • Brillig Understanding, Inc., Atascadero, CA, USA;
  • The Department of Linguistics, The Ohio State University, Columbus, OH, USA;
Corresponding author
Danforth, Douglas R.
ORCID
Venue
Medical Teacher
Publisher
Informa UK Limited
Published
2019-06-22
Volume
41
Issue
9
Pages
1053-1059
ISSN
  • 0142-159X (ISSN-L)
  • 1466-187X
DOI
10.1080/0142159x.2019.1616683
OpenAlex
W2951192900
Cited by
72
References
9
Citations by year
2026: 16 · 2025: 18 · 2024: 10 · 2023: 5 · 2022: 8 · 2021: 8 · 2020: 5 · 2019: 2
FWCI
3.94
Type
journal article
Language
en
Open access
closed
Locations
  • Medical Teacher (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • National Board of Medical Examiners (NBME) Edward J. Stemmler, MD Medical Education Research Fund (NBME 1112-064)
  • Department of Health and Human Services Health Resources and Services Administration (HRSA D56HP020687)
Topics
  • Health Sciences › Medicine › Physiology › Simulation-Based Education in Healthcare
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
virtual standardized patients, medical history, information search skills, medical students, computer-based testing, automated assessment, inter-rater reliability, natural language dialogue, immediate feedback, communication skills training
MeSH terms
Virtual Reality, Analysis of Variance, Artificial Intelligence, Education, Medical, Undergraduate, Humans, Medical History Taking, Physician-Patient Relations, Surveys and Questionnaires, Students, Medical, User-Computer Interface

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review