Assessment of Diagnostic Competences With Standardized Patients Versus Virtual Patients: Experimental Study in the Context of History Taking
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Background: Standardized patients (SPs) have been one of the popular assessment methods in clinical teaching for decades, although they are resource intensive. Nowadays, simulated virtual patients (VPs) are increasingly used because they are permanently available and fully scalable to a large audience. However, empirical studies comparing the differential effects of these assessment methods are lacking. Similarly, the relationships between key variables associated with diagnostic competences (ie, diagnostic accuracy and evidence generation) in these assessment methods still require further research. Objective: The aim of this study is to compare perceived authenticity, cognitive load, and diagnostic competences in performance-based assessment using SPs and VPs. This study also aims to examine the relationships of perceived authenticity, cognitive load, and quality of evidence generation with diagnostic accuracy. Methods: We conducted an experimental study with 86 medical students (mean 26.03 years, SD 4.71) focusing on history taking in dyspnea cases. Participants solved three cases with SPs and three cases with VPs in this repeated measures study. After each case, students provided a diagnosis and rated perceived authenticity and cognitive load. The provided diagnosis was scored in terms of diagnostic accuracy; the questions asked by the medical students were rated with respect to their quality of evidence generation. In addition to regular null hypothesis testing, this study used equivalence testing to investigate the absence of meaningful effects. Results: Perceived authenticity (1-tailed t81=11.12; P<.001) was higher for SPs than for VPs. The correlation between diagnostic accuracy and perceived authenticity was very small (r=0.05) and neither equivalent (P=.09) nor statistically significant (P=.32). Cognitive load was equivalent in both assessment methods (t82=2.81; P=.003). Intrinsic cognitive load (1-tailed r=−0.30; P=.003) and extraneous load (1-tailed r=−0.29; P=.003) correlated negatively with the combined score for diagnostic accuracy. The quality of evidence generation was positively related to diagnostic accuracy for VPs (1-tailed r=0.38; P<.001); this finding did not hold for SPs (1-tailed r=0.05; P=.32). Comparing both assessment methods with each other, diagnostic accuracy was higher for SPs than for VPs (2-tailed t85=2.49; P=.01). Conclusions: The results on perceived authenticity demonstrate that learners experience SPs as more authentic than VPs. As higher amounts of intrinsic and extraneous cognitive loads are detrimental to performance, both types of cognitive load must be monitored and manipulated systematically in the assessment. Diagnostic accuracy was higher for SPs than for VPs, which could potentially negatively affect students’ grades with VPs. We identify and discuss possible reasons for this performance difference between both assessment methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The assessment of clinical skills/competence/performance
Miller, G. E. (1990) · Academic Medicine · doi:10.1097/00001888-199009000-00045 · Cited by 4,883
Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 3,001
Cognitive Architecture and Instructional Design: 20 Years Later
Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09465-5 · Cited by 2,337 · cited 1×
“reported negative effects of intrinsic and extraneous cognitive loads on complex problem solving in medical education [27] and other domains [51].”
BackgroundThe Experience of Presence: Factor Analytic Insights
Schubert, T., Friedmann, F., & Regenbrecht, H. (2001) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474601300343603 · Cited by 2,303
Assessment of clinical competence using objective structured examination.
Harden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975) · BMJ · doi:10.1136/bmj.1.5955.447 · Cited by 1,687
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · cited 4×
“These inconclusive findings may stem from the lack of statistical power because of the relatively small sample size [45].”
BackgroundMethodInfluentialExpectancy-value theory of achievement motivation: A developmental perspective
Wigfield, A. (1994) · Educational Psychology Review · doi:10.1007/bf02209024 · Cited by 1,305 · cited 1×
“The value component of motivation was measured with a 4-item, 5-point scale based on a questionnaire by Wigfield [43].”
MethodSimulation-Based Learning in Higher Education: A Meta-Analysis
Chernikova, O., Heitzmann, N., Stadler, M., Holzberger, D., Seidel, T., & Fischer, F. (2020) · Review of Educational Research · doi:10.3102/0034654320933544 · Cited by 1,072 · cited 1×
“A meta-analysis on simulation-based learning of complex skills [12] added to this framework that authenticity should also be explored as an important moderator in assessment and learning.”
BackgroundSelf-citationCognitive Load Theory: Implications for medical education: AMEE Guide No. 86
Young, J. Q., Van Merrienboer, J., Durning, S., & Ten Cate, O. (2014) · Medical Teacher · doi:10.3109/0142159x.2014.889290 · Cited by 935
The Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP)
Lewis, K. L., Bohnert, C. A., Gammon, W. L., Hölzer, H., Lyman, L., Smith, C., Thompson, T. M., Wallace, A., & Gliva-McConvey, G. (2017) · Advances in Simulation · doi:10.1186/s41077-017-0043-4 · Cited by 580
Cited by
Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025) · Journal of Medical Internet Research · doi:10.2196/68486 · Cited by 96
“We added a third empirically derived framework for evaluating “presence” in virtual reality (ie, realness, involvement, and spatial “physical” presence) [54,55].”
Virtual Standardized Patients for Mental Health Education
Reger, G. M., Norr, A. M., Gramlich, M. A., & Buchman, J. M. (2021) · Current Psychiatry Reports · doi:10.1007/s11920-021-01273-5 · Cited by 38
Authentic and Creative Assessment in a World with AI
Miserandino, M. (2025) · Teaching of Psychology · doi:10.1177/00986283241260370 · Cited by 15
“Medical students developed better diagnostic skills by interacting with real patients in a hospital setting while receiving feedback from their instructor and their peers (Fink et al., 2021).”
Qiu, K., Zeng, T., Xia, W., Peng, M., & Kong, W. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07631-8 · Cited by 6
“Proper case selection can significantly enhance the authenticity of the teaching process and ensure that all students can actively engage with the educational content [21, 22].”
Metadata
- Authors
- Fink, M. C., Reitmeier, V., Stadler, M., Siebeck, M., Fischer, F., & Fischer, M. R.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-03-04
- Volume
- 23
- Issue
- 3
- Pages
- e21196
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W3118220099
- Cited by
- 36
- References
- 11
- FWCI
- 2.14
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.