The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
A Novel Application of the Modified Angoff Method to Rate Case Difficulty in Simulation-Based Research
Abstract
Introduction: Simulation use in research is often limited by controlling for scenario difficulty when using repeated measures. Our study assesses the feasibility of the Modified Angoff Method to reach expert consensus regarding difficulty of medical simulations. We compared scores with participant physiologic stress. Methods: Emergency medicine physicians with expertise in simulation education were asked to review 8 scenarios and estimate the percentage of resident physicians who would perform all critical actions using the modified Angoff method. A standard deviation (SD) of less than 10% of estimated percentage correct signified consensus. Twenty-five residents then performed the 6 scenarios that met consensus and heart rate variability (HRV) was measured. Results: During round 1, experts rated 4/8 scenarios within a 10% SD for postgraduate year 3 (PGY3) and 3/8 for PGY4 residents. In round 2, 6/8 simulation scenarios were within an SD of 10% points for both years. Intraclass correlation coefficient was 0.84 for PGY3 ratings and 0.89 for PGY4 ratings. A mixed effects analysis of variance showed no significant difference in HRV change from rest to simulation between teams or scenarios. Modified Angoff Score was not a predictor of HRV (multiple R 2 = 0.0176). Conclusions: Modified Angoff ratings demonstrated consensus in quantifying the estimated percentage of participants who would complete all critical actions for most scenarios. Although participant HRV did decrease during the scenarios, we were unable to significantly correlate this with ratings. This modified Angoff method is a feasible approach to evaluate simulation difficulty for educational and research purposes and may decrease the time and resources necessary for scenario piloting.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The rating of item difficulty was assessed based on modified Angoff methods (Table 1) [33, 34].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [34]
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 15,518
Heart rate variability: Standards of measurement, physiological interpretation, and clinical use
Malik, M., Bigger, J. T., Camm, A. J., Kleiger, R. E., Malliani, A., Moss, A. J., & Schwartz, P. J. (1996) · European Heart Journal · doi:10.1093/oxfordjournals.eurheartj.a014868 · Cited by 15,289
APACHE II-A Severity of Disease Classification System
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,379
An overview of heart rate variability metrics and norms
Shaffer, F., & Ginsberg, J. P. (2017) · Frontiers in Public Health · doi:10.3389/fpubh.2017.00258 · Cited by 7,185
The Intraclass Correlation Coefficient as a Measure of Reliability
Bartko, J. (1966) · Psychological Reports · doi:10.2466/pr0.1966.19.1.3 · Cited by 2,206
A critical review of simulation‐based mastery learning with translational outcomes
McGaghie, W. C., Issenberg, S. B., Barsuk, J. H., & Wayne, D. B. (2014) · Medical Education · doi:10.1111/medu.12391 · Cited by 660
Simulation-based medical teaching and learning
Al-Elq, A. (2010) · Journal of Family and Community Medicine · doi:10.4103/1319-1683.68787 · Cited by 621
Barsuk, J. H., McGaghie, W. C., Cohen, E. R., O’Leary, K. J., & Wayne, D. B. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181a57bc1 · Cited by 612
Flagged first, then by citations
Cited by
Sorted by citations
AI in medical education: uses of AI in construction type A MCQs
Rezigalla, A. A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05250-3 · Cited by 36
Sridharan, K., & Sivaramakrishnan, G. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08300-6 · Cited by 9
Joseph, M., Ray, J. M., Chang, J., Cramer, L. D., Bonz, J. W., Yang, T. J., Wong, A. H.-W., Auerbach, M. A., & Evans, L. V. (2022) · AEM Education and Training · doi:10.1002/aet2.10726 · Cited by 6
Self-citationAkbari, A., & Shahrokhi, M. (2024) · Quality Assurance in Education · doi:10.1108/qae-12-2023-0208 · Cited by 2
Aslaner, M. A., Kadı, G., Kesen, S., Kılıç, A. C. K., Çoşkun, Ö., Bildik, F., Keleş, A., Demircan, A., Kılıç, K., Şişik, B., et al. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.08.038 · Cited by 1
Metadata
- Authors
- Joseph, M. N., Chang, J., Buck, S. G., Auerbach, M. A., Wong, A. H., Beardsley, T. D., Reeves, P. M., Ray, J. M., & Evans, L. V.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Joseph, Melissa N.
- ORCID
- Venue
- Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare
- Published
- 2020-12-03
- Volume
- 16
- Issue
- 6
- Pages
- e142-e150
- ISSN
- 1559-2332 (ISSN-L)
- 1559-713X
- OpenAlex
- W3109271711
- Cited by
- 5
- References
- 44
- Citations by year
- 2025: 1 · 2024: 3 · 2022: 1
- FWCI
- 0.15
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Simulation in Healthcare The Journal of the Society for Simulation in Healthcare (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Keywords
- expert consensus, emergency medicine residents, heart rate variability, physiological stress, simulation-based education
- MeSH terms
- Clinical Competence, Educational Measurement, Emergency Medicine, Humans, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
