CoChat
Sign Up
Cited by 243Innovations in Medical EducationHealth and Medical Research ImpactsStatistical Methods in Clinical Trials

Getting Off the “Gold Standard”: Randomized Controlled Trials and Education Research

Open access
Sullivan, Gail M·2011·Journal of Graduate Medical Education
DOI 10.4300/jgme-d-11-00147.1PMC PMC3179209
Open this paper in CoChat — free accountOpen PDF

Abstract

Randomization may be achieved at the expense of relevance. L. J. Cronbach, Designing Evaluations of Educational and Social Problems1Medical education researchers are inevitably more familiar with the multisite randomized controlled trial (RCT) “gold standard” from the clinical research paradigm. In an RCT, trainees are randomly assigned to receive 1 of 2 or more educational interventions. Randomized controlled trials are quantitative, comparative, controlled experiments in which treatment effect sizes may be determined with less bias than observational trials.2 Randomization is considered the most powerful experimental design in clinical trials: with other variables equal between groups, on average, any differences in outcome can be attributed to the intervention.2With trainees moving through educational processes in real time, the difficulties of randomization become immediately clear. Residents and fellows usually experience rotations at different times: those experiencing the intervention later may learn from intervening experiences and no longer be comparable to those experiencing the intervention at an earlier time. Also, trainees have the option to refuse randomization, yet they cannot miss critical educational experiences. Use of a “placebo” is often contraindicated. Multisite interventions, while providing more subjects (with greater power to detect differences) and more generalizability, present challenges due to training differences. Also, the cost of multisite studies can be high and industry support for education trials rarely exists.While large numbers of medical undergraduates often receive nearly identical “treatments,” in which 1 or more variables may be altered, this is not true for graduate medical education. Residency and fellowship training are usually highly individualized, which makes the RCT model increasingly unsuitable as training advances.In addition to feasibility considerations, experts question the applicability of research models, derived from clinical research, for education studies.3,4 The highly complex system of education may be a poor fit for the RCT model, which requires clear inclusion/exclusion criteria and interventions administered identically via multiple physicians (ie, teachers).3,5,6 Regehr asks whether simulating placebo-controlled efficacy clinical trials, in which 1 or a few variables may be tightly controlled, is a worthwhile goal for medical education research.3 In education studies, variables can rarely be controlled tightly and blinding of subjects and study personnel may be unethical or impossible.3,7 Finally, defining the therapeutic intervention in education research is much more difficult than in clinical trials. As Norman suggests, one cannot “apply curriculum daily” in the same way that one can prescribe a medication.5The primary advantage of randomization is that it reduces allocation bias, which derives from baseline variables that may influence outcome(s). Randomization ensures that baseline characteristics, not known to be related to the outcome of interest, are equally distributed among the groups. Although differences among participants are 1 source of error, randomization will not control for other sources of error, which are likely to occur in education studies (table 1). Variations in those implementing the intervention, settings, and other execution factors may have more impact than baseline variations in the subjects. Other common “confounders” that randomization may not control for are effects of pretests on learning (encouraging differential study or learning);8 Hawthorne effects (changes in participant motivation); effects of other, nonintervention training experiences occurring during the study intervention;5 and high participant dropout (eg, less than 75% response rates).9 Contextual factors may affect outcomes in ways that randomization cannot fix.10 Especially if the intervention is fairly dilute (eg, a workshop, short course, or online cases), it may not be apparent whether the intervention is causing the outcome effects versus contextual factors.In education studies it is often difficult to “blind” learners to their assigned group. Without blinding, residents can react to the knowledge that they are being studied or assigned to a particular group. Within training programs trainees interact to a great extent, resulting in contamination effects (ie, trainees sharing learning with each other) that further compromise randomization. Active interventions that are deemed critical to learning cannot be withheld, although crossover designs may be used in this situation. However, crossover designs may also involve contamination of learning between groups.When should an RCT be used in education experiments? According to Norman,6 randomization is most useful in examining relatively standardized interventions, such as web-based learning and, possibly, clinical simulation. He recommends that randomization be considered when (1) prior observation

Integrity signals

Registry

Crossref · 10.4300/jgme-d-11-00147.1

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Sullivan, G. M.
Affiliations
University of Connecticut (US)
Corresponding author
Sullivan, Gail M
ORCID
Sullivan, Gail M
Venue
Journal of Graduate Medical Education
Publisher
Journal of Graduate Medical Education
Published
2011-09
Volume
3
Issue
3
Pages
285-289
ISSN
  • 1949-8349 (ISSN-L)
  • 1949-8357
DOI
10.4300/jgme-d-11-00147.1
PMC
PMC3179209
OpenAlex
W1966974590
Cited by
243
References
18
Citations by year
2026: 12 · 2025: 16 · 2024: 22 · 2023: 17 · 2022: 17 · 2021: 20 · 2020: 27 · 2019: 30 · 2018: 25 · 2017: 21 · 2016: 19 · 2015: 7 · 2014: 4 · 2013: 4 · 2012: 2
FWCI
3.15
Type
journal article
Language
en
Open access
Locations
  • Journal of Graduate Medical Education (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
randomized controlled trials, medical education research, graduate medical education, residency training, allocation bias, individualized training, educational intervention, study design limitations, generalizability of research findings, individual differences in learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review