Randomised controlled trials for policy interventions: a review of reviews and meta-regression
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To determine whether randomised controlled trials (RCTs) lead to the same effect size and variance as non-randomised studies (NRSs) of similar policy interventions, and whether these findings can be explained by other factors associated with the interventions or their evaluation. DATA SOURCES: Two RCTs were resampled to compare randomised and non-randomised arms. Comparable field trials were identified from a series of health promotion systematic reviews and a systematic review of transition for youths with disabilities. Previous methodological studies were sought from 14 electronic bibliographic databases (Applied Social Sciences Index and Abstracts, Australian Education Index, British Education Index, CareData, Dissertation Abstracts, EconLIT, Educational Resources Information Centre, International Bibliography of the Sociological Sciences, ISI Proceedings: Social Sciences and Humanities, PAIS International, PsycINFO, SIGLE, Social Science Citation Index, Sociological Abstracts) in June and July 2004. These were supplemented by citation searching for key authors, contacting review authors and searching key internet sites. REVIEW METHODS: Analyses of previous resampling studies, replication studies, comparable field studies and meta-epidemiology investigated the relationship between randomisation and effect size of policy interventions. New resampling studies and new analyses of comparable field studies and meta-epidemiology were strengthened by testing pre-specified associations supported by carefully argued hypotheses. RESULTS: Resampling studies offer no evidence that the absence of randomisation directly influences the effect size of policy interventions in a systematic way. Prior methodological reviews and meta-analyses of existing reviews comparing effects from RCTs and non-randomised controlled trials (nRCTs) suggested that effect sizes from RCTs and nRCTs may indeed differ in some circumstances and that these differences may well be associated with factors confounded with design. No consistent explanations were found for randomisation being associated with changes in effect sizes of policy interventions in field trials. CONCLUSIONS: From the resampling studies we have no evidence that the absence of randomisation directly influences the effect size of policy interventions in a systematic way. At the level of individual studies, non-randomised trials may lead to different effect sizes, but this is unpredictable. Many of the examples reviewed and the new analyses in the current study reveal that randomisation is indeed associated with changes in effect sizes of policy interventions in field trials. Despite extensive analysis, we have identified no consistent explanations for these differences. Researchers mounting new evaluations need to avoid, wherever possible, allocation bias. New policy evaluations should adopt randomised designs wherever possible.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hao (2015) · doi:10.1002/14651858 · Cited by 11,321
Audit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Making psychological theory useful for implementing evidence based practice: a consensus approach
Michie, S. (2005) · Quality and Safety in Health Care · doi:10.1136/qshc.2004.011155 · Cited by 3,440
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Self-citationDes Jarlais, D. C., Lyles, C., Crepaz, N., & the TREND Group (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.361 · Cited by 2,817
The effectiveness of interventions to increase physical activityA systematic review1 and 2
KAHN, E., RAMSEY, L., BROWNSON, R., HEATH, G., HOWZE, E., POWELL, K., STONE, E., RAJAB, M., & CORSO, P. (2002) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(02)00434-8 · Cited by 2,022
Educational outreach visits: effects on professional practice and health care outcomes
O'Brien, M. A., Rogers, S., Jamtvedt, G., Oxman, A. D., Odgaard-Jensen, J., Kristoffersen, D. T., Forsetlund, L., Bainbridge, D., Freemantle, N., Davis, D., et al. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000409.pub2 · Cited by 1,279
Criteria for evaluating evidence on public health interventions
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002) · Journal of Epidemiology and Community Health · doi:10.1136/jech.56.2.119 · Cited by 1,042
Interventions for promoting smoking cessation during pregnancy
Lumley, J., Chamberlain, C., Dowswell, T., Oliver, S., Oakley, L., & Watson, L. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001055.pub3 · Cited by 890
Self-citation
Cited by
Clarifying differences between review designs and methods
Gough, D., Thomas, J., & Oliver, S. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-28 · Cited by 1,063
Self-citationPsychosocial interventions for supporting women to stop smoking in pregnancy
Chamberlain, C., O'Mara-Eves, A., Porter, J., Coleman, T., Perlen, S. M., Thomas, J., & McKenzie, J. E. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001055.pub5 · Cited by 650
Self-citationAnglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534
O’Mara-Eves, A., Brunton, G., McDaid, D., Oliver, S., Kavanagh, J., Jamal, F., Matosevic, T., Harden, A., & Thomas, J. (2013) · Public Health Research · doi:10.3310/phr01040 · Cited by 438
Self-citationon behalf of the CONSORT-SPI Group, Montgomery, P., Grant, S., Mayo-Wilson, E., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2733-1 · Cited by 327
Metadata
- Authors
- Oliver, S., Bagnall, A., Thomas, J., Shepherd, J., Sowden, A., White, I., Dinnes, J., Rees, R., Colquitt, J., Oliver, K., & Garrett, Z.
- Venue
- Health Technology Assessment
- Published
- 2010-03
- Volume
- 14
- Issue
- 16
- Pages
- 1-165, iii
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2041546853
- Cited by
- 76
- References
- 26
- FWCI
- 5.29
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Randomised controlled trials for policy interventions: a review of reviews and meta-regression. https://app.cochat.ai/papers/10.3310/hta14160
- The paper
- Oliver, S., Bagnall, A., Thomas, J., Shepherd, J., Sowden, A., White, I., Dinnes, J., Rees, R., Colquitt, J., Oliver, K., & Garrett, Z. (2010). Randomised controlled trials for policy interventions: a review of reviews and meta-regression. Health Technology Assessment. https://doi.org/10.3310/hta14160
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.