Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
BACKGROUND: There is an increasing number of meta-analyses including data from non-randomized studies for therapeutic evaluation. We aimed to systematically assess the methods used in meta-analyses including non-randomized studies evaluating therapeutic interventions. METHODS: For this methodological review, we searched MEDLINE via PubMed, from January 1, 2013 to December 31, 2013 for meta-analyses including at least one non-randomized study evaluating therapeutic interventions. Etiological assessments and meta-analyses with no comparison group were excluded. Two reviewers independently assessed the general characteristics and key methodological components of the systematic review process and meta-analysis methods. RESULTS: One hundred eighty eight meta-analyses were selected: 119 included both randomized controlled trials (RCTs) and non-randomized studies of interventions (NRSI) and 69 only NRSI. Half of the meta-analyses (n = 92, 49%) evaluated non-pharmacological interventions. "Grey literature" was searched for 72 meta-analyses (38%). An assessment of methodological quality or risk of bias was reported in 135 meta-analyses (72%) but this assessment considered the risk of confounding bias in only 33 meta-analyses (18%). In 130 meta-analyses (69%), the design of each NRSI was not clearly specified. In 131 (70%), whether crude or adjusted estimates of treatment effect for NRSI were combined was unclear or not reported. Heterogeneity across studies was assessed in 182 meta-analyses (97%) and further explored in 157 (84%). Reporting bias was assessed in 127 (68%). CONCLUSIONS: Some key methodological components of the systematic review process-search for grey literature, description of the type of NRSI included, assessment of risk of confounding bias and reporting of whether crude or adjusted estimates were combined-are not adequately carried out or reported in meta-analyses including NRSI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563 · cited 1×
“However, some elements should be adapted specifically to the inclusion of non-randomized studies because their study designs inherently differ from RCTs [7, 9, 14, [18] [19] [20] [21] [22] [23] .”
BackgroundA comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347
Why we need observational studies to evaluate the effectiveness of health care
Black, N. (1996) · BMJ · doi:10.1136/bmj.312.7040.1215 · Cited by 1,598 · cited 1×
“Also, conducting an RCT is sometimes impossible or inappropriate (eg, when studying rare or long-term events) [1, 3, 5], which results in critical information gaps.”
BackgroundThe appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · cited 1×
“Evaluating reporting bias does not exempt the reviewers from searching for grey literature because the assessment of Funnel plot asymmetry may be subjective and statistical methods to test for asymmetry of the plot may lack power [38, 39].”
BackgroundAdherence to cardiovascular therapy: a meta-analysis of prevalence and clinical consequences
Chowdhury, R., Khan, H., Heydon, E., Shroufi, A., Fahimi, S., Moore, C., Stricker, B., Mendis, S., Hofman, A., Mant, J., et al. (2013) · European Heart Journal · doi:10.1093/eurheartj/eht295 · Cited by 931
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919
When are observational studies as credible as randomised trials?
Vandenbroucke, J. P. (2004) · The Lancet · doi:10.1016/s0140-6736(04)16261-2 · Cited by 538
Murad, M. H., Montori, V. M., Ioannidis, J. P. A., Jaeschke, R., Devereaux, P. J., Prasad, K., Neumann, I., Carrasco-Labra, A., Agoritsas, T., Hatala, R., et al. (2014) · JAMA · doi:10.1001/jama.2014.5559 · Cited by 454
Cited by
Witt, K., Spittal, M. J., Carter, G., Pirkis, J., Hetrick, S., Currier, D., Robinson, J., & Milner, A. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1458-0 · Cited by 179
“Sub-group analyses The inclusion of RCTs and non-randomized observational studies within a single meta-analysis is becoming increasingly common as reliance on RCT evidence alone can lead to knowledge translation bias [32].”
BackgroundPrado, C. E., Watt, S., & Crowe, S. F. (2018) · Neuropsychology Review · doi:10.1007/s11065-018-9369-5 · Cited by 188
Meta-analysis of the efficacy of virtual reality exposure therapy for social anxiety
Chesham, R. K., Malouff, J. M., & Schutte, N. S. (2018) · Behaviour Change · doi:10.1017/bec.2018.15 · Cited by 135
Lucchetta, R. C., da Mata, B. P. M., & Mastroianni, P. D. C. (2018) · Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy · doi:10.1002/phar.2170 · Cited by 130
Sharma, S., Hucker, A., Matthews, T., Grohmann, D., & Laws, K. R. (2021) · BMC Psychology · doi:10.1186/s40359-021-00658-8 · Cited by 109
“Whilst it is useful to summarise all available evidence, in meta-analysis, this requires consideration of study design features and adjustment of estimates of effect sizes [35].”
Method
Metadata
- Authors
- Faber, T., Ravaud, P., Riveros, C., Perrodeau, E., & Dechartres, A.
- Venue
- BMC Medical Research Methodology
- Published
- 2016-03-22
- Volume
- 16
- Issue
- 1
- Pages
- 35-35
- ISSN
- 1471-2288 (ISSN-L)
- OpenAlex
- W2310809385
- Cited by
- 100
- References
- 25
- FWCI
- 7.94
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review. https://app.cochat.ai/papers/10.1186/s12874-016-0136-0
- The paper
- Faber, T., Ravaud, P., Riveros, C., Perrodeau, E., & Dechartres, A. (2016). Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-016-0136-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.