An introduction to methodological issues when including non‐randomised studies in systematic reviews on the effects of interventions
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Background: Methods need to be further developed to include non‐randomised studies (NRS) in systematic reviews of the effects of health care interventions. NRS are often required to answer questions about harms and interventions for which evidence from randomised controlled trials (RCTs) is not available. Methods used to review randomised controlled trials may be inappropriate or insufficient for NRS. Aim and methods: A workshop was convened to discuss relevant methodological issues. Participants were invited from important stakeholder constituencies, including methods and review groups of the Cochrane and Campbell Collaborations, the Cochrane Editorial Unit and organisations that commission reviews and make health policy decisions. The aim was to discuss methods for reviewing evidence when including NRS and to formulate methodological guidance for review authors. Workshop format: The workshop was structured around four sessions on topics considered in advance to be most critical: (i) study designs and bias; (ii) confounding and meta‐analysis; (iii) selective reporting; and (iv) applicability. These sessions were scheduled between introductory and concluding sessions. Summary: This is the first of six papers and provides an overview. Subsequent papers describe the discussions and conclusions from the four main sessions (papers 2 to 5) and summarise the proposed guidance into lists of issues for review authors to consider (paper 6). Copyright © 2013 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Hao (2015) · doi:10.1002/14651858 · Cited by 11,321
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Self-citationEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · cited 2×
“Moreover, regression adjustment using measured risk factors does not usually fully correct for the biases (Deeks et al., 2003).”
BackgroundSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 4×
“…RCTs; Dwan and colleagues noted that statistically significant outcomes had a higher odds of being fully reported compared with non-significant outcomes (range of OR, 2.2 to 4.7) and that changes in pre-specified outcomes (not documented in amendments) also frequently occur (Dwan et al., 2008).”
BackgroundResultInfluentialKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · cited 4×
“The impact of bias from selective outcome reporting on the conclusions of Cochrane systematic reviews has also recently been described (Kirkham et al., 2010).”
BackgroundInfluentialMeta-analysis Spurious precision? Meta-analysis of observational studies
Egger, M., Schneider, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7125.140 · Cited by 997
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · cited 1×
“A systematic review of data from five inception cohorts (registered clinical trials) found that studies with ‘positive’ findings were about twice as likely to be published as studies with ‘negative’ or ‘null’ results (Hopewell et al., 2009).”
BackgroundMacLehose, Reeves, Harvey, Sheldon, Russell, & Black (2000) · Health Technology Assessment · doi:10.3310/hta4340 · Cited by 311
Schünemann, H. J., Tugwell, P., Reeves, B. C., Akl, E. A., Santesso, N., Spencer, F. A., Shea, B., Wells, G., & Helfand, M. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1078 · Cited by 248 · cited 2×
“Schünemann et al. (2013) describe how such a judgement can be made more systematically.”
BackgroundSelf-citation
Research works printed but not linked: 16
- Concato J, 1992, Problems of comorbidity in mortality after prostatectomy, JAMA: The Journal of the American Medical Association, 267, 1077, 10.1001/jama.1992.03480080047025 · DOI · Paper page
- 10.1136/bmj.312.7033.760 · DOI · Paper page
- 10.1371/journal.pmed.1001026 · DOI · Paper page
- Haynes RB, 2005, Clinical Epidemiology: How to Do Clinical Practice Research, 11
- 10.1002/9780470712184.ch4 · DOI · Paper page
- 10.1002/9780470712184.ch14 · DOI
- National Health and Medical Research Council, 1999, A guide to the development, implementation and evaluation of clinical practice guidelines
- 10.1002/jrsm.1062 · DOI · Paper page
- Oxford Centre for Evidence‐based Medicine.Levels of Evidence.2001.http://www.cebm.net/index.aspx?o=1047Accessed 8 May 2012.
- Oxman AD, 1994, The Cochrane Collaboration Tool Kit, Section VI: Preparing and maintaining systematic reviews
- PalmerDA MacLennanSJ ImamuraM ReevesBC ClubbA DuboyAJ NeubergerMM DahmP.2011.Initial experience with a pilot Cochrane tool for assessing risk of bias for non‐randomized studies applying a web‐based survey of content experts to derive criteria for imbalance. B804. Cochrane Database of Systematic Reviews Supplement 2011: Art No.: CD000003. DOI:10.1002/14651858. · doi:10.1002/14651858
- 10.1002/9780470712184.ch13 · DOI · Paper page
- 10.1002/9780470712184.ch11 · DOI · Paper page
- 10.1016/S1473-3099(05)70024-4 · DOI
- 10.1016/0091-7435(91)90006-P · DOI · Paper page
- 10.1093/ije/dyh191 · DOI
Cited by
Anglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534
Incidence and Prevalence of Drug-Resistant Epilepsy: A Systematic Review and Meta-analysis
Sultana, B., Panzini, M.-A., Veilleux Carpentier, A., Comtois, J., Rioux, B., Gore, G., Bauer, P. R., Kwon, C.-S., Jetté, N., Josephson, C. B., et al. (2021) · Neurology · doi:10.1212/wnl.0000000000011839 · Cited by 435
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348
“Research Synthesis Methods published by John Wiley & Sons, Ltd. Keywords: meta-analysis; IPD; evidence synthesis; review; RCT; non-randomized intervention studies; NRSI; cross-design”
BackgroundResultInfluentialSchünemann, H. J., Tugwell, P., Reeves, B. C., Akl, E. A., Santesso, N., Spencer, F. A., Shea, B., Wells, G., & Helfand, M. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1078 · Cited by 248
“Defining the PICO elements at the outset of a review (e.g., male and female, urban and rural, less and more developed country settings) provides an explicit framework within which to ‘consider NRS’ and ensures patient centeredness (Reeves et al., 2013).”
BackgroundSelf-citationCarroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235
Combining randomized and non-randomized evidence in network meta-analysis
Efthimiou, O., Mavridis, D., Debray, T. P. A., Samara, M., Belger, M., Siontis, G. C. M., Leucht, S., Salanti, G., & 4, O. B. O. G. W. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7223 · Cited by 183
Wells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1077 · Cited by 141
Self-citationVerde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123
“The interest of including non-randomized studies in evidence synthesis has recently been highlighted in a special issue of this journal, where the authors presented the outcomes of a special workshop led by the NonRandomized Studies Methods Group of the Cochrane Collaboration (Reeves et al., 2013).”
BackgroundBasic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108
“’’ We define non-RCTs as ‘‘any quantitative trial estimating the effects of an intervention that does not use randomization to allocate units to comparison groups’’ [18].”
BackgroundInfluentialToews, I., Anglemyer, A., Nyirenda, J. L., Alsaid, D., Balduzzi, S., Grummich, K., Schwingshackl, L., & Bero, L. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub3 · Cited by 60
Meta-analyses including data from observational studies
O’Connor, A. M., & Sargeant, J. M. (2014) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.10.017 · Cited by 38
Althuis, M. D., Weed, D. L., & Frankenfeld, C. L. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-80 · Cited by 30
“In 2013, two journals focusing on research synthesis methods (Systematic Reviews and Research Synthesis Methods) emphasized the importance of qualitative evaluation of studies selected for meta-analysis, calling for more strategies to aid conduct and reporting [12,13].”
Background
Metadata
- Authors
- Reeves, B. C., Higgins, J. P. T., Ramsay, C., Shea, B., Tugwell, P., & Wells, G. A.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2013-01-24
- Volume
- 4
- Issue
- 1
- Pages
- 1-11
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W1484962366
- Cited by
- 112
- References
- 15
- FWCI
- 3.18
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.