The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Issues relating to study design and risk of bias when including non‐randomized studies in systematic reviews on the effects of interventions
Abstract
Non‐randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long‐term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non‐randomized studies. In this second paper in a series, we address how review authors might articulate the particular non‐randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non‐randomized study is at risk of important biases. We offer guidance for describing and classifying different non‐randomized designs based on specific features of the studies in place of using non‐informative study design labels. We also suggest criteria to consider when deciding whether to include non‐randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non‐randomized designs that involve a different inferential logic compared with parallel group trials. Copyright © 2012 John Wiley & Sons, Ltd.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The non-randomised study was scored as high risk for 'random sequence generation' and 'allocation concealment' while grading for other items was similar to those of randomised trial.41”
From Mindfulness-based interventions for psychological wellbeing and quality of life in men with prostate cancer: A systematic review and meta-analysis (2021) · cites this paper as [41]
In the citing paper
“About one third of the articles acknowledged that searching for observational studies requires additional time and resources because of lack of specific search filters and poorly established or inconsistently used indexing terms [38, 39, 47, 49, 56, 58, 59, 64, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Many recommended that the eligibility criteria need to be pre-specified [30, 33, 38, 39, 47, 55, 65] and that the rationale for defining the eligibility criteria should be explicitly justified [38, 39, 65], unambiguous [65], and derived from the review question [47].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Two recent articles advocated deciding on a case-by-case basis [37, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Recommendations were made mainly on 2 topics: whether different study designs should be included in a systematic review; [29, 34, 37–39, 49, 56, 65, 73, 75] and whether different study designs should be analysed together in a single meta-analysis [8, 34, 35, 37, 38, 42, 49, 51, 56, 61, 70, 75–78].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Citations
Cites
(2002) · Cited by 13,211
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Self-citationEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Self-citationRegression-discontinuity analysis: An alternative to the ex post facto experiment.
Thistlethwaite, D. L., & Campbell, D. T. (1960) · Journal of Educational Psychology · doi:10.1037/h0044319 · Cited by 1,469
Ukoumunne, Gulliford, M. C., Chinn, Sterne, & Burney, P. (1999) · Health Technology Assessment · doi:10.3310/hta3050 · Cited by 1,211
Clinical Epidemiology: The Architecture of Clinical Research.
(1985) · Annals of Internal Medicine · doi:10.7326/0003-4819-103-4-649_2 · Cited by 954
Research works printed but not linked: 3
- Cochrane Effective Practice and Organisation of Care Group.2002.Data collection checklist.http://epoc.cochrane.org/epoc‐resources‐review‐authors(accessed 12 August 2011).
- Norris SL, Issues relating to selective reporting when including non‐randomized studies in systematic reviews on the effects of interventions, Research Synthesis Methods
- Oxford Centre for Evidence‐based Medicine.2001.Levels of evidence[May 2001].http://www.cebm.net/index.aspx?o=1047. 1‐1‐2008
Cited by
Sorted by citations
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Interventions to Reduce Ageism Against Older Adults: A Systematic Review and Meta-Analysis
Burnes, D. P. R., Sheppard, C. L., Henderson, C., Wassel, M., Cope, R., Barber, C., & Pillemer, K. A. (2019) · American Journal of Public Health · doi:10.2105/ajph.2019.305123 · Cited by 838
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradicto…”
BackgroundInfluentialGet real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348
“First, researchers should critically appraise the risk of bias in available studies prior to cross-design synthesis and carefully decide whether inclusion is appropriate (Berger et al., 2014; Higgins et al., 2013; Reeves et al., 2003; Wells et al., 2013).”
BackgroundRauch, B., Davos, C. H., Doherty, P., Saure, D., Metzendorf, M., Salzwedel, A., Völler, H., Jensen, K., & Schmid, J. (2016) · European Journal of Preventive Cardiology · doi:10.1177/2047487316671181 · Cited by 318
Metadata
- Authors
- Higgins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G.
- Affiliations
- MRC Biostatistics Unit (GB)
- University of Aberdeen (GB)
- Bristol Royal Infirmary (GB)
- University of Birmingham (GB)
- University of Ottawa (CA)
- University of Louisville (US)
- Corresponding author
- Higgins, Julian PT
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2012-09-25
- Volume
- 4
- Issue
- 1
- Pages
- 12-25
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2062040186
- Cited by
- 183
- References
- 31
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 17 · 2023: 23 · 2022: 8 · 2021: 17 · 2020: 18 · 2019: 21 · 2018: 10 · 2017: 16 · 2016: 11 · 2015: 11 · 2014: 11 · 2013: 9 · 2012: 1
- FWCI
- 2.82
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- Scottish Government
- National Institute for Health and Care Research
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- non-randomized studies, risk of bias, study design classification, intervention effects, observational study design, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
