Using Multiple Types of Studies in Systematic Reviews of Health Care Interventions – A Systematic Review
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
BACKGROUND: A systematic review may evaluate different aspects of a health care intervention. To accommodate the evaluation of various research questions, the inclusion of more than one study design may be necessary. One aim of this study is to find and describe articles on methodological issues concerning the incorporation of multiple types of study designs in systematic reviews on health care interventions. Another aim is to evaluate methods studies that have assessed whether reported effects differ by study types. METHODS AND FINDINGS: We searched PubMed, the Cochrane Database of Systematic Reviews, and the Cochrane Methodology Register on 31 March 2012 and identified 42 articles that reported on the integration of single or multiple study designs in systematic reviews. We summarized the contents of the articles qualitatively and assessed theoretical and empirical evidence. We found that many examples of reviews incorporating multiple types of studies exist and that every study design can serve a specific purpose. The clinical questions of a systematic review determine the types of design that are necessary or sufficient to provide the best possible answers. In a second independent search, we identified 49 studies, 31 systematic reviews and 18 trials that compared the effect sizes between randomized and nonrandomized controlled trials, which were statistically different in 35%, and not different in 53%. Twelve percent of studies reported both, different and non-different effect sizes. CONCLUSIONS: Different study designs addressing the same question yielded varying results, with differences in about half of all examples. The risk of presenting uncertain results without knowing for sure the direction and magnitude of the effect holds true for both nonrandomized and randomized controlled trials. The integration of multiple study designs in systematic reviews is required if patients should be informed on the many facets of patient relevant issues of health care interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 1×
“The Cochrane Collaboration offers a guide for inclusion of nonrandomized studies [59] and it has developed a tool for assessing the risk of bias in both RCT and controlled nonrandomized studies[60].”
MethodEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · cited 1×
“Cheng 2012 [71] same no Choi 2012 [72] same no Clagett 1984 [73] same no Colditz 1989 [74] various yes Conaty 2004 [75] same no Concato 2000 [76] same no Deeks 2003 [77] various various Edwards 2012 [78] various no Flossman 2007 [79] same no Franklin 2000 [80] same no Furlan (Exam) 2008 [81] same various Furlan (Meth)…”
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347
Why we need observational studies to evaluate the effectiveness of health care
Black, N. (1996) · BMJ · doi:10.1136/bmj.312.7040.1215 · Cited by 1,598 · cited 2×
“Black 1996 [16] Various 1 1 Chumbler 2008 [21] Tele 1 1 1 1 Concato (Comp) 2010 [7] Surgery 1 1 1 1 Concato (Observ) 2010 [22] Various 1 1 1 1 Essock 2003 [23] Mental 1 1 Fletcher 2002 [24] Various 1 1 1 1 Fletcher 2009 [25] HRT 1 1 1 Gale 2009 [26] Transplant 1 1 Grzeskowiak 2012 [27] Pregnancy 1 1 1 1 1 Hadley 2009…”
BackgroundContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Evidence-Based Public Health: Moving Beyond Randomized Trials
Victora, C. G., Habicht, J.-P., & Bryce, J. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.400 · Cited by 1,056 · cited 1×
“In some situations RCTs are not feasible due to ethical concerns or due to strong patients' preferences and the results may not be applicable to everyday practice [3].”
BackgroundSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919
Flossmann, E., & Rothwell, P. M. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60747-8 · Cited by 820
Cited by
Kennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0925-0 · Cited by 212
Prado, C. E., Watt, S., & Crowe, S. F. (2018) · Neuropsychology Review · doi:10.1007/s11065-018-9369-5 · Cited by 188
Phillips, E. A., Gordeev, V. S., & Schreyögg, J. (2019) · Scandinavian Journal of Work, Environment & Health · doi:10.5271/sjweh.3839 · Cited by 148
“Studies with other designs would not be fully comparable resulting in biased results of the meta-analysis, which additionally would have been difficult to interpret (30, 31).”
Toews, I., Anglemyer, A., Nyirenda, J. L., Alsaid, D., Balduzzi, S., Grummich, K., Schwingshackl, L., & Bero, L. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub3 · Cited by 60
Corticosteroids and cognition: a meta-analysis
Prado, C. E., & Crowe, S. F. (2019) · Neuropsychology Review · doi:10.1007/s11065-019-09405-8 · Cited by 58
Metadata
- Authors
- Peinemann, F., Tushabe, D. A., & Kleijnen, J.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-12-26
- Volume
- 8
- Issue
- 12
- Pages
- e85035
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2075910843
- Cited by
- 89
- References
- 22
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using Multiple Types of Studies in Systematic Reviews of Health Care Interventions – A Systematic Review. https://app.cochat.ai/papers/10.1371/journal.pone.0085035
- The paper
- Peinemann, F., Tushabe, D. A., & Kleijnen, J. (2013). Using Multiple Types of Studies in Systematic Reviews of Health Care Interventions – A Systematic Review. PLoS ONE. https://doi.org/10.1371/journal.pone.0085035
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.