The DOI matches its Crossref record; 39 of the 41 references it cites were found in a registry or an index.
Issues relating to confounding and meta‐analysis when including non‐randomized studies in systematic reviews on the effects of interventions
Abstract
Background: Confounding caused by selection bias is often a key difference between non‐randomized studies (NRS) and randomized controlled trials (RCTs) of interventions. Key methodological issues: In this third paper of the series, we consider issues relating to the inclusion of NRS in systematic reviews on the effects of interventions. We discuss whether potential biases from confounding in NRS can be accounted for, the limitations of current methods for attempting to do so, the different contexts of NRS and RCTs, the problems these issues create for reviewers, and a research agenda for the future. Guidance: Reviewers who are considering whether or not to include NRS in meta‐analyses must weigh a number of factors. Including NRS may allow a review to address outcomes or pragmatic implementations of an intervention not studied in RCTs, but it will also increase the workload for the review team, as well as their required technical repertoire. Furthermore, the results of a synthesis involving NRS will likely be more difficult to interpret, and less certain, relative to the results of a synthesis involving only randomized studies. When both randomized and non‐randomized evidence are available, we favor a strategy of including NRS and RCTs in the same systematic review but synthesizing their results separately. Conclusion: Including NRS will often make the limitations of the evidence derived from RCTs more apparent, thereby guiding inferences about generalizability, and may help with the design of the next generation of RCTs. Copyright © 2012 John Wiley & Sons, Ltd.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Workload, which is generally higher when observational studies are included, was also mentioned as a factor in the decision of which study designs to include [75].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
In the citing paper
“Authors agreed that combining different study designs in a meta-analysis can lead to misleading results if done without careful consideration [35, 38, 56, 70, 75, 76].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
In the citing paper
“Recommendations were made mainly on 2 topics: whether different study designs should be included in a systematic review; [29, 34, 37–39, 49, 56, 65, 73, 75] and whether different study designs should be analysed together in a single meta-analysis [8, 34, 35, 37, 38, 42, 49, 51, 56, 61, 70, 75–78].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
In the citing paper
“Five articles discouraged combining different study designs in meta-analyses but saw the added value of including different designs in a systematic review [35, 38, 56, 61, 75].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
In the citing paper
“Most articles considered the combination of RCTs and non-randomised designs [29, 35, 38, 39, 49, 51, 56, 61, 75, 81].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
In the citing paper
“Only five articles referred to the combination of different non-randomised studies [38, 70, 75, 79, 80].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
In the citing paper
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
(1963) · Medical Entomology and Zoology · Cited by 15,085
(2002) · Cited by 13,211
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Regression-discontinuity analysis: An alternative to the ex post facto experiment.
Thistlethwaite, D. L., & Campbell, D. T. (1960) · Journal of Educational Psychology · doi:10.1037/h0044319 · Cited by 1,469
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294
Self-citationFitzpatrick, R. M., Shortall, Sculpher, Murray, Morris, Lodge, Dawson, Carr, Britton, & Briggs, A. H. (1998) · Health Technology Assessment · doi:10.3310/hta2200 · Cited by 969
Flagged first, then by citations
2 of the 41 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Cochrane Non‐Randomized Studies Methods Group.2010.Classifying non‐randomised studies (NRS) and the assessing the risk of bias for a systematic review. 18th Cochrane Colloquium Keystone Colorado.
- Reeves BC, 2009, Cochrane Handbook for Systematic Reviews of Interventions
Cited by
Sorted by citations
Items 101–129 of 129
Li, Q., Deng, K., Jiang, X., Tao, H., Liu, H., & Chen, J. (2019) · doi:10.21203/rs.2.12469/v1 · Cited by 3
Emergency cricothyroidotomy for airway management
Andersson, M. L., Møller, A. M., & Pace, N. L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010921.pub2 · Cited by 2
Zhou, X., Zou, N., Shi, C., Chen, S., & Shi, J. (2026) · Aging & Mental Health · doi:10.1080/13607863.2025.2573071 · Cited by 1
Elevating the importance of Risk of Bias assessment for ecology and evolutionary biology
Čulina, A., Foster, D., Grainger, M., O’Dea, R. E., Pescott, O. L., Sánchez‐Tójar, A., Boyd, R. J., Koricheva, J., Nakagawa, S., & Stewart, G. B. (2026) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70356 · Cited by 4
Oremus, M., Girouard, H., & Raina, P. (2016) · Springer eBooks · doi:10.1007/978-3-319-25616-0_1 · Cited by 1
Pérez, S. G., Díaz‐Brochero, C., Herazo, J. R. G., & Múñoz, Ó. (2024) · Therapeutic Advances in Infectious Disease · doi:10.1177/20499361241237615 · Cited by 1
The perils of conducting meta‐analyses of observational data
Harling, G., & Subramanian, S. V. (2014) · Journal of the International AIDS Society · doi:10.7448/ias.17.1.19112 · Cited by 1
Akshintala, V. S., & Kalloo, A. N. (2019) · Digestive Diseases and Sciences · doi:10.1007/s10620-019-05732-1 · Cited by 1
Emergency cricothyroidotomy for airway management
Andersson, M. L., Møller, A. M., & Pace, N. L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010921 · Cited by 1
RetractedBauer, S. R., Wieruszewski, P. M., Bissell, B. D., Dugar, S. P., Sacha, G. L., Sato, R., Siuba, M., Schleicher, M. C., Vachharajani, V., Falck-Ytter, Y. T., et al. (2023) · medRxiv · doi:10.1101/2023.07.29.23293364 · Cited by 1
Elevating the importance of Risk of Bias assessment for ecology and evolutionPreprint
Čulina, A., Foster, D., Grainger, M., O’Dea, R. E., Sánchez‐Tójar, A., Pescott, O. L., Boyd, R., Koricheva, J., Nakagawa, S., & Stewart, G. B. (2025) · doi:10.32942/x2764c · Cited by 1
Letter to the Editor The perils of conducting meta-analyses of observational data
Harling, G., & Subramanian, S. V. (2014)
Statistical methods for evidence synthesisPreprint
Mathur, M. B., & VanderWeele, T. J. (2018) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/kd6ja
Ponce, M. F. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3164259
Gong, L., Dong, C., Cai, Q., & Ouyang, W. (2020) · Research Square · doi:10.21203/rs.2.116/v2
Gong, L., Dong, C., Cai, Q., & Ouyang, W. (2018) · Research Square · doi:10.21203/rs.2.116/v1
Natural killer cell profiling in women with recurrent pregnancy loss
Habets, D. H. J. (2022) · doi:10.26481/dis.20220915dh
Jay, M., Andrei, S., Hoang, P., Samhat, H., Jones, R. M., Fu, R., Lipscombe, L. L., & Eskander, A. E. (2025) · medRxiv · doi:10.1101/2025.01.22.25320716
van Leeuwen, H., Sinnaeve, R., Witteveen, U., Van Daele, T., Ossewaarde, L., Egger, J. I. M., & van den Bosch, L. M. C. (2021) · Figshare · doi:10.6084/m9.figshare.16909806
van Leeuwen, H., Sinnaeve, R., Witteveen, U., Van Daele, T., Ossewaarde, L., Egger, J. I. M., & van den Bosch, L. M. C. (2021) · Figshare · doi:10.6084/m9.figshare.16909800
van Leeuwen, H., Sinnaeve, R., Witteveen, U., Van Daele, T., Ossewaarde, L., Egger, J. I. M., & van den Bosch, L. M. C. (2021) · Figshare · doi:10.6084/m9.figshare.16909803
Cheurfa, C., Tsokani, S., Kontouli, K.-M., Boutron, I., & Chaimani, A. (2024) · Figshare · doi:10.6084/m9.figshare.26683834
Cheurfa, C., Tsokani, S., Kontouli, K.-M., Boutron, I., & Chaimani, A. (2024) · Figshare · doi:10.6084/m9.figshare.26683831
Moran, J. L., & Linden, A. (2024) · Figshare · doi:10.6084/m9.figshare.25708964
Cheurfa, C., Tsokani, S., Kontouli, K.-M., Boutron, I., & Chaimani, A. (2024) · Figshare · doi:10.6084/m9.figshare.26683837
Yan, R., Liu, T., Peng, Y., & Peng, X. (2020) · Figshare · doi:10.6084/m9.figshare.13467205
Cheurfa, C., Tsokani, S., Kontouli, K.-M., Boutron, I., & Chaimani, A. (2024) · Figshare · doi:10.6084/m9.figshare.26683828
Yan, R., Liu, T., Peng, Y., & Peng, X. (2020) · Figshare · doi:10.6084/m9.figshare.13467207
Contribution des études observationnelles dans la méta-analyse
Cheurfa, C. (2023) · theses.fr (ABES) · doi:10.70675/ecaa782bz76f7z4c43za043ze1d3e0cfbfaa
Metadata
- Authors
- Valentine, J. C., & Thompson, S. G.
- Affiliations
- University of Louisville (US)
- University of Cambridge (GB)
- Corresponding author
- Valentine, Jeffrey C.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2012-11-06
- Volume
- 4
- Issue
- 1
- Pages
- 26-35
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W1918813775
- Cited by
- 130
- References
- 41
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 14 · 2023: 6 · 2022: 10 · 2021: 11 · 2020: 15 · 2019: 11 · 2018: 10 · 2017: 15 · 2016: 5 · 2015: 8 · 2014: 10 · 2013: 5 · 2012: 2
- FWCI
- 2.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute for Health and Care Research (NF-SI-0512-10165)
- British Heart Foundation (RG/08/014/24067)
- Medical Research Council (MR/L003120/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- confounding, selection bias, non-randomized studies, meta-analysis, systematic reviews, randomized controlled trials, generalizability, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
