Issues relating to confounding and meta‐analysis when including non‐randomized studies in systematic reviews on the effects of interventions
Abstract
Background: Confounding caused by selection bias is often a key difference between non‐randomized studies (NRS) and randomized controlled trials (RCTs) of interventions. Key methodological issues: In this third paper of the series, we consider issues relating to the inclusion of NRS in systematic reviews on the effects of interventions. We discuss whether potential biases from confounding in NRS can be accounted for, the limitations of current methods for attempting to do so, the different contexts of NRS and RCTs, the problems these issues create for reviewers, and a research agenda for the future. Guidance: Reviewers who are considering whether or not to include NRS in meta‐analyses must weigh a number of factors. Including NRS may allow a review to address outcomes or pragmatic implementations of an intervention not studied in RCTs, but it will also increase the workload for the review team, as well as their required technical repertoire. Furthermore, the results of a synthesis involving NRS will likely be more difficult to interpret, and less certain, relative to the results of a synthesis involving only randomized studies. When both randomized and non‐randomized evidence are available, we favor a strategy of including NRS and RCTs in the same systematic review but synthesizing their results separately. Conclusion: Including NRS will often make the limitations of the evidence derived from RCTs more apparent, thereby guiding inferences about generalizability, and may help with the design of the next generation of RCTs. Copyright © 2012 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Workload, which is generally higher when observational studies are included, was mentioned as a factor in the decision of which study designs to include.
“Workload, which is generally higher when observational studies are included, was also mentioned as a factor in the decision of which study designs to include [75].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Authors agreed that combining different study designs in a meta-analysis can lead to misleading results if done without careful consideration.
“Authors agreed that combining different study designs in a meta-analysis can lead to misleading results if done without careful consideration [35, 38, 56, 70, 75, 76].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Recommendations were made mainly on two topics: whether different study designs should be included in a systematic review; and whether different study designs should be analysed together in a single meta-analysis.
“Recommendations were made mainly on 2 topics: whether different study designs should be included in a systematic review; [29, 34, 37–39, 49, 56, 65, 73, 75] and whether different study designs should be analysed together in a single meta-analysis [8, 34, 35, 37, 38, 42, 49, 51, 56, 61, 70, 75–78].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Five articles discouraged combining different study designs in meta-analyses but saw the added value of including different designs in a systematic review.
“Five articles discouraged combining different study designs in meta-analyses but saw the added value of including different designs in a systematic review [35, 38, 56, 61, 75].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Most articles considered the combination of RCTs and non-randomised designs.
“Most articles considered the combination of RCTs and non-randomised designs [29, 35, 38, 39, 49, 51, 56, 61, 75, 81].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Only five articles referred to the combination of different non-randomised studies.
“Only five articles referred to the combination of different non-randomised studies [38, 70, 75, 79, 80].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Many authoritative articles providing sound advice exist, such as those by Greenland, Egger et al., and a series in Research Synthesis Methods.
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [75]
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
(2002) · Cited by 13,211 · see citing works
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · see citing works
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230 · see citing works
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Regression-discontinuity analysis: An alternative to the ex post facto experiment.
Thistlethwaite, D. L., & Campbell, D. T. (1960) · Journal of Educational Psychology · doi:10.1037/h0044319 · Cited by 1,469 · see citing works
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294 · see citing works
Self-citationFitzpatrick, R. M., Shortall, Sculpher, Murray, Morris, Lodge, Dawson, Carr, Britton, & Briggs, A. H. (1998) · Health Technology Assessment · doi:10.3310/hta2200 · Cited by 969 · see citing works
Analysis of Observational Studies in the Presence of Treatment Selection Bias
Stukel, T. A., Fisher, E. S., Wennberg, D. E., Alter, D. A., Gottlieb, D. J., & Vermeulen, M. J. (2007) · JAMA · doi:10.1001/jama.297.3.278 · Cited by 789 · see citing works
Hume, D. (2011) · University of Michigan Library Repository · Cited by 633 · see citing works
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604 · see citing works
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479 · see citing works
Choosing between randomised and non-randomised studies: a systematic review.
Britton, McKee, M., Black, McPherson, Sanderson, C. F. B., & Bain (1998) · Health Technology Assessment · doi:10.3310/hta2130 · Cited by 330 · see citing works
Bias Modelling in Evidence Synthesis
Turner, R., Spiegelhalter, D. J., Smith, G. C. S., & Thompson, S. G. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00547.x · Cited by 322 · see citing works
Self-citationMacLehose, Reeves, Harvey, Sheldon, Russell, & Black (2000) · Health Technology Assessment · doi:10.3310/hta4340 · Cited by 311 · see citing works
Golder, S., Loke, Y. K., & Bland, M. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1001026 · Cited by 270 · see citing works
Meta-Analysis in Medicine and Health Policy
Stangl, D. K. (2000) · CRC Press eBooks · doi:10.1201/9780203909935 · Cited by 213 · see citing works
Systematic Reviews of Observational Studies
Egger, M., Smith, G. D., & Schneider, M. (2001) · Wiley eBooks · doi:10.1002/9780470693926.ch12 · Cited by 211 · see citing works
Meta-Analysis for Explanation: A Casebook
Spilker, B. (1993) · JAMA · doi:10.1001/jama.1993.03500230123042 · Cited by 199 · see citing works
Research design for program evaluation: The regression-discontinuity approach
Spiegelman, C. H. (1985) · Evaluation and Program Planning · doi:10.1016/0149-7189(85)90035-7 · Cited by 188 · see citing works
Does Alcoholics Anonymous Work? The Results from a Meta-Analysis of Controlled Experiments
Kownacki, R., & Shadish, W. R. (1999) · Substance Use & Misuse · doi:10.3109/10826089909039431 · Cited by 184 · see citing works
Valentine, J. C., & Cooper, H. M. (2008) · Psychological Methods · doi:10.1037/1082-989x.13.2.130 · Cited by 183 · see citing works
Self-citationRobust, Accurate Confidence Intervals with a Weak Instrument: Quarter of Birth and Education
Imbens, G. W., & Rosenbaum, P. R. (2004) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.00339.x · Cited by 171 · see citing works
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166 · see citing works
Self-citationThe Way in Which Intervention Studies Have "Personality" and Why It Is Important to Meta-Analysis
Lipsey, M. W., & Wilson, D. B. (2001) · Evaluation & the Health Professions · doi:10.1177/016327870102400302 · Cited by 152 · see citing works
Petrosino, A., & Soydan, H. (2005) · Journal of Experimental Criminology · doi:10.1007/s11292-005-3540-8 · Cited by 152 · see citing works
Models for Potentially Biased Evidence in Meta-Analysis using Empirically Based Priors
Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G., & Sterne, J. A. C. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00548.x · Cited by 150 · see citing works
A proposed method of bias adjustment for meta-analyses of published observational studies
Thompson, S., Ekelund, U., Jebb, S., Lindroos, A. K., Mander, A., Sharp, S., Turner, R., & Wilks, D. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyq248 · Cited by 101 · see citing works
Self-citationThe Fibrinogen Studies Collaboration (2009) · Statistics in Medicine · doi:10.1002/sim.3540 · Cited by 56 · see citing works
Loke, Y. K., Golder, S. P., & Vandenbroucke, J. P. (2011) · Therapeutic Advances in Drug Safety · doi:10.1177/2042098611401129 · Cited by 49 · see citing works
MacLehose, R., Reeves, B., Harvey, I. M., National Coordinating Centre for Health Technology Assessment, S. (. K., & NHS R and D HTA Programme, S. (. K. (2000) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 32 · see citing works
Development of a combined database for meta‐epidemiological research
Savović, J., Harris, R., Wood, L., Beynon, R., Altman, D. G., Als-Nielsen, B. E. T., Balk, E. M., Deeks, J. J., Gluud, L. L., Gluud, C. N., et al. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.18 · Cited by 25 · see citing works
Pesaran, M. H., Bowden, R. J. J., & Turkington, D. A. (1986) · Economica · doi:10.2307/2553961 · Cited by 24 · see citing works
Bounds on potential risks and causal risk differences under assumptions about confounding parameters
Chiba, Y., Sato, T. S., & Greenland, S. (2007) · Statistics in Medicine · doi:10.1002/sim.2927 · Cited by 17 · see citing works
Structural Approach to Bias in Meta‐analyses
Shrier, I. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.52 · Cited by 13 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsey, C., Tugwell, P., & Wells, G. A. (2013) · Bristol Research (University of Bristol) · Cited by 2 · see citing works
D’Agostino (1998) · doi:10.1002/(sici)1097-0258(19981015)17:19
Flagged first, then by citations
2 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Interventions to Reduce Ageism Against Older Adults: A Systematic Review and Meta-Analysis
Burnes, D. P. R., Sheppard, C. L., Henderson, C., Wassel, M., Cope, R., Barber, C., & Pillemer, K. A. (2019) · American Journal of Public Health · doi:10.2105/ajph.2019.305123 · Cited by 733 · see citing works
Incidence and Prevalence of Drug-Resistant Epilepsy
Sultana, B., Panzini, M.-A., Carpentier, A. V., Comtois, J., Rioux, B., Gore, G. C., Bauer, P. R., Kwon, C., Jetté, N., Josephson, C. B., et al. (2021) · Neurology · doi:10.1212/wnl.0000000000011839 · Cited by 442 · see citing works
Music therapy for stress reduction: a systematic review and meta-analysis
de Witte, M., Pinho, A. D. S., Stams, G.-J., Moonen, X., Bos, A. E. R., & van Hooren, S. (2022) · Health Psychology Review · doi:10.1080/17437199.2020.1846580 · Cited by 444 · see citing works
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348 · see citing works
Defining the criteria for including studies and how they will be grouped for the synthesis
McKenzie, J. E., Brennan, S., Ryan, R. E., Thomson, H., Johnston, R. V., & Thomas, J. (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch3 · Cited by 378 · see citing works
Liebowitz, D. D., & Porter, L. (2019) · Review of Educational Research · doi:10.3102/0034654319866133 · Cited by 320 · see citing works
Reeves, B. C., Wells, G. A., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.016 · Cited by 199 · see citing works
Higgins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1056 · Cited by 183 · see citing works
Self-citationClough, B. A., March, S., Chan, R. J., Casey, L. M., Phillips, R., & Ireland, M. J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0526-3 · Cited by 186 · see citing works
Amplification with hearing aids for patients with tinnitus and co-existing hearing loss.
Hoare, D. J., Edmondson-Jones, M., Sereda, M., Akeroyd, M. A., & Hall, D. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010151.pub2 · Cited by 170 · see citing works
Combining randomized and non-randomized evidence in network meta-analysis
Efthimiou, O., Mavridis, D., Debray, T. P. A., Samara, M., Belger, M., Siontis, G. C. M., Leucht, S., Salanti, G., & 4, O. B. O. G. W. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7223 · Cited by 183 · see citing works
Sensitivity Analysis for Unmeasured Confounding in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2018.1529598 · Cited by 154 · see citing works
Wells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1077 · Cited by 141 · see citing works
Outcome-Reporting Bias in Education Research
Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D. (2013) · Educational Researcher · doi:10.3102/0013189x13507104 · Cited by 127 · see citing works
Self-citationVerde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 112 · see citing works
Risk factors for diabetic foot complications in type 2 diabetes—A systematic review
Rossboth, S., Lechleitner, M., & Oberaigner, W. (2020) · Endocrinology Diabetes & Metabolism · doi:10.1002/edm2.175 · Cited by 109 · see citing works
Basic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108 · see citing works
Vézina-Im, L.-A., Beaulieu, D., Bélanger-Gravel, A., Boucher, D., Sirois, C., Dugas, M., & Provencher, V. (2017) · Public Health Nutrition · doi:10.1017/s1368980017000076 · Cited by 106 · see citing works
Complications and Patient-Reported Outcomes in Male-to-Female Vaginoplasty—Where We Are Today
Manrique, O. J., Adabi, K., Martinez‐Jorge, J., Ciudad, P., Nicoli, F., & Kiranantawat, K. (2018) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000001393 · Cited by 103 · see citing works
Acceptance and commitment therapy for family caregivers: A systematic review and meta-analysis.
Han, A., Yuen, H. K., & Jenkins, J. (2021) · Journal of Health Psychology · doi:10.1177/1359105320941217 · Cited by 82 · see citing works
Ye, Y., Yang, M., Zhang, S., & Zeng, Y. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179756 · Cited by 87 · see citing works
Valentine, J. C., Konstantopoulos, S., & Goldrick‐Rab, S. (2017) · Review of Educational Research · doi:10.3102/0034654317709237 · Cited by 81 · see citing works
Self-citationSeptoplasty for nasal obstruction due to a deviated nasal septum in adults: a systematic review
van Egmond, M. M. H. T., Rovers, M. M., Tillema, A. H. J., & van Heerbeek, N. (2018) · Rhinology Journal · doi:10.4193/rhin18.016 · Cited by 82 · see citing works
Bias from historical control groups used in orthodontic research: a meta-epidemiological study
Papageorgiou, S. N., Koretsi, V., & Jäger, A. (2016) · European Journal of Orthodontics · doi:10.1093/ejo/cjw035 · Cited by 80 · see citing works
Effects of shade trees on robusta coffee growth, yield and quality. A meta-analysis
Piato, K., Lefort, F., Subía, C., Caicedo, C., Calderón, D., Pico, J. T., & Norgrove, L. (2020) · Agronomy for Sustainable Development · doi:10.1007/s13593-020-00642-3 · Cited by 74 · see citing works
Arditi, C., Burnand, B., & Peytremann‐Bridevaux, I. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1816-5 · Cited by 68 · see citing works
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 62 · see citing works
Baumeister, S. E., Leitzmann, M. F., Linseisen, J., & Schlesinger, S. (2019) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djz111 · Cited by 58 · see citing works
The comorbid relationship between migraine and epilepsy: a systematic review and meta‐analysis
Keezer, M. R., Bauer, P. R., Ferrari, M. D., & Sander, J. W. A. S. (2014) · European Journal of Neurology · doi:10.1111/ene.12612 · Cited by 54 · see citing works
Interventions to enhance maternal awareness of decreased fetal movement: a systematic review
Winje, B., Wojcieszek, A. M., Gonzalez‐Angulo, L., Teoh, Z., Norman, J. E., Frøen, J. F., & Flenady, V. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13802 · Cited by 49 · see citing works
Li, J., Xu, R., Hu, D., Zhang, Y., Gong, T.-P., & Wu, X. (2019) · Nutrition and Cancer · doi:10.1080/01635581.2019.1577986 · Cited by 46 · see citing works
Systematic Reviews: A Social Work Perspective
Crisp, B. R. (2015) · Australian Social Work · doi:10.1080/0312407x.2015.1024266 · Cited by 44 · see citing works
Disconnected by design: analytic approach in treatment networks having no common comparator
Goring, S. M., Gustafson, P., Liu, Y., Saab, S., Cline, S. K., & Platt, R. W. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1204 · Cited by 42 · see citing works
Lock, M., Post, D., Dollman, J., & Parfitt, G. (2020) · Health Psychology Review · doi:10.1080/17437199.2020.1718528 · Cited by 42 · see citing works
Ijaz, S., Verbeek, J. H. A. M., Tikka, C., & Ruotsalainen, J. H. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.01.001 · Cited by 41 · see citing works
Han, A., Yuen, H. K., Lee, H. Y., & Zhou, X. (2020) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2020.10.004 · Cited by 39 · see citing works
Daly, J., Mackenzie, L., Freund, M., Wolfenden, L., Roseby, R., & Wiggers, J. (2015) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2015.3342 · Cited by 38 · see citing works
O’Connor, J., McCaughan, D., McDaid, C., Booth, A., Fayter, D., López, R. R., Bowers, R. J., Dyson, L., Iglesias, C. P., Lalor, S., et al. (2016) · Health Technology Assessment · doi:10.3310/hta20550 · Cited by 39 · see citing works
Agnes, A., Lirosi, M. C., Panunzi, S., Santocchi, P., Persiani, R., & D’Ugo, D. (2018) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2018.01.006 · Cited by 37 · see citing works
Improving Causal Inferences in Meta-analyses of Longitudinal Studies: Spanking as an Illustration
Larzelere, R. E., Gunnoe, M. L., & Ferguson, C. J. (2018) · Child Development · doi:10.1111/cdev.13097 · Cited by 37 · see citing works
Treatment of extravasation injuries in infants and young children: a scoping review and survey
Corbett, M. S., Marshall, D. C., Harden, M., Oddie, S., Phillips, B., & McGuire, W. L. (2018) · Health Technology Assessment · doi:10.3310/hta22460 · Cited by 38 · see citing works
Zhang, Y. W., Wang, J., & Fang, T. H. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.953363 · Cited by 38 · see citing works
Quasi-experimental study designs series—paper 9: collecting data from quasi-experimental studies
Aloe, A. M., Becker, B. J., Duvendack, M., Valentine, J. C., Shemilt, I., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.013 · Cited by 35 · see citing works
Self-citationMaughan, B. C., Frueh, L., McDonagh, M. S., Casciere, B., & Kline, J. A. (2020) · Academic Emergency Medicine · doi:10.1111/acem.14108 · Cited by 35 · see citing works
Hormonal contraception and HIV acquisition among women: an updated systematic review
Curtis, K. M., Hannaford, P. C., Rodriguez, M. I., Chipato, T., Steyn, P. S., & Kiarie, J. N. (2020) · BMJ Sexual & Reproductive Health · doi:10.1136/bmjsrh-2019-200509 · Cited by 34 · see citing works
van Leeuwen, H., Sinnaeve, R., Witteveen, U., Daele, T. V., Ossewaarde, L., Egger, J. I. M., & van den Bosch, L. M. C. (2021) · Borderline Personality Disorder and Emotion Dysregulation · doi:10.1186/s40479-021-00165-7 · Cited by 35 · see citing works
Petersen, J. M., Barrett, M., Ahrens, K. A., Murray, E. J., Bryant, A., HOGUE, C. J. R., Mumford, S. L., Gadupudi, S. S., Fox, M. P., & Trinquart, L. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1544 · Cited by 33 · see citing works
Music therapy for the treatment of anxiety: a systematic review with multilevel meta-analyses
de Witte, M., Aalbers, S., Vink, A. C., Friederichs, S. A. H., Knapen, A., Pelgrim, T. C., Lampit, A., Baker, F. A., & van Hooren, S. (2025) · EClinicalMedicine · doi:10.1016/j.eclinm.2025.103293 · Cited by 48 · see citing works
Liu, T., Nie, X., Wu, Z., Zhang, Y., Feng, G., Cai, S., Lv, Y., & Peng, X. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0454-x · Cited by 33 · see citing works
The delivery of chemotherapy at home: an evidence synthesis
Corbett, M. S., Heirs, M. K., Rose, M., Smith, A. M., Stirk, L., Richardson, G. A., Stark, D., Swinson, D., Craig, D., & Eastwood, A. J. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03140 · Cited by 30 · see citing works
Pilates to Improve Core Muscle Activation in Chronic Low Back Pain: A Systematic Review
Franks, J., Thwaites, C. K., & Morris, M. E. (2023) · Healthcare · doi:10.3390/healthcare11101404 · Cited by 29 · see citing works
Glen, P., Chassé, M., Doyle, M.-A., Nasr, A., & Fergusson, D. A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112417 · Cited by 29 · see citing works
Achana, F. A., Hubbard, S., Sutton, A. J., Kendrick, D., & Cooper, N. J. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.09.018 · Cited by 29 · see citing works
Habets, D. H. J., Pelzner, K., Wieten, L., Spaanderman, M. E. A., Villamor, E., & Al‐Nasiry, S. (2022) · Allergy Asthma and Clinical Immunology · doi:10.1186/s13223-022-00660-8 · Cited by 27 · see citing works
An introduction to systematic reviews in animal health, animal welfare, and food safety
O'Connor, A. M. C., & Sargeant, J. M. (2014) · Animal Health Research Reviews · doi:10.1017/s146625231400005x · Cited by 26 · see citing works
Lino, V. T. S., Marinho, D. S., Rodrigues, N. C. P., & de Andrade, C. A. F. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1348028 · Cited by 25 · see citing works
Gianola, S., Bargeri, S., Nembrini, G., Varvello, A., Lunny, C., & Castellini, G. (2022) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2022.09.005 · Cited by 26 · see citing works
Ai, Y., Zhao, J., Liu, H., Li, J., & Zhu, T. (2022) · Frontiers in Pediatrics · doi:10.3389/fped.2022.936813 · Cited by 23 · see citing works
Shimonovich, M., Campbell, M., Thomson, R., Broadbent, P., Wells, V., Kopasker, D., McCartney, G., Thomson, H., Pearce, A., & Katikireddi, S. V. (2024) · Milbank Quarterly · doi:10.1111/1468-0009.12689 · Cited by 23 · see citing works
Rvachew, S., & Matthews, T. (2017) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2017.04.003 · Cited by 23 · see citing works
Leahy, J., Thom, H. H. Z., Jansen, J. P., Gray, E., O’Leary, A. C., White, A., & Walsh, C. (2019) · Statistics in Medicine · doi:10.1002/sim.8139 · Cited by 22 · see citing works
Hatami, M., Soveid, N., Lesani, A., Djafarian, K., & Shab‐Bidar, S. (2021) · CNS & Neurological Disorders - Drug Targets · doi:10.2174/1871527320666210713114840 · Cited by 21 · see citing works
Schlund, A., Reimers, A. K., Bucksch, J., Linder, S., & Demetriou, Y. (2021) · German Journal of Exercise and Sport Research · doi:10.1007/s12662-021-00724-8 · Cited by 21 · see citing works
Verde, P. E., Ohmann, C., Morbach, S., & Icks, A. (2015) · Statistics in Medicine · doi:10.1002/sim.6809 · Cited by 20 · see citing works
Becker, B. J., Aloe, A. M., Duvendack, M., Stanley, T. D., Valentine, J. C., Fretheim, A., & Tugwell, P. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.014 · Cited by 18 · see citing works
Self-citationCheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02464-w · Cited by 18 · see citing works
Message Effectiveness of Fear Appeals in Vaccination Communication Campaigns: A Systematic Review
Limbu, Y. B., & Huhmann, B. A. (2024) · Vaccines · doi:10.3390/vaccines12060653 · Cited by 18 · see citing works
Transversus Abdominis Plane Block in Laparoscopic Colorectal Surgery: A Systematic Review
Wu, R. C., Jensen, C. C., Douaiher, J., Madoff, R. D., & Kwaan, M. R. (2019) · Diseases of the Colon & Rectum · doi:10.1097/dcr.0000000000001464 · Cited by 18 · see citing works
Efficacy and safety of tai chi exercise on bone health: An umbrella review
Li, J., Guo, J., Wang, X., Zhang, X., Zhang, Y., Bu, M., Yao, X., & Yanfen, S. H. E. (2023) · Osteoporosis International · doi:10.1007/s00198-023-06830-7 · Cited by 16 · see citing works
Luxton, T., King, N. V., Wälti, C., Jeuken, L. J. C., & Sandoe, J. A. T. (2022) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkac101 · Cited by 16 · see citing works
Mei, F., Yao, M., Wang, Y., Huan, J., Ma, Y., Li, G., Zou, K., Li, L., & Sun, X. (2024) · BMC Medicine · doi:10.1186/s12916-024-03778-1 · Cited by 15 · see citing works
Luxton, T., King, N. V., Wälti, C., Jeuken, L. J. C., & Sandoe, J. A. T. (2022) · Antibiotics · doi:10.3390/antibiotics11101311 · Cited by 14 · see citing works
Burns, J., Polus, S., Brereton, L., Chilcott, J. B., Ward, S. E., Pfadenhauer, L. M., & Rehfuess, E. A. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1284 · Cited by 14 · see citing works
Han, A. (2023) · Rehabilitation Counseling Bulletin · doi:10.1177/00343552211043257 · Cited by 14 · see citing works
Yan, R., Liu, T., Peng, Y., & Peng, X. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01343-3 · Cited by 12 · see citing works
Developing, Conducting, and Publishing Appropriate Systematic Review and Meta-Analysis Articles
Kelley, B., & Chung, K. C. (2017) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000004031 · Cited by 12 · see citing works
Yao, M., Wang, Y., Ren, Y., Jia, Y., Zou, K., Li, L., & Sun, X. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1648 · Cited by 11 · see citing works
Sensitivity analysis for unmeasured confounding in meta-analysesPreprint
Mathur, M. B., & VanderWeele, T. J. (2017) · doi:10.31219/osf.io/jkhfg · Cited by 11 · see citing works
Low quality evidence on the stability of orthodontic mini-implants
Reynders, R. M. (2013) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6400950 · Cited by 10 · see citing works
Dhiman, P., Lee, H., Kirtley, S., & Collins, P. G. S. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.09.027 · Cited by 10 · see citing works
Tamim, R. M., Borokhovski, E., Bernard, R. M., Schmid, R. F., Abrami, P. C., & Pickup, D. I. (2021) · Australasian Journal of Educational Technology · doi:10.14742/ajet.6322 · Cited by 10 · see citing works
Múñoz, Ó., & Morales, Á. J. R. (2018) · Acta Médica Colombiana · doi:10.36104/amc.2018.1095 · Cited by 9 · see citing works
Ghai, S., Ghai, I., & Narciss, S. (2022) · Children · doi:10.3390/children9111752 · Cited by 9 · see citing works
Bauer, S. R., Wieruszewski, P. M., Bissell, B. D., Dugar, S. P., Sacha, G. L., Sato, R., Siuba, M., Schleicher, M. C., Vachharajani, V., Falck-Ytter, Y. T., et al. (2025) · Shock · doi:10.1097/shk.0000000000002558 · Cited by 8 · see citing works
Evaluating Treatment Options for Common Bovine Diseases Using Published Data and Clinical Experience
O'Connor, A. M. C., & Fajt, V. R. (2015) · Veterinary Clinics of North America Food Animal Practice · doi:10.1016/j.cvfa.2014.11.001 · Cited by 7 · see citing works
Joint incorporation of randomised and observational evidence in estimating treatment effects
Ferguson, J. P., Alvarez‐Iglesias, A., Newell, J., Hinde, J., & Donnell, M. O. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217720854 · Cited by 6 · see citing works
ECMO, ARDS and meta-analyses: Bayes to the rescue?
Graham, P. L., & Moran, J. L. (2020) · Journal of Critical Care · doi:10.1016/j.jcrc.2020.05.009 · Cited by 6 · see citing works
Impact of Technology on Physical Activity Levels of Childhood Cancer Survivors: A Systematic Review
Sümengen, A. A., Savaş, E. H., Ay, A., Koyuncu, İ. E., Erkul, M., & Semerci, R. (2024) · Seminars in Oncology Nursing · doi:10.1016/j.soncn.2024.151757 · Cited by 5 · see citing works
Moran, J. L., & Linden, A. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02215-4 · Cited by 5 · see citing works
Zhou, Y., Yao, M., Mei, F., Ma, Y., Huan, J., Zou, K., Li, L., & Sun, X. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02347-7 · Cited by 5 · see citing works
Mu, W., Huang, Y., Chaumont, A., Létourneau, I., El‐Chaâr, D., Xia, T., & Wen, S. W. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038309 · Cited by 5 · see citing works
Faron, M., Blanchard, P., Ribassin-Majed, L., Pignon, J., Michiels, S., & Teuff, G. L. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259121 · Cited by 5 · see citing works
Pocock, H., Deakin, C. D., Lall, R., Smith, C. M., & Perkins, G. D. (2022) · Resuscitation Plus · doi:10.1016/j.resplu.2022.100232 · Cited by 5 · see citing works
Mental Health Outcomes Associated with Elder Abuse: A Systematic Review and Meta-Analysis
Fang, B., Ye, W., Zhang, Q., & Yan, E. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251383932 · Cited by 4 · see citing works
Dai, M., Furuya‐Kanamori, L., Syed, A., Lin, L., & Wang, Q. (2023) · Frontiers in Pharmacology · doi:10.3389/fphar.2023.1064567 · Cited by 5 · see citing works
Rodríguez-Feria, D., Díaz‐Brochero, C., Múñoz, Ó., López, J. M., & Herazo, J. R. G. (2025) · Infectious Diseases Now · doi:10.1016/j.idnow.2025.105080 · Cited by 4 · see citing works
Soto-Chávez, M. J., Múñoz, Ó., Álzate, J. P., Moreno, C. E. L., Henao, D. C., & Medina, A. M. G. (2022) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2022.110019 · Cited by 4 · see citing works
Hesam‐Shariati, N., Chang, W., McAuley, J. H., Booth, A., Trost, Z., Lin, C., Newton-John, T. R. O., & Gustin, S. M. (2020) · JMIR Research Protocols · doi:10.2196/22821 · Cited by 3 · see citing works
- Authors
- Valentine, J. C., & Thompson, S. G.
- Affiliations
- University of Louisville (US)
- University of Cambridge (GB)
- Corresponding author
- Valentine, Jeffrey C.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2012-11-06
- Volume
- 4
- Issue
- 1
- Pages
- 26-35
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W1918813775
- Cited by
- 130
- References
- 41
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 14 · 2023: 6 · 2022: 10 · 2021: 11 · 2020: 15 · 2019: 11 · 2018: 10 · 2017: 15 · 2016: 5 · 2015: 8 · 2014: 10 · 2013: 5 · 2012: 2
- FWCI
- 2.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute for Health and Care Research (NF-SI-0512-10165)
- British Heart Foundation (RG/08/014/24067)
- Medical Research Council (MR/L003120/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- confounding, selection bias, non-randomized studies, meta-analysis, systematic reviews, randomized controlled trials, generalizability, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
