Issues relating to study design and risk of bias when including non‐randomized studies in systematic reviews on the effects of interventions
Abstract
Non‐randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long‐term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non‐randomized studies. In this second paper in a series, we address how review authors might articulate the particular non‐randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non‐randomized study is at risk of important biases. We offer guidance for describing and classifying different non‐randomized designs based on specific features of the studies in place of using non‐informative study design labels. We also suggest criteria to consider when deciding whether to include non‐randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non‐randomized designs that involve a different inferential logic compared with parallel group trials. Copyright © 2012 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The non-randomised study was scored as high risk for random sequence generation and allocation concealment.
“The non-randomised study was scored as high risk for 'random sequence generation' and 'allocation concealment' while grading for other items was similar to those of randomised trial.41”
From Mindfulness-based interventions for psychological wellbeing and quality of life in men with prostate cancer: A systematic review and meta-analysis (2021) · cites this paper as [41]
About one third of the articles acknowledged that searching for observational studies requires additional time and resources because of lack of specific search filters and poorly established or inconsistently used indexing terms.
“About one third of the articles acknowledged that searching for observational studies requires additional time and resources because of lack of specific search filters and poorly established or inconsistently used indexing terms [38, 39, 47, 49, 56, 58, 59, 64, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Many articles recommended that the eligibility criteria need to be pre-specified and that the rationale for defining the eligibility criteria should be explicitly justified, unambiguous, and derived from the review question.
“Many recommended that the eligibility criteria need to be pre-specified [30, 33, 38, 39, 47, 55, 65] and that the rationale for defining the eligibility criteria should be explicitly justified [38, 39, 65], unambiguous [65], and derived from the review question [47].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Two recent articles advocated deciding on a case-by-case basis which study designs to include.
“Two recent articles advocated deciding on a case-by-case basis [37, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Recommendations were made mainly on two topics: whether different study designs should be included in a systematic review; and whether different study designs should be analysed together in a single meta-analysis.
“Recommendations were made mainly on 2 topics: whether different study designs should be included in a systematic review; [29, 34, 37–39, 49, 56, 65, 73, 75] and whether different study designs should be analysed together in a single meta-analysis [8, 34, 35, 37, 38, 42, 49, 51, 56, 61, 70, 75–78].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Many authoritative articles providing sound advice exist, such as those by Greenland, Egger et al., and a series in Research Synthesis Methods.
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Cites
(2002) · Cited by 13,211 · see citing works
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467 · see citing works
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252 · see citing works
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · see citing works
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230 · see citing works
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · see citing works
Self-citationEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Self-citationRegression-discontinuity analysis: An alternative to the ex post facto experiment.
Thistlethwaite, D. L., & Campbell, D. T. (1960) · Journal of Educational Psychology · doi:10.1037/h0044319 · Cited by 1,469 · see citing works
Ukoumunne, Gulliford, M. C., Chinn, Sterne, & Burney, P. (1999) · Health Technology Assessment · doi:10.3310/hta3050 · Cited by 1,209 · see citing works
Clinical Epidemiology: The Architecture of Clinical Research.
(1985) · Annals of Internal Medicine · doi:10.7326/0003-4819-103-4-649_2 · Cited by 954 · see citing works
Savović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668 · see citing works
Self-citationEfficacy and effectiveness of influenza vaccines in elderly people: a systematic review
Jefferson, T., Rivetti, D., Rivetti, A., Rudin, M., Di Pietrantonj, C., & Demicheli, V (2005) · The Lancet · doi:10.1016/s0140-6736(05)67339-4 · Cited by 601 · see citing works
Koro, C. E., Fedder, D. O., L'Italien, G. J., Weiss, S., Magder, L., Kreyenbuhl, J., Revicki, D., & Buchanan, R. (2002) · BMJ · doi:10.1136/bmj.325.7358.243 · Cited by 421 · see citing works
Assessment and Control for Confounding by Indication in Observational Studies
Psaty, B. M., Koepsell, T. D., Lin, D., Weiss, N. S., Siscovick, D. S., Rosendaal, F. R., Pahor, M., & Furberg, C. D. (1999) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.1999.tb01603.x · Cited by 363 · see citing works
Including Non‐Randomized Studies
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., & Wells, G. A. (2008) · Wiley eBooks · doi:10.1002/9780470712184.ch13 · Cited by 306 · see citing works
Self-citationGolder, S., Loke, Y. K., & Bland, M. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1001026 · Cited by 270 · see citing works
Randomized trials published in some Chinese journals: how many are randomized?
Wu, T., Li, Y., Bian, Z., Liu, G., & Moher, D. (2009) · Trials · doi:10.1186/1745-6215-10-46 · Cited by 229 · see citing works
North of England evidence based guidelines development project: methods of guideline development
Eccles, M., Clapp, Z., Grimshaw, J., Adams, P. C., Higgins, B. G., Purves, I., & Russell, I. T. (1996) · BMJ · doi:10.1136/bmj.312.7033.760 · Cited by 215 · see citing works
Presenting Results and ‘Summary of Findings’ Tables
Schünemann, H. J., Oxman, A. D., Higgins, J. P. T., Vist, G. E., Glasziou, P., & Guyatt, G. H. (2008) · Wiley eBooks · doi:10.1002/9780470712184.ch11 · Cited by 209 · see citing works
Self-citationJefferson, T., Smith, S., Demicheli, V., Harnden, A., Rivetti, A., & Pietrantonj, C. D. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17984-7 · Cited by 200 · see citing works
Tugwell, P., Petticrew, M. P., Kristjansson, E., Welch, V. A., Ueffing, E., Waters, E., Bonnefoy, J., Morgan, A., Doohan, E., & Kelly, M. P. (2010) · BMJ · doi:10.1136/bmj.c4739 · Cited by 184 · see citing works
Self-citationVandenbroucke, J., & Psaty, B. (2008) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.2008.723 · Cited by 148 · see citing works
Models for Potentially Biased Evidence in Meta-Analysis using Empirically Based Priors
Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G., & Sterne, J. A. C. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00548.x · Cited by 147 · see citing works
Valentine, J. C., & Thompson, S. G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1064 · Cited by 130 · see citing works
Self-citationMowatt, G., Grimshaw, J., DAVIS, D. A., & Mazmanian, P. E. (2001) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.1340210109 · Cited by 123 · see citing works
Moss, B. G., & Yeaton, W. H. (2006) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737028003215 · Cited by 89 · see citing works
HIV and male circumcision?a systematic review with assessment of the quality of studies
Siegfried, N., Muller, M., Deeks, J. J., Volmink, J. A., Egger, M., LOW, N., Walker, S., & WILLIAMSON, P. (2005) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(05)70024-4 · Cited by 88 · see citing works
Self-citationHartling, L., Bond, K., Santaguida, P. L., Viswanathan, M., & Dryden, D. M. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.010 · Cited by 69 · see citing works
Agreement between randomized and non-randomized studies: the effects of bias and confounding
Henry, D. A., Moxey, A. J., & O’Connell, D. L. (2001) · BMC News and views · doi:10.1186/2048-4623-1-s3-pa013 · Cited by 5 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsey, C., Tugwell, P., & Wells, G. A. (2013) · Bristol Research (University of Bristol) · Cited by 2 · see citing works
Self-citationD’Agostino (1998) · doi:10.1002/(sici)1097-0258(19981015)17:19
Flagged first, then by citations
Cited by
Sorted by citations
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · see citing works
Interventions to Reduce Ageism Against Older Adults: A Systematic Review and Meta-Analysis
Burnes, D. P. R., Sheppard, C. L., Henderson, C., Wassel, M., Cope, R., Barber, C., & Pillemer, K. A. (2019) · American Journal of Public Health · doi:10.2105/ajph.2019.305123 · Cited by 733 · see citing works
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348 · see citing works
Rauch, B., Davos, C. H., Doherty, P., Saure, D., Metzendorf, M., Salzwedel, A., Völler, H., Jensen, K., & Schmid, J. (2016) · European Journal of Preventive Cardiology · doi:10.1177/2047487316671181 · Cited by 318 · see citing works
Moustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300 · see citing works
Self-citationSalzwedel, A., Jensen, K., Rauch, B., Doherty, P., Metzendorf, M., Hackbusch, M., Völler, H., Schmid, J., & Davos, C. H. (2020) · European Journal of Preventive Cardiology · doi:10.1177/2047487320905719 · Cited by 279 · see citing works
Liebowitz, D. D., & Porter, L. (2019) · Review of Educational Research · doi:10.3102/0034654319866133 · Cited by 320 · see citing works
Farrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1172-8 · Cited by 246 · see citing works
Schünemann, H. J., Tugwell, P., Reeves, B. C., Akl, E. A., Santesso, N., Spencer, F. A., Shea, B., Wells, G., & Helfand, M. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1078 · Cited by 248 · see citing works
Self-citationReeves, B. C., Wells, G. A., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.016 · Cited by 199 · see citing works
Self-citationA meta-analytic review of food safety risk perception
Nardi, V. A. M., Teixeira, R., Ladeira, W. J., & Santini, F. D. O. (2020) · Food Control · doi:10.1016/j.foodcont.2020.107089 · Cited by 202 · see citing works
Mercadante, V., AlHamad, A. F., Lodi, G., Porter, S. R., & Fedele, S. (2017) · Oral Oncology · doi:10.1016/j.oraloncology.2016.12.031 · Cited by 187 · see citing works
Including non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch24 · Cited by 167 · see citing works
Self-citationGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 190 · see citing works
Wells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1077 · Cited by 141 · see citing works
Self-citationBurns, J., Boogaard, H., Polus, S., Pfadenhauer, L. M., Rohwer, A. C., van Erp, A. M., Turley, R., & Rehfuess, E. A. (2019) · Environment International · doi:10.1016/j.envint.2019.105400 · Cited by 134 · see citing works
Risk of AKI with Gentamicin as Surgical Prophylaxis
Bell, S., Davey, P. G., Nathwani, D., Marwick, C., Vadiveloo, T., Sneddon, J., Patton, A., Bennie, M., Fleming, S., & Donnan, P. T. (2014) · Journal of the American Society of Nephrology · doi:10.1681/asn.2014010035 · Cited by 125 · see citing works
Verde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123 · see citing works
Merluzzi, T. V., Pustejovsky, J. E., Philip, E. J., Sohl, S. J., Berendsen, M., & Salsman, J. M. (2019) · Psycho-Oncology · doi:10.1002/pon.5148 · Cited by 122 · see citing works
Johnson, B. A., Waddimba, A. C., Ogola, G., Fleshman, J. W., & Preskitt, J. T. (2020) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2020.12.015 · Cited by 120 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 111 · see citing works
Self-citationBasic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108 · see citing works
Konstantonis, D., Vasileiou, D., Papageorgiou, S. N., & Eliades, T. (2018) · European Journal Of Oral Sciences · doi:10.1111/eos.12409 · Cited by 100 · see citing works
Quasi-experimental study designs series—paper 6: risk of bias assessment
Waddington, H. S., Aloe, A. M., Becker, B. J., Djimeu, E. W., Hombrados, J. G., Tugwell, P., Wells, G. A., & Reeves, B. C. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.015 · Cited by 102 · see citing works
Self-citationRobertson, C., Close, A. J., Fraser, C. M., Gurung, T., Jia, X., Sharma, P., Vale, L. D., Ramsay, C. R., & Pickard, R. S. (2013) · British Journal of Urology · doi:10.1111/bju.12247 · Cited by 95 · see citing works
Self-citationHarris, P. N. A., Wei, J. Y., Shen, A., Abdile, A. A., Paynter, S., Huxley, R., Pandeya, N., Doi, Y., Huh, K., O’Neal, C. S., et al. (2015) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkv346 · Cited by 92 · see citing works
Bellerose, M., Collin, L., & Daw, J. R. (2022) · Health Affairs · doi:10.1377/hlthaff.2021.01150 · Cited by 89 · see citing works
Interventions to reduce ambient particulate matter air pollution and their effect on health
Burns, J., Boogaard, H., Polus, S., Pfadenhauer, L. M., Rohwer, A. C., van Erp, A. M., Turley, R., & Rehfuess, E. A. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010919.pub2 · Cited by 90 · see citing works
Armstrong, E., Eggins, E., Reid, N., Harnett, P. H., & Dawe, S. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9290-6 · Cited by 80 · see citing works
The use of real-world data in cancer drug development
Skovlund, E., Leufkens, H. G. M., & Smyth, J. F. (2018) · European Journal of Cancer · doi:10.1016/j.ejca.2018.06.036 · Cited by 76 · see citing works
Montgomery, P., Movsisyan, A., Grant, S., Macdonald, G. M., & Rehfuess, E. A. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000848 · Cited by 72 · see citing works
Brigden, A., Anderson, E., Linney, C., Morris, R., Parslow, R., Serafimova, T., Smith, L., Briggs, E., Loades, M., & Crawley, E. (2020) · Journal of Medical Internet Research · doi:10.2196/16924 · Cited by 63 · see citing works
Paulus, J. K., Dahabreh, I. J., Balk, E. M., Avendano, E. E., Lau, J., & Ip, S. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1101 · Cited by 66 · see citing works
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 62 · see citing works
Interventions to reduce ambient particulate matter air pollution and their effect on health
Burns, J., Boogaard, H., Turley, R., Pfadenhauer, L. M., van Erp, A. M., Rohwer, A. C., & Rehfuess, E. A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010919 · Cited by 63 · see citing works
Types of Intelligence and Academic Performance: A Systematic Review and Meta-Analysis
Lozano-Blasco, R., Quílez-Robres, A., Supervía, P. U., Salavera, C., & López, R. C. (2022) · Journal of Intelligence · doi:10.3390/jintelligence10040123 · Cited by 84 · see citing works
Mathes, T., & Pieper, D. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1131-4 · Cited by 58 · see citing works
Savva, M., Higgins, S., & Beckmann, N. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12623 · Cited by 64 · see citing works
Paull, D. E., DeLeeuw, L. D., Wolk, S. W., Paige, J. T., Neily, J. B., & Mills, P. D. (2013) · The Journal of Continuing Education in Nursing · doi:10.3928/00220124-20130903-38 · Cited by 51 · see citing works
Norris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50 · see citing works
Self-citationObservational evidence and strength of evidence domains: case examples
O'Neil, M. E., Berkman, N. D., Hartling, L., Chang, S., Anderson, J., Motu’apuaka, M. L., Guise, J., & McDonagh, M. S. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-35 · Cited by 51 · see citing works
Interventions to enhance maternal awareness of decreased fetal movement: a systematic review
Winje, B., Wojcieszek, A. M., Gonzalez‐Angulo, L., Teoh, Z., Norman, J. E., Frøen, J. F., & Flenady, V. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13802 · Cited by 49 · see citing works
Amiodarone or lidocaine for cardiac arrest: A systematic review and meta-analysis
Sanfilippo, F., Corredor, C., Santonocito, C., Panarello, G., Arcadipane, A. F., Ristagno, G., & Pellis, T. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.07.235 · Cited by 48 · see citing works
Bergman, H., Walker, D., Nikolakopoulou, A., Soares‐Weiser, K., & Adams, C. E. (2017) · Health Technology Assessment · doi:10.3310/hta21430 · Cited by 45 · see citing works
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00139 · Cited by 53 · see citing works
Hennegan, J. M., Henderson, J., & Redshaw, M. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008616 · Cited by 44 · see citing works
Systematic Reviews: A Social Work Perspective
Crisp, B. R. (2015) · Australian Social Work · doi:10.1080/0312407x.2015.1024266 · Cited by 44 · see citing works
Sanfilippo, F., Johnson, C., Bellavia, D., Morsolini, M., Romano, G., Santonocito, C., Centineo, L., Pastore, F., Pilato, M., & Arcadipane, A. (2017) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2017.02.046 · Cited by 30 · see citing works
Meta-analyses including data from observational studies
O’Connor, A. M., & Sargeant, J. M. (2014) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.10.017 · Cited by 38 · see citing works
Polus, S., Pieper, D., Burns, J., Fretheim, A., Ramsay, C. R., Higgins, J. P. T., Mathes, T., Pfadenhauer, L. M., & Rehfuess, E. A. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.008 · Cited by 41 · see citing works
Self-citationIjaz, S., Verbeek, J. H. A. M., Tikka, C., & Ruotsalainen, J. H. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.01.001 · Cited by 41 · see citing works
Martin, A., Ogilvie, D., & Suhrcke, M. (2014) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-014-0142-8 · Cited by 41 · see citing works
O’Connor, J., McCaughan, D., McDaid, C., Booth, A., Fayter, D., López, R. R., Bowers, R. J., Dyson, L., Iglesias, C. P., Lalor, S., et al. (2016) · Health Technology Assessment · doi:10.3310/hta20550 · Cited by 39 · see citing works
Agnes, A., Lirosi, M. C., Panunzi, S., Santocchi, P., Persiani, R., & D’Ugo, D. (2018) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2018.01.006 · Cited by 37 · see citing works
Evaluation of root resorption following orthodontic intrusion: a systematic review and meta-analysis
Bellini‐Pereira, S. A., de Almeida, J. C., Castillo, A. A.-D., dos Santos, C. C. O., Henriques, J. F. C., & Janson, G. (2020) · European Journal of Orthodontics · doi:10.1093/ejo/cjaa054 · Cited by 37 · see citing works
Quasi-experimental study designs series—paper 9: collecting data from quasi-experimental studies
Aloe, A. M., Becker, B. J., Duvendack, M., Valentine, J. C., Shemilt, I., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.013 · Cited by 35 · see citing works
Self-citationParkinson’s disease patients benefit from bicycling - a systematic review and meta-analysis
Tiihonen, M., Westner, B. U., Butz, M., & Dalal, S. S. (2021) · npj Parkinson s Disease · doi:10.1038/s41531-021-00222-6 · Cited by 35 · see citing works
Janjua, S., Powell, P., Atkinson, R. W., Stovold, E. M., & Normansell, R. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013441.pub2 · Cited by 40 · see citing works
Cuello‐García, C. A., Morgan, R. L., Brożek, J., Santesso, N. A., Verbeek, J. H. A. M., Thayer, K. A., Guyatt, G. H., & Schünemann, H. J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.010 · Cited by 32 · see citing works
Study design classification of registry-based studies in systematic reviews
Mathes, T., & Pieper, D. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.09.016 · Cited by 32 · see citing works
Boulet, N., Pensier, J., Occéan, B., Fabbro-Péray, P., Mimoz, O., Rickard, C. M., Buetti, N., Lefrant, J.-Y., Muller, L., & Roger, C. (2024) · Critical Care · doi:10.1186/s13054-024-05162-0 · Cited by 31 · see citing works
Velasco, M., Zarco, L. A., Agudelo-Arrieta, M., Torres-Camacho, I., Garcia-Cifuentes, E. R., & Múñoz, Ó. (2021) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2021.102869 · Cited by 31 · see citing works
Fuller, T., Peters, J., Pearson, M., & Anderson, R. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.302195 · Cited by 30 · see citing works
The delivery of chemotherapy at home: an evidence synthesis
Corbett, M. S., Heirs, M. K., Rose, M., Smith, A. M., Stirk, L., Richardson, G. A., Stark, D., Swinson, D., Craig, D., & Eastwood, A. J. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03140 · Cited by 30 · see citing works
Effects of Deep Brain Stimulation on Postural Trunk Deformities: A Systematic Review
Lizárraga, K. J., & Fasano, A. A. (2019) · Movement Disorders Clinical Practice · doi:10.1002/mdc3.12829 · Cited by 29 · see citing works
Cheng, M. P., Lee, R. S., Cheng, A. P., l’Étoile-Morel, S. D., Demir, K. K., Yansouni, C. P., Harris, P. N. A., McDonald, E. G., & Lee, T. C. (2019) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofz248 · Cited by 27 · see citing works
Pace, C. S., Muzi, S., Madera, F., Sansò, A., & Zavattini, G. C. (2021) · Attachment & Human Development · doi:10.1080/14616734.2021.1991664 · Cited by 26 · see citing works
Via, L. L., Astuto, M., Bignami, E. G., Busalacchi, D., Dezio, V., Girardis, M., Lanzafame, B., Ristagno, G., Pelosi, P. P., & Sanfilippo, F. (2022) · Minerva Anestesiologica · doi:10.23736/s0375-9393.22.16449-7 · Cited by 26 · see citing works
An introduction to systematic reviews in animal health, animal welfare, and food safety
O'Connor, A. M. C., & Sargeant, J. M. (2014) · Animal Health Research Reviews · doi:10.1017/s146625231400005x · Cited by 26 · see citing works
Nnate, D. A., Anyachukwu, C. C., Igwe, S. E., & Abaraogu, U. O. (2021) · Psycho-Oncology · doi:10.1002/pon.5749 · Cited by 24 · see citing works
Is paid sick leave bad for business? A systematic review
Weerdt, C. V., Stoddard-Dare, P. A., & DeRigne, L. (2023) · American Journal of Industrial Medicine · doi:10.1002/ajim.23469 · Cited by 25 · see citing works
Guidance to best tools and practices for systematic reviews1
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Journal of Pediatric Rehabilitation Medicine · doi:10.3233/prm-230019 · Cited by 29 · see citing works
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · British Journal of Pharmacology · doi:10.1111/bph.16100 · Cited by 27 · see citing works
Smilowitz, J. T., Lemay, D. G., Kalanetra, K. M., Chin, E. L., Zivkovic, A. M., Breck, M. A., German, J. B., Mills, D. A., Slupsky, C. M., & Barile, D. (2017) · Journal of Nutritional Science · doi:10.1017/jns.2017.2 · Cited by 21 · see citing works
Salomon, M., Pastore, C., Maselli, F., Bari, M. D., Pellegrino, R., & Brindisino, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159715 · Cited by 20 · see citing works
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08304-x · Cited by 23 · see citing works
Verde, P. E., Ohmann, C., Morbach, S., & Icks, A. (2015) · Statistics in Medicine · doi:10.1002/sim.6809 · Cited by 20 · see citing works
Walker, H., Patton, A., Bayne, G., Marwick, C., Sneddon, J., Davey, P. G., Nathwani, D., & Bell, S. (2016) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkw166 · Cited by 19 · see citing works
Applications of Temperament: A Review of Caregiver-Focused Temperament-Driven Interventions
Iverson, S. L., & Gartstein, M. A. (2017) · Early Education and Development · doi:10.1080/10409289.2017.1341805 · Cited by 19 · see citing works
Cheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02464-w · Cited by 18 · see citing works
Merlotti, A. M., Mazzola, R., Alterio, D., Alongi, F., Bacigalupo, A., Bonomo, P., Maddalo, M., Russi, E. G., & Orlandi, E. M. (2017) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2017.01.008 · Cited by 18 · see citing works
Guidance to Best Tools and Practices for Systematic Reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBJS Reviews · doi:10.2106/jbjs.rvw.23.00077 · Cited by 20 · see citing works
Brindisino, F., Girardi, G., Crestani, M., Fiore, A., Giovannico, G., Garzonio, F., Venturin, D., & Struyf, F. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2251880 · Cited by 17 · see citing works
Carbonation as a sensory enhancement strategy: a narrative synthesis of existing evidence
Turkington, L., Ward, E. C., & Farrell, A. M. (2016) · Disability and Rehabilitation · doi:10.1080/09638288.2016.1213894 · Cited by 16 · see citing works
Sterile processing in low- and middle-income countries: an integrative review
Cuncannon, A., Dosani, A., & Fast, O. (2020) · Journal of Infection Prevention · doi:10.1177/1757177420947468 · Cited by 20 · see citing works
Outbreaks of publications about emerging infectious diseases: the case of SARS-CoV-2 and Zika virus
Ipekci, A. M., Buitrago‐García, D., Meili, K. W., Krauer, F., Prajapati, N., Thapa, S., Wildisen, L., Araujo-Chaveron, L., Baumann, L., Shah, S., et al. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01244-7 · Cited by 16 · see citing works
Mei, F., Yao, M., Wang, Y., Huan, J., Ma, Y., Li, G., Zou, K., Li, L., & Sun, X. (2024) · BMC Medicine · doi:10.1186/s12916-024-03778-1 · Cited by 15 · see citing works
Quintanilla, B. P. A., Taft, A., McDonald, S., Pollock, W., & Roque-Henriquez, J. C. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-013270 · Cited by 15 · see citing works
Incorporating Both Randomized and Observational Data into a Single Analysis
Kaizar, E. E. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020249 · Cited by 15 · see citing works
Zamora, N. V., Christensen, R., Goel, N., Klokker, L., Lopez-Olivo, M. A., Kristensen, L. E., Carmona, L., Strand, V., Curtis, J. R., & Suarez-Almazor, M. E. (2017) · The Journal of Rheumatology · doi:10.3899/jrheum.161108 · Cited by 15 · see citing works
Faggion, C. M., Huda, F., & Wasiak, J. (2014) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.12251 · Cited by 13 · see citing works
Prina, E., Tedeschi, F., Salazzari, D., Botte, T., Ballarin, M., Rabbi, L., Imperadore, G., Roccato, S., Nicolaou, S., Ruggeri, M., et al. (2023) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796023000100 · Cited by 12 · see citing works
Cheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.02.008 · Cited by 12 · see citing works
Girolamo, N. D., & Mans, C. (2018) · Elsevier eBooks · doi:10.1016/b978-0-323-55228-8.00011-4 · Cited by 12 · see citing works
Geidl, W., Schlesinger, S., Mino, E., Miranda, L., Ryan, A., Bartsch, K. M., Janz, L., & Pfeifer, K. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-028653 · Cited by 11 · see citing works
The Methodological Quality Scale (MQS) for intervention programs: validity evidence
Moscoso, S. C., Sanduvete‐Chaves, S., Lozano-Lozano, J. A., & Tello, F. P. H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1217661 · Cited by 11 · see citing works
Chim, K., Lai, J. T. C., & Chan, B. T. Y. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1450192 · Cited by 15 · see citing works
Drahota, A., Felix, L. M., Raftery, J. P., Keenan, B. E., Lachance, C. C., Mackey, D. C., Markham, C. M., Laing, A. C., Farrell, K. R. C., & Okunribido, O. O. (2022) · Health Technology Assessment · doi:10.3310/zowl2323 · Cited by 11 · see citing works
Yamamoto, S., Ishii, D., Noguchi, A., Tanamachi, K., Okamoto, Y., Takasaki, Y., Miyata, K., Fujita, Y., Kishimoto, H., Hotta, K., et al. (2019) · International Quarterly of Community Health Education · doi:10.1177/0272684x19896732 · Cited by 11 · see citing works
- Authors
- Higgins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G.
- Affiliations
- MRC Biostatistics Unit (GB)
- University of Aberdeen (GB)
- Bristol Royal Infirmary (GB)
- University of Birmingham (GB)
- University of Ottawa (CA)
- University of Louisville (US)
- Corresponding author
- Higgins, Julian PT
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2012-09-25
- Volume
- 4
- Issue
- 1
- Pages
- 12-25
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2062040186
- Cited by
- 183
- References
- 31
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 17 · 2023: 23 · 2022: 8 · 2021: 17 · 2020: 18 · 2019: 21 · 2018: 10 · 2017: 16 · 2016: 11 · 2015: 11 · 2014: 11 · 2013: 9 · 2012: 1
- FWCI
- 2.82
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- Scottish Government
- National Institute for Health and Care Research
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- non-randomized studies, risk of bias, study design classification, intervention effects, observational study design, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
