The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Issues relating to study design and risk of bias when including non‐randomized studies in systematic reviews on the effects of interventions
Abstract
Non‐randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long‐term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non‐randomized studies. In this second paper in a series, we address how review authors might articulate the particular non‐randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non‐randomized study is at risk of important biases. We offer guidance for describing and classifying different non‐randomized designs based on specific features of the studies in place of using non‐informative study design labels. We also suggest criteria to consider when deciding whether to include non‐randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non‐randomized designs that involve a different inferential logic compared with parallel group trials. Copyright © 2012 John Wiley & Sons, Ltd.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The non-randomised study was scored as high risk for 'random sequence generation' and 'allocation concealment' while grading for other items was similar to those of randomised trial.41”
From Mindfulness-based interventions for psychological wellbeing and quality of life in men with prostate cancer: A systematic review and meta-analysis (2021) · cites this paper as [41]
In the citing paper
“About one third of the articles acknowledged that searching for observational studies requires additional time and resources because of lack of specific search filters and poorly established or inconsistently used indexing terms [38, 39, 47, 49, 56, 58, 59, 64, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Many recommended that the eligibility criteria need to be pre-specified [30, 33, 38, 39, 47, 55, 65] and that the rationale for defining the eligibility criteria should be explicitly justified [38, 39, 65], unambiguous [65], and derived from the review question [47].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Two recent articles advocated deciding on a case-by-case basis [37, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Recommendations were made mainly on 2 topics: whether different study designs should be included in a systematic review; [29, 34, 37–39, 49, 56, 65, 73, 75] and whether different study designs should be analysed together in a single meta-analysis [8, 34, 35, 37, 38, 42, 49, 51, 56, 61, 70, 75–78].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
In the citing paper
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [65]
Citations
Cites
(2002) · Cited by 13,211
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Self-citationEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Self-citationRegression-discontinuity analysis: An alternative to the ex post facto experiment.
Thistlethwaite, D. L., & Campbell, D. T. (1960) · Journal of Educational Psychology · doi:10.1037/h0044319 · Cited by 1,469
Ukoumunne, Gulliford, M. C., Chinn, Sterne, & Burney, P. (1999) · Health Technology Assessment · doi:10.3310/hta3050 · Cited by 1,211
Clinical Epidemiology: The Architecture of Clinical Research.
(1985) · Annals of Internal Medicine · doi:10.7326/0003-4819-103-4-649_2 · Cited by 954
Flagged first, then by citations
Research works printed but not linked: 3
- Cochrane Effective Practice and Organisation of Care Group.2002.Data collection checklist.http://epoc.cochrane.org/epoc‐resources‐review‐authors(accessed 12 August 2011).
- Norris SL, Issues relating to selective reporting when including non‐randomized studies in systematic reviews on the effects of interventions, Research Synthesis Methods
- Oxford Centre for Evidence‐based Medicine.2001.Levels of evidence[May 2001].http://www.cebm.net/index.aspx?o=1047. 1‐1‐2008
Cited by
Sorted by citations
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Interventions to Reduce Ageism Against Older Adults: A Systematic Review and Meta-Analysis
Burnes, D. P. R., Sheppard, C. L., Henderson, C., Wassel, M., Cope, R., Barber, C., & Pillemer, K. A. (2019) · American Journal of Public Health · doi:10.2105/ajph.2019.305123 · Cited by 838
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradicto…”
BackgroundInfluentialGet real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348
“First, researchers should critically appraise the risk of bias in available studies prior to cross-design synthesis and carefully decide whether inclusion is appropriate (Berger et al., 2014; Higgins et al., 2013; Reeves et al., 2003; Wells et al., 2013).”
BackgroundRauch, B., Davos, C. H., Doherty, P., Saure, D., Metzendorf, M., Salzwedel, A., Völler, H., Jensen, K., & Schmid, J. (2016) · European Journal of Preventive Cardiology · doi:10.1177/2047487316671181 · Cited by 318
Moustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300
“7 32 We suggest careful consideration of the risk of confounding and of bias, such as bias due to misclassification of methodological characteristics or due to erroneous identification of treatments as experimental and control, in meta-epidemiological studies.”
BackgroundSelf-citationSalzwedel, A., Jensen, K., Rauch, B., Doherty, P., Metzendorf, M., Hackbusch, M., Völler, H., Schmid, J., & Davos, C. H. (2020) · European Journal of Preventive Cardiology · doi:10.1177/2047487320905719 · Cited by 290
“*Specific actions to compare groups: (1) prospectively evaluated intervention group versus retrospectively evaluated control group; (2) linkage of Canadian APPROACH and NACPR registry; (3) data extracted from the United States renal data System, USRDS; (4) retrospective identification of groups by questionnaires withi…”
Liebowitz, D. D., & Porter, L. (2019) · Review of Educational Research · doi:10.3102/0034654319866133 · Cited by 384
“groups have noted (Higgins et al., 2013; Stroup et al., 2000; Valentine & Thompson, 2013), while metaanalyses of causal studies are superior to observational ones, many fields do not lend themselves easily to”
BackgroundMethodInfluentialFarrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1172-8 · Cited by 246
Schünemann, H. J., Tugwell, P., Reeves, B. C., Akl, E. A., Santesso, N., Spencer, F. A., Shea, B., Wells, G., & Helfand, M. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1078 · Cited by 248
Self-citationReeves, B. C., Wells, G. A., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.016 · Cited by 199
“Question 5-7 are essentially the same as in the original checklist [1,2].”
BackgroundMethodInfluentialSelf-citationA meta-analytic review of food safety risk perception
Nardi, V. A. M., Teixeira, R., Ladeira, W. J., & Santini, F. D. O. (2020) · Food Control · doi:10.1016/j.foodcont.2020.107089 · Cited by 202
Mercadante, V., AlHamad, A. F., Lodi, G., Porter, S. R., & Fedele, S. (2017) · Oral Oncology · doi:10.1016/j.oraloncology.2016.12.031 · Cited by 187
Including non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch24 · Cited by 167
Self-citationGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 215
“Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [79, 80] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with…”
BackgroundWells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1077 · Cited by 141
“studies formed by classifying clusters by intervention and comparator) as appropriate, to each primary study (Table 1 and Table 2) [Higgins et al., 2013].”
BackgroundMethodInfluentialSelf-citationBurns, J., Boogaard, H., Polus, S., Pfadenhauer, L. M., Rohwer, A. C., van Erp, A. M., Turley, R., & Rehfuess, E. A. (2019) · Environment International · doi:10.1016/j.envint.2019.105400 · Cited by 134
Risk of AKI with Gentamicin as Surgical Prophylaxis
Bell, S., Davey, P. G., Nathwani, D., Marwick, C., Vadiveloo, T., Sneddon, J., Patton, A., Bennie, M., Fleming, S., & Donnan, P. T. (2014) · Journal of the American Society of Nephrology · doi:10.1681/asn.2014010035 · Cited by 125
Verde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123
“Four discussion papers covered the following: issues in study design and risk of bias by Higgins et al. (2013), issues relating to confounding factors when including non-randomized evidence by Valentine and Thompson (2013), issues in selective reporting by Norris et al. (2013), applicability of…”
BackgroundInfluentialMerluzzi, T. V., Pustejovsky, J. E., Philip, E. J., Sohl, S. J., Berendsen, M., & Salsman, J. M. (2019) · Psycho-Oncology · doi:10.1002/pon.5148 · Cited by 122
“For each category, we used the coded values to inform an assessment of bias as either low, unclear, or high risk of bias.(39)”
MethodJohnson, B. A., Waddimba, A. C., Ogola, G., Fleshman, J. W., & Preskitt, J. T. (2020) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2020.12.015 · Cited by 122
Reeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 116
“Different research fields may call some study designs by different names or may tend to use specific study designs because of the nature of the contexts or interventions of interest (Higgins et al., 2012).”
BackgroundSelf-citationBasic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108
“risk of bias and correctness of the analyses used, as well as awareness of the topic area [54,55].”
MethodKonstantonis, D., Vasileiou, D., Papageorgiou, S. N., & Eliades, T. (2018) · European Journal Of Oral Sciences · doi:10.1111/eos.12409 · Cited by 100
“On the other hand, baseline similarity of samples in observational studies can also be ensured, using refined statistical methods, such as propensity scores/discrimant analyses (13, 15, 48, 49), to ‘match’ the compared samples and reduce bias (50).”
MethodQuasi-experimental study designs series—paper 6: risk of bias assessment
Waddington, H. S., Aloe, A. M., Becker, B. J., Djimeu, E. W., Hombrados, J. G., Tugwell, P., Wells, G. A., & Reeves, B. C. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.015 · Cited by 110
“[6] Higgins JPT, Ramsay C, Reeves BC, Deeks JJ, Shea B,”
BackgroundSelf-citationRobertson, C., Close, A. J., Fraser, C. M., Gurung, T., Jia, X., Sharma, P., Vale, L. D., Ramsay, C. R., & Pickard, R. S. (2013) · British Journal of Urology · doi:10.1111/bju.12247 · Cited by 95
Self-citationHarris, P. N. A., Wei, J. Y., Shen, A., Abdile, A. A., Paynter, S., Huxley, R., Pandeya, N., Doi, Y., Huh, K., O’Neal, C. S., et al. (2015) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkv346 · Cited by 92
Bellerose, M., Collin, L., & Daw, J. R. (2022) · Health Affairs · doi:10.1377/hlthaff.2021.01150 · Cited by 89
Interventions to reduce ambient particulate matter air pollution and their effect on health
Burns, J., Boogaard, H., Polus, S., Pfadenhauer, L. M., Rohwer, A. C., van Erp, A. M., Turley, R., & Rehfuess, E. A. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010919.pub2 · Cited by 93
Armstrong, E., Eggins, E., Reid, N., Harnett, P. H., & Dawe, S. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9290-6 · Cited by 80
“…affected by the methodological quality of evaluation studies more generally, and the type of research design specifically (for reviews, see e.g., Higgins et al. 2013; Shadish et al. 2008), it was anticipated that study effects may differ depending onwhether studies used randomized versus…”
The use of real-world data in cancer drug development
Skovlund, E., Leufkens, H. G. M., & Smyth, J. F. (2018) · European Journal of Cancer · doi:10.1016/j.ejca.2018.06.036 · Cited by 79
Montgomery, P., Movsisyan, A., Grant, S., Macdonald, G. M., & Rehfuess, E. A. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000848 · Cited by 72
Brigden, A., Anderson, E., Linney, C., Morris, R., Parslow, R., Serafimova, T., Smith, L., Briggs, E., Loades, M., & Crawley, E. (2020) · Journal of Medical Internet Research · doi:10.2196/16924 · Cited by 63
Paulus, J. K., Dahabreh, I. J., Balk, E. M., Avendano, E. E., Lau, J., & Ip, S. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1101 · Cited by 66
“Implementation of this guidance may result in increased consistency among reviews in the near future (Higgins et al., 2013).”
BackgroundTurner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 66
“Second, issues of study design arise when assessing risk of bias when including nonrandomised studies in systematic reviews [112-114].”
BackgroundInterventions to reduce ambient particulate matter air pollution and their effect on health
Burns, J., Boogaard, H., Turley, R., Pfadenhauer, L. M., van Erp, A. M., Rohwer, A. C., & Rehfuess, E. A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010919 · Cited by 66
Types of Intelligence and Academic Performance: A Systematic Review and Meta-Analysis
Lozano-Blasco, R., Quílez-Robres, A., Supervía, P. U., Salavera, C., & López, R. C. (2022) · Journal of Intelligence · doi:10.3390/jintelligence10040123 · Cited by 96
Mathes, T., & Pieper, D. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1131-4 · Cited by 56
Savva, M., Higgins, S., & Beckmann, N. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12623 · Cited by 87
Paull, D. E., DeLeeuw, L. D., Wolk, S. W., Paige, J. T., Neily, J. B., & Mills, P. D. (2013) · The Journal of Continuing Education in Nursing · doi:10.3928/00220124-20130903-38 · Cited by 51
Norris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50
Self-citationObservational evidence and strength of evidence domains: case examples
O'Neil, M. E., Berkman, N. D., Hartling, L., Chang, S., Anderson, J., Motu’apuaka, M. L., Guise, J., & McDonagh, M. S. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-35 · Cited by 52
“While all three groups support circumstantial use of observational studies in systematic review, all also note concern about the higher risk of bias associated with observational studies compared to RCTs While Higgins, et al.(10) provided recommendations for a priori inclusion criteria, they highlighted the complexiti…”
ResultInterventions to enhance maternal awareness of decreased fetal movement: a systematic review
Winje, B., Wojcieszek, A. M., Gonzalez‐Angulo, L., Teoh, Z., Norman, J. E., Frøen, J. F., & Flenady, V. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13802 · Cited by 49
Amiodarone or lidocaine for cardiac arrest: A systematic review and meta-analysis
Sanfilippo, F., Corredor, C., Santonocito, C., Panarello, G., Arcadipane, A. F., Ristagno, G., & Pellis, T. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.07.235 · Cited by 48
Bergman, H., Walker, D., Nikolakopoulou, A., Soares‐Weiser, K., & Adams, C. E. (2017) · Health Technology Assessment · doi:10.3310/hta21430 · Cited by 45
“Risk-of-bias assessment for observational studies was performed by a senior systematic reviewer (Artemisia Kakourou) using a tool that is currently being tested by Cochrane.(59) The following domains were assessed: (1) confounding and selection bias (including confounders measured and addressed, use of matching and me…”
MethodGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00139 · Cited by 57
Hennegan, J. M., Henderson, J., & Redshaw, M. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008616 · Cited by 44
“Quality of included studies An assessment of the strengths and weaknesses of included studies is essential in estimating the reliability of effect sizes presented.(65) Risk ratings for aspects of study quality are presented in table 2.”
BackgroundSystematic Reviews: A Social Work Perspective
Crisp, B. R. (2015) · Australian Social Work · doi:10.1080/0312407x.2015.1024266 · Cited by 44
“For example, changes over time in the social policy context, including legislation, policies, and institutional processes can all impact on the effectiveness of interventions (Higgins et al., 2013; Shlonsky & Mildon, 2014).”
BackgroundSanfilippo, F., Johnson, C., Bellavia, D., Morsolini, M., Romano, G., Santonocito, C., Centineo, L., Pastore, F., Pilato, M., & Arcadipane, A. (2017) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2017.02.046 · Cited by 30
Meta-analyses including data from observational studies
O’Connor, A. M., & Sargeant, J. M. (2014) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.10.017 · Cited by 38
Polus, S., Pieper, D., Burns, J., Fretheim, A., Ramsay, C. R., Higgins, J. P. T., Mathes, T., Pfadenhauer, L. M., & Rehfuess, E. A. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.008 · Cited by 41
“nature and may include a control group (controlled ITS) [8, 15, 22].”
BackgroundSelf-citationIjaz, S., Verbeek, J. H. A. M., Tikka, C., & Ruotsalainen, J. H. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.01.001 · Cited by 42
“The idea of using the Cochrane risk of bias tool for all RCTs, CCTs, quasi-randomized trials, CBA studies, cohort studies, and all studies with a concurrent control group and a prospective follow-up seems logical [19].”
MethodResultMartin, A., Ogilvie, D., & Suhrcke, M. (2014) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-014-0142-8 · Cited by 41
“It also supports the case for improved tools to distinguish between studies in policy areas, including public health, criminology, education, the labour market and international development, where observational study designs are the norm [24,86-90].”
BackgroundO’Connor, J., McCaughan, D., McDaid, C., Booth, A., Fayter, D., López, R. R., Bowers, R. J., Dyson, L., Iglesias, C. P., Lalor, S., et al. (2016) · Health Technology Assessment · doi:10.3310/hta20550 · Cited by 39
“However, there is a broad consensus in the methodological literature that selection bias and confounding are key sources of bias in observational study designs.(30,31) We used a similar approach to Siegfried et al.”
MethodAgnes, A., Lirosi, M. C., Panunzi, S., Santocchi, P., Persiani, R., & D’Ugo, D. (2018) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2018.01.006 · Cited by 37
Evaluation of root resorption following orthodontic intrusion: a systematic review and meta-analysis
Bellini‐Pereira, S. A., de Almeida, J. C., Castillo, A. A.-D., dos Santos, C. C. O., Henriques, J. F. C., & Janson, G. (2020) · European Journal of Orthodontics · doi:10.1093/ejo/cjaa054 · Cited by 37
Quasi-experimental study designs series—paper 9: collecting data from quasi-experimental studies
Aloe, A. M., Becker, B. J., Duvendack, M., Valentine, J. C., Shemilt, I., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.013 · Cited by 35
Self-citationParkinson’s disease patients benefit from bicycling - a systematic review and meta-analysis
Tiihonen, M., Westner, B. U., Butz, M., & Dalal, S. S. (2021) · npj Parkinson s Disease · doi:10.1038/s41531-021-00222-6 · Cited by 35
Janjua, S., Powell, P., Atkinson, R. W., Stovold, E. M., & Normansell, R. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013441.pub2 · Cited by 40
Cuello‐García, C. A., Morgan, R. L., Brożek, J., Santesso, N. A., Verbeek, J. H. A. M., Thayer, K. A., Guyatt, G. H., & Schünemann, H. J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.010 · Cited by 33
Study design classification of registry-based studies in systematic reviews
Mathes, T., & Pieper, D. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.09.016 · Cited by 32
Boulet, N., Pensier, J., Occéan, B., Fabbro-Péray, P., Mimoz, O., Rickard, C. M., Buetti, N., Lefrant, J.-Y., Muller, L., & Roger, C. (2024) · Critical Care · doi:10.1186/s13054-024-05162-0 · Cited by 31
Velasco, M., Zarco, L. A., Agudelo-Arrieta, M., Torres-Camacho, I., Garcia-Cifuentes, E. R., & Múñoz, Ó. (2021) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2021.102869 · Cited by 31
Fuller, T., Peters, J., Pearson, M., & Anderson, R. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.302195 · Cited by 30
“Higgins et al. recently reached a similar conclusion in their article, which was based on a workshop organized by the Non-Randomized Studies Methods Group of the Cochrane Collaboration.36”
MethodThe delivery of chemotherapy at home: an evidence synthesis
Corbett, M. S., Heirs, M. K., Rose, M., Smith, A. M., Stirk, L., Richardson, G. A., Stark, D., Swinson, D., Craig, D., & Eastwood, A. J. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03140 · Cited by 30
Effects of Deep Brain Stimulation on Postural Trunk Deformities: A Systematic Review
Lizárraga, K. J., & Fasano, A. A. (2019) · Movement Disorders Clinical Practice · doi:10.1002/mdc3.12829 · Cited by 29
“Thus, the level of evidence on this topic is of low quality and has a high risk of bias that might be further increased by the use of meta-analytical or other statistical techniques.(44) Despite these methodological constraints, we were able to identify five groups of patients with similar tendencies (Tables 1–5): (1)…”
BackgroundCheng, M. P., Lee, R. S., Cheng, A. P., l’Étoile-Morel, S. D., Demir, K. K., Yansouni, C. P., Harris, P. N. A., McDonald, E. G., & Lee, T. C. (2019) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofz248 · Cited by 27
“Study quality was assessed using the NewcastleOttawa Quality Assessment Scale [7], but there was no quality threshold to be included.”
MethodPace, C. S., Muzi, S., Madera, F., Sansò, A., & Zavattini, G. C. (2021) · Attachment & Human Development · doi:10.1080/14616734.2021.1991664 · Cited by 26
“Despite this being common practice in performing reviews and meta-analyses, it exposes the current study to the risk of selection bias (Higgins et al., 2013).”
Via, L. L., Astuto, M., Bignami, E. G., Busalacchi, D., Dezio, V., Girardis, M., Lanzafame, B., Ristagno, G., Pelosi, P. P., & Sanfilippo, F. (2022) · Minerva Anestesiologica · doi:10.23736/s0375-9393.22.16449-7 · Cited by 25
An introduction to systematic reviews in animal health, animal welfare, and food safety
O'Connor, A. M. C., & Sargeant, J. M. (2014) · Animal Health Research Reviews · doi:10.1017/s146625231400005x · Cited by 26
“Knowledge of sources of biases and methods of meta-analysis for review questions about causation, disease burden, and diagnostic test evaluation are evolving and improving (Begg, 1987; Begg and Mazumdar, 1994; Deeks et al., 2005; Glasziou et al., 2008; Higgins et al., 2013).”
MethodNnate, D. A., Anyachukwu, C. C., Igwe, S. E., & Abaraogu, U. O. (2021) · Psycho-Oncology · doi:10.1002/pon.5749 · Cited by 24
Is paid sick leave bad for business? A systematic review
Weerdt, C. V., Stoddard-Dare, P. A., & DeRigne, L. (2023) · American Journal of Industrial Medicine · doi:10.1002/ajim.23469 · Cited by 25
Guidance to best tools and practices for systematic reviews1
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Journal of Pediatric Rehabilitation Medicine · doi:10.3233/prm-230019 · Cited by 35
“Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [79, 80].”
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · British Journal of Pharmacology · doi:10.1111/bph.16100 · Cited by 29
“Non-randomized studies of intervention (NRSI) are affected by a greater potential range of biases and thus vary more than RCTs in their ability to estimate a causal effect (Higgins et al., 2013).”
BackgroundSmilowitz, J. T., Lemay, D. G., Kalanetra, K. M., Chin, E. L., Zivkovic, A. M., Breck, M. A., German, J. B., Mills, D. A., Slupsky, C. M., & Barile, D. (2017) · Journal of Nutritional Science · doi:10.1017/jns.2017.2 · Cited by 21
“These observations are unsurprising since the supplemental arms were not randomly assigned and probably due to performance bias that is reduced with randomisation((40)).”
Salomon, M., Pastore, C., Maselli, F., Bari, M. D., Pellegrino, R., & Brindisino, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159715 · Cited by 20
“that can feasibly be studied in RCTs, but for which only a small number of RCTs are available (or likely to be available) [34,35].”
BackgroundMethodGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08304-x · Cited by 24
“Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [79, 80] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with…”
BackgroundVerde, P. E., Ohmann, C., Morbach, S., & Icks, A. (2015) · Statistics in Medicine · doi:10.1002/sim.6809 · Cited by 21
“[41], issues relating to confounding factors when including non-randomized evidence by Valentine and Thompson [42], issues in selective reporting by Norris et al.”
BackgroundWalker, H., Patton, A., Bayne, G., Marwick, C., Sneddon, J., Davey, P. G., Nathwani, D., & Bell, S. (2016) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkw166 · Cited by 19
Applications of Temperament: A Review of Caregiver-Focused Temperament-Driven Interventions
Iverson, S. L., & Gartstein, M. A. (2017) · Early Education and Development · doi:10.1080/10409289.2017.1341805 · Cited by 19
“Given the dearth of temperament intervention research, nonrandomized studies were considered, limiting generalizability and increasing potential biases (Higgins et al., 2013).”
BackgroundCheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02464-w · Cited by 20
“In such cases, the risk of establishing misleading conclusions is high, and the interpretation of summary results from highly heterogeneous sets of studies should explicitly account for the inherently high uncertainty they are accompanied with [19–21].”
BackgroundMerlotti, A. M., Mazzola, R., Alterio, D., Alongi, F., Bacigalupo, A., Bonomo, P., Maddalo, M., Russi, E. G., & Orlandi, E. M. (2017) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2017.01.008 · Cited by 18
Guidance to Best Tools and Practices for Systematic Reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBJS Reviews · doi:10.2106/jbjs.rvw.23.00077 · Cited by 24
Brindisino, F., Girardi, G., Crestani, M., Fiore, A., Giovannico, G., Garzonio, F., Venturin, D., & Struyf, F. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2251880 · Cited by 17
“Consistent with the recommendations of the Cochrane Collaboration [27,28], only RCTs were eligible for inclusion in the current study.”
Carbonation as a sensory enhancement strategy: a narrative synthesis of existing evidence
Turkington, L., Ward, E. C., & Farrell, A. M. (2016) · Disability and Rehabilitation · doi:10.1080/09638288.2016.1213894 · Cited by 16
“It is acknowledged that there are multiple methods for assessing research quality of non-randomized and uncontrolled studies (such as included in this synthesis), but there remains no one ideal tool.[23] Hence, the quality process selected for this review was chosen as it enabled appraisal of evidence other than rando…”
MethodSterile processing in low- and middle-income countries: an integrative review
Cuncannon, A., Dosani, A., & Fast, O. (2020) · Journal of Infection Prevention · doi:10.1177/1757177420947468 · Cited by 20
“The studies included in this review are susceptible to numerous biases (Boutron et al., 2019; Higgins et al., 2013).”
BackgroundOutbreaks of publications about emerging infectious diseases: the case of SARS-CoV-2 and Zika virus
Ipekci, A. M., Buitrago‐García, D., Meili, K. W., Krauer, F., Prajapati, N., Thapa, S., Wildisen, L., Araujo-Chaveron, L., Baumann, L., Shah, S., et al. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01244-7 · Cited by 16
Mei, F., Yao, M., Wang, Y., Huan, J., Ma, Y., Li, G., Zou, K., Li, L., & Sun, X. (2024) · BMC Medicine · doi:10.1186/s12916-024-03778-1 · Cited by 15
“An important caveat to this finding is that combining effect estimates across studies is rarely justified, as estimated effects for NRSIs with different study design characteristics may be influenced by different sources of bias [23, 32].”
Quintanilla, B. P. A., Taft, A., McDonald, S., Pollock, W., & Roque-Henriquez, J. C. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-013270 · Cited by 17
“However, evaluating the risk of bias and methodological quality of observational studies might be problematic, and there is no consensus particularly for evaluating risk of bias.(79) According to Grading of Recommendations, Assessment, Development and Evaluation (GRADE)”
BackgroundIncorporating Both Randomized and Observational Data into a Single Analysis
Kaizar, E. E. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020249 · Cited by 15
Zamora, N. V., Christensen, R., Goel, N., Klokker, L., Lopez-Olivo, M. A., Kristensen, L. E., Carmona, L., Strand, V., Curtis, J. R., & Suarez-Almazor, M. E. (2017) · The Journal of Rheumatology · doi:10.3899/jrheum.161108 · Cited by 15
Faggion, C. M., Huda, F., & Wasiak, J. (2014) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.12251 · Cited by 13
Prina, E., Tedeschi, F., Salazzari, D., Botte, T., Ballarin, M., Rabbi, L., Imperadore, G., Roccato, S., Nicolaou, S., Ruggeri, M., et al. (2023) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796023000100 · Cited by 12
“A ‘difference in differences’ (Higgins et al., 2013) approach was used: in particular, the difference of the number of contacts between 2019 and 2020 on the weeks of lockdown and intermediate restrictions was compared with the same difference in weeks of no or reduced restrictions, and such difference was interpreted…”
Cheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.02.008 · Cited by 13
Girolamo, N. D., & Mans, C. (2018) · Elsevier eBooks · doi:10.1016/b978-0-323-55228-8.00011-4 · Cited by 12
Geidl, W., Schlesinger, S., Mino, E., Miranda, L., Ryan, A., Bartsch, K. M., Janz, L., & Pfeifer, K. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-028653 · Cited by 11
“risk of bias assessment Assessment of bias across the included studies is very important, as the results can affect the variability among single studies and consequently, the meta-analysis.(21) We will use the Cochrane Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) for assessing bias.”
MethodThe Methodological Quality Scale (MQS) for intervention programs: validity evidence
Moscoso, S. C., Sanduvete‐Chaves, S., Lozano-Lozano, J. A., & Tello, F. P. H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1217661 · Cited by 11
“In most cases, these professionals are not experts in methodological issues, and when they seek out an instrument to gauge MQ, they encounter a wide variety of them (Higgins et al., 2013).”
BackgroundChim, K., Lai, J. T. C., & Chan, B. T. Y. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1450192 · Cited by 16
Drahota, A., Felix, L. M., Raftery, J. P., Keenan, B. E., Lachance, C. C., Mackey, D. C., Markham, C. M., Laing, A. C., Farrell, K. R. C., & Okunribido, O. O. (2022) · Health Technology Assessment · doi:10.3310/zowl2323 · Cited by 11
Yamamoto, S., Ishii, D., Noguchi, A., Tanamachi, K., Okamoto, Y., Takasaki, Y., Miyata, K., Fujita, Y., Kishimoto, H., Hotta, K., et al. (2019) · International Quarterly of Community Health Education · doi:10.1177/0272684x19896732 · Cited by 11
Metadata
- Authors
- Higgins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G.
- Affiliations
- MRC Biostatistics Unit (GB)
- University of Aberdeen (GB)
- Bristol Royal Infirmary (GB)
- University of Birmingham (GB)
- University of Ottawa (CA)
- University of Louisville (US)
- Corresponding author
- Higgins, Julian PT
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2012-09-25
- Volume
- 4
- Issue
- 1
- Pages
- 12-25
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2062040186
- Cited by
- 183
- References
- 31
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 17 · 2023: 23 · 2022: 8 · 2021: 17 · 2020: 18 · 2019: 21 · 2018: 10 · 2017: 16 · 2016: 11 · 2015: 11 · 2014: 11 · 2013: 9 · 2012: 1
- FWCI
- 2.82
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- Scottish Government
- National Institute for Health and Care Research
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- non-randomized studies, risk of bias, study design classification, intervention effects, observational study design, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
