Checklists of methodological issues for review authors to consider when including non‐randomized studies in systematic reviews
Abstract
Background: There is increasing interest from review authors about including non‐randomized studies (NRS) in their systematic reviews of health care interventions. This series from the Ottawa Non‐Randomized Studies Workshop consists of six papers identifying methodological issues when doing this. Aim: To format the guidance from the preceding papers on study design and bias, confounding and meta‐analysis, selective reporting, and applicability/directness into checklists of issues for review authors to consider when including NRS in a systematic review. Checklists: Checklists were devised providing frameworks to describe/assess: (1) study designs based on study design features; (2) risk of residual confounding and when to consider meta‐analysing data from NRS; (3) risk of selective reporting based on the Cochrane framework for detecting selective outcome reporting in trials but extended to selective reporting of analyses; and (4) directness of evidence contributed by a study to aid integration of NRS findings into summary of findings tables. Summary: The checklists described will allow review groups to operationalize the inclusion of NRS in systematic reviews in a more consistent way. The next major step is extending the existing Cochrane Risk of Bias tool so that it can assess the risk of bias to NRS included in a review. Copyright © 2013 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most common reason given for writing a protocol was to reduce bias in the selection of the studies by pre-specifying the study selection criteria.
“The most common reason given for writing a protocol was to reduce bias in the selection of the studies by pre-specifying the study selection criteria [37, 38, 40, 42].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
The articles recommended that the protocol should state the objectives, hypotheses to be tested and the rationale of the review, describe eligibility criteria, define the type of studies to be included, and give the reason when including observational studies in the review.
“The articles recommended that the protocol should state the objectives, hypotheses to be tested and the rationale of the review, [29] and that it should describe eligibility criteria [29, 33, 35, 36, 38, 39, 41], define the type of studies to be included [35, 37, 42, 44], and give the reason when including observational studies in the review [35, 37, 40].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Sixteen articles (17%) provided recommendations on the key item "protocol development", with publication dates between 1994 and 2015 (median year 2009).
“Sixteen articles (17%) provided recommendations on the key item “protocol development” (Table 3), [29–44] with publication dates between 1994 and 2015 (median year of publication 2009).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Twenty-two articles (24%) provided recommendations on the key item "study eligibility", with publication dates between 1995 and 2016 (median year 2009).
“Twenty-two articles (24%) provided recommendations on the key item “study eligibility”, [12, 29, 30, 33, 34, 37–39, 41–43, 46–48, 54, 55, 61, 65, 70–73] with publication dates between 1995 and 2016 (median year of publication 2009).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Two recent articles advocated deciding on a case-by-case basis which study designs to include.
“Two recent articles advocated deciding on a case-by-case basis [37, 65].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Recommendations were made mainly on two topics: whether different study designs should be included in a systematic review; and whether different study designs should be analysed together in a single meta-analysis.
“Recommendations were made mainly on 2 topics: whether different study designs should be included in a systematic review; [29, 34, 37–39, 49, 56, 65, 73, 75] and whether different study designs should be analysed together in a single meta-analysis [8, 34, 35, 37, 38, 42, 49, 51, 56, 61, 70, 75–78].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Twenty-five (27%) articles gave recommendations on item "considering different study designs", with publication dates between 1995 and 2016 (median year 2007).
“Twenty-five (27%) articles gave recommendations on item “considering different study designs” [8, 29, 34, 35, 37–39, 42, 44, 49, 51, 56, 61, 65, 70, 73, 75, 76, 77–83], with publication dates between 1995 and 2016 (median year of publication 2007).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Thirty-nine articles (42%) made recommendations on the assessment of quality and risk of bias of individual studies, with publication dates between 1994 and 2016 (median year 2009).
“Thirty-nine articles (42%) made recommendations on the assessment of quality and risk of bias of individual studies, [12, 16, 29, 30, 32–35, 37, 38, 43, 47–50, 52–54, 56, 64–66, 70, 71, 73, 75, 77, 82, 84–94] with publication dates between 1994 and 2016 (median year of publication 2009).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Many authoritative articles providing sound advice exist, such as those by Greenland, Egger et al., and a series in Research Synthesis Methods.
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [37]
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · see citing works
Self-citationAssessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · see citing works
Self-citationSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · see citing works
Including Non‐Randomized Studies
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., & Wells, G. A. (2008) · Wiley eBooks · doi:10.1002/9780470712184.ch13 · Cited by 306 · see citing works
Self-citationSchünemann, H. J., Tugwell, P., Reeves, B. C., Akl, E. A., Santesso, N., Spencer, F. A., Shea, B., Wells, G., & Helfand, M. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1078 · Cited by 248 · see citing works
Self-citationRubin, D. B. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.033 · Cited by 211 · see citing works
Rubin, D. B. (1991) · Biometrics · doi:10.2307/2532381 · Cited by 203 · see citing works
Higgins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1056 · Cited by 183 · see citing works
Self-citationValentine, J. C., & Thompson, S. G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1064 · Cited by 130 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 112 · see citing works
Self-citationNorris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50 · see citing works
Self-citationNorris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y. K., Garner, S. E., Anderson, L., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1074 · Cited by 36 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Non-alcoholic fatty liver disease and risk of incident cardiovascular disease: A meta-analysis
Targher, G., Byrne, C. D., Lonardo, A., Zoppini, G., & Barbui, C. (2016) · Journal of Hepatology · doi:10.1016/j.jhep.2016.05.013 · Cited by 1,429 · see citing works
Incidence and Prevalence of Drug-Resistant Epilepsy
Sultana, B., Panzini, M.-A., Carpentier, A. V., Comtois, J., Rioux, B., Gore, G. C., Bauer, P. R., Kwon, C., Jetté, N., Josephson, C. B., et al. (2021) · Neurology · doi:10.1212/wnl.0000000000011839 · Cited by 442 · see citing works
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348 · see citing works
Rauch, B., Davos, C. H., Doherty, P., Saure, D., Metzendorf, M., Salzwedel, A., Völler, H., Jensen, K., & Schmid, J. (2016) · European Journal of Preventive Cardiology · doi:10.1177/2047487316671181 · Cited by 318 · see citing works
Salzwedel, A., Jensen, K., Rauch, B., Doherty, P., Metzendorf, M., Hackbusch, M., Völler, H., Schmid, J., & Davos, C. H. (2020) · European Journal of Preventive Cardiology · doi:10.1177/2047487320905719 · Cited by 279 · see citing works
Depression and anxiety in pregnancy during COVID-19: A rapid review and meta-analysis
Tomfohr‐Madsen, L. M., Racine, N. M., Giesbrecht, G. F., Lebel, C. A., & Madigan, S. L. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.113912 · Cited by 248 · see citing works
Inflammatory markers and the risk of dementia and Alzheimer's disease: a meta-analysis
Darweesh, S. K. L., Wolters, F. J., Ikram, M. A., de Wolf, F., Bos, D., & Hofman, A. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2018.02.014 · Cited by 244 · see citing works
Bos, D., Wolters, F. J., Darweesh, S. K. L., Vernooij, M. W., de Wolf, F., Ikram, M. A., & Albert Hofman, D. (2018) · Alzheimer s & Dementia · doi:10.1016/j.jalz.2018.04.007 · Cited by 194 · see citing works
Amplification with hearing aids for patients with tinnitus and co-existing hearing loss.
Hoare, D. J., Edmondson-Jones, M., Sereda, M., Akeroyd, M. A., & Hall, D. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010151.pub2 · Cited by 170 · see citing works
Including non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch24 · Cited by 167 · see citing works
Self-citationPlatelet transfusion: a systematic review of the clinical evidence
Kumar, A., Mhaskar, R. S., Grossman, B. J., Kaufman, R. M., Tobian, A. A. R., Kleinman, S. H., Gernsheimer, T., Tinmouth, A. T., Djulbegović, B., & Panel, A. P. T. G. (2014) · Transfusion · doi:10.1111/trf.12943 · Cited by 165 · see citing works
Updated Method Guidelines for Cochrane Musculoskeletal Group Systematic Reviews and Metaanalyses
Ghogomu, E. A. T., Maxwell, L. J., Buchbinder, R., Rader, T., Pardo, J. P., Johnston, R. V., Christensen, R., Rutjes, A. W., Winzenberg, T. M., Singh, J. A., et al. (2013) · The Journal of Rheumatology · doi:10.3899/jrheum.121306 · Cited by 150 · see citing works
Self-citationBlood Pressure Variability and Cerebral Small Vessel Disease
Ma, Y., Song, A., Viswanathan, A., Blacker, D., Vernooij, M. W., Albert Hofman, D., & Papatheodorou, S. (2019) · Stroke · doi:10.1161/strokeaha.119.026739 · Cited by 147 · see citing works
Yetley, E. A. D., MacFarlane, A. J., Greene-Finestone, L. S., Garza, C., Ard, J. D., Atkinson, S. A., Bier, D. M., Carriquiry, A. L., Harlan, W. R., Hattis, D. B., et al. (2016) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.116.139097 · Cited by 124 · see citing works
Self-citationVerde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123 · see citing works
Touma, L., Dansereau, B., Chan, A. Y., Jetté, N., Kwon, C., Braun, K. P. J., Friedman, D., Jehi, L. E., Rolston, J. D., Vadera, S., et al. (2022) · Epilepsia · doi:10.1111/epi.17243 · Cited by 124 · see citing works
Sarri, G., Patorno, E., Yuan, H., Guo, J., Bennett, D., Wen, X., Zullo, A. R., Largent, J. A., Panaccio, M. P., Gokhale, M., et al. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111493 · Cited by 120 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 112 · see citing works
Self-citationThe Effects of Preoperative Embolization on Carotid Body Paraganglioma Surgery
Jackson, R. S., Myhill, J. A., Padhya, T. A., McCaffrey, J. C., McCaffrey, T. V., & Mhaskar, R. S. (2015) · Otolaryngology · doi:10.1177/0194599815605323 · Cited by 107 · see citing works
Physical Activity Preferences Among Older Adults: A Systematic Review
Amireault, S., Baier, J. M., & Spencer, J. R. (2017) · Journal of Aging and Physical Activity · doi:10.1123/japa.2017-0234 · Cited by 106 · see citing works
Zhang, H., Wang, Y., Lyu, D., Li, Y., Li, W., Wang, Q., Li, Y., Qin, Q., Wang, X., Gong, M., et al. (2021) · Ageing Research Reviews · doi:10.1016/j.arr.2021.101450 · Cited by 93 · see citing works
Upala, S., Jaruvongvanich, V., Wijarnpreecha, K., & Sanguankeo, A. (2016) · QJM · doi:10.1093/qjmed/hcw048 · Cited by 89 · see citing works
Risk Factors for Cholangiocarcinoma in Thailand: A Systematic Review and Meta-Analysis
Kamsa‐ard, S., Kamsa‐ard, S., Luvira, V., Suwanrungruang, K., Vatanasapt, P., & Wiangnon, S. (2018) · Europe PMC (PubMed Central) · doi:10.22034/apjcp.2018.19.3.605 · Cited by 86 · see citing works
Generic immunosuppression in solid organ transplantation: systematic review and meta-analysis
Molnar, A. O., Fergusson, D. A., Tsampalieros, A. K., Bennett, A., Fergusson, N. A., Ramsay, T., & Knoll, G. A. (2015) · BMJ · doi:10.1136/bmj.h3163 · Cited by 85 · see citing works
Upala, S., Panichsillapakit, T., Wijarnpreecha, K., Jaruvongvanich, V., & Sanguankeo, A. (2016) · Transplant International · doi:10.1111/tri.12721 · Cited by 81 · see citing works
Barrón-Cabrera, E., Ramos-López, O., González-Becerra, K., Riezu-Boj, J., Milagro, F., Martínez-López, E., & Martínez, J. A. (2019) · Lifestyle Genomics · doi:10.1159/000503289 · Cited by 64 · see citing works
Abramowicz, D., Hazzan, M., Maggiore, U., Peruzzi, L., Cochat, P., Oberbauer, R. W., Haller, M. C., & Biesen, W. V. (2015) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfv378 · Cited by 82 · see citing works
Septoplasty for nasal obstruction due to a deviated nasal septum in adults: a systematic review
van Egmond, M. M. H. T., Rovers, M. M., Tillema, A. H. J., & van Heerbeek, N. (2018) · Rhinology Journal · doi:10.4193/rhin18.016 · Cited by 82 · see citing works
Subclinical Hypothyroidism and Depression: A Systematic Review and Meta-Analysis
Tang, R., Wang, J., Yang, L., Ding, X., Zhong, Y., Pan, J., Yang, H., Mu, L., Chen, X., & Chen, Z. (2019) · Frontiers in Endocrinology · doi:10.3389/fendo.2019.00340 · Cited by 80 · see citing works
Barbui, C., Conti, V., & Cipriani, A. (2014) · Drug Safety · doi:10.1007/s40264-013-0127-6 · Cited by 75 · see citing works
Suwalska, J., Napierała, M., Bogdański, P., Łojko, D., Wszołek, K., Suchowiak, S., & Suwalska, A. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10112406 · Cited by 74 · see citing works
Potter, S., Browning, D., Savović, J., Holcombe, C., & Blazeby, J. M. (2015) · British journal of surgery · doi:10.1002/bjs.9804 · Cited by 72 · see citing works
Sumner, J., Harrison, A. S., & Doherty, P. (2017) · PLoS ONE · doi:10.1371/journal.pone.0177658 · Cited by 70 · see citing works
Predictors of Patency after Balloon Angioplasty in Hemodialysis Fistulas: A Systematic Review
Neuen, B. L., Gunnarsson, R. K., Webster, A. C., Baer, R. A., Golledge, J., & Mantha, M. L. (2014) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2014.02.010 · Cited by 63 · see citing works
Best Practices for Chiropractic Care for Older Adults: A Systematic Review and Consensus Update
Hawk, C., Schneider, M., Haas, M., Katz, P. R., Dougherty, P. E., Gleberzon, B. J., Killinger, L. Z., & Weeks, J. (2017) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2017.02.001 · Cited by 60 · see citing works
Poppe, K. G., Autin, C., Veltri, F., Kleynen, P., Grabczan, L., Rozenberg, S., & Ameye, L. (2018) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2017-02633 · Cited by 60 · see citing works
Systematic review and narrative synthesis of surgeons' perception of postoperative outcomes and risk
Dilaver, N., Gwilym, B. L., Preece, R., Twine, C. P., & Bosanquet, D. C. (2019) · BJS Open · doi:10.1002/bjs5.50233 · Cited by 60 · see citing works
Kevat, P. M., Reeves, B., Ruben, A. R., & Gunnarsson, R. K. (2017) · Current Cardiology Reviews · doi:10.2174/1573403x13666170116120828 · Cited by 58 · see citing works
Self-citationSteroid Refractory Chronic Graft-Versus-Host Disease: Cost-Effectiveness Analysis
Yalniz, F., Murad, M. H., Lee, S. J., Pavletic, S. Z., Khera, N., Shah, N. D., & Hashmi, S. K. (2018) · Transplantation and Cellular Therapy · doi:10.1016/j.bbmt.2018.03.008 · Cited by 56 · see citing works
The comorbid relationship between migraine and epilepsy: a systematic review and meta‐analysis
Keezer, M. R., Bauer, P. R., Ferrari, M. D., & Sander, J. W. A. S. (2014) · European Journal of Neurology · doi:10.1111/ene.12612 · Cited by 54 · see citing works
Berild, J. D., Winje, B. A., Vestrheim, D. F., Slotved, H., Valentiner‐Branth, P., Roth, A., & Storsæter, J. (2020) · Pathogens · doi:10.3390/pathogens9040259 · Cited by 51 · see citing works
Liu, J., Jin, X., Liu, W., Chen, W., Wang, L., Feng, Z., & Huang, J. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1131251 · Cited by 50 · see citing works
Observational evidence and strength of evidence domains: case examples
O'Neil, M. E., Berkman, N. D., Hartling, L., Chang, S., Anderson, J., Motu’apuaka, M. L., Guise, J., & McDonagh, M. S. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-35 · Cited by 51 · see citing works
Interventions to enhance maternal awareness of decreased fetal movement: a systematic review
Winje, B., Wojcieszek, A. M., Gonzalez‐Angulo, L., Teoh, Z., Norman, J. E., Frøen, J. F., & Flenady, V. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13802 · Cited by 49 · see citing works
Roorda, D., Königs, M., Schattenkerk, L. D. E., van der Steeg, L., van Heurn, E. L. W., & Oosterlaan, J. (2021) · Archives of Disease in Childhood Fetal & Neonatal · doi:10.1136/archdischild-2021-322158 · Cited by 46 · see citing works
Acute carotid stenting in patients undergoing thrombectomy: a systematic review and meta-analysis
Dufort, G., Chen, B. Y., Jacquin, G., Keezer, M. R., Labrie, M., Rioux, B., Stapf, C., Ziegler, D., & Poppe, A. Y. (2020) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2020-015817 · Cited by 45 · see citing works
Sanz‐Martín, I., Cha, J., Sanz‐Sánchez, I., Figuero, E., Herrera, D., & Sanz, M. (2021) · Clinical Oral Implants Research · doi:10.1111/clr.13840 · Cited by 45 · see citing works
Best Practices for Chiropractic Care of Children: A Consensus Update
Hawk, C., Schneider, M., Vallone, S., & Hewitt, E. G. (2016) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2016.02.015 · Cited by 42 · see citing works
Risk Factors for Recurrent Intussusception in Children: A Systematic Review and Meta-Analysis
Ye, X., Tang, R., Chen, S., Lin, Z., & Zhu, J. (2019) · Frontiers in Pediatrics · doi:10.3389/fped.2019.00145 · Cited by 42 · see citing works
Ijaz, S., Verbeek, J. H. A. M., Tikka, C., & Ruotsalainen, J. H. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.01.001 · Cited by 41 · see citing works
Establishment of a typology for startups 4.0
Kaczam, F., Siluk, J. C. M., Guimarães, G. E., de Moura, G. L., da Silva, W. V., & da Veiga, C. P. (2021) · Review of Managerial Science · doi:10.1007/s11846-021-00463-y · Cited by 40 · see citing works
Primary Aldosteronism and Bone Metabolism: A Systematic Review and Meta-Analysis
Shi, S., Lu, C., Tian, H., Ren, Y., & Chen, T. (2020) · Frontiers in Endocrinology · doi:10.3389/fendo.2020.574151 · Cited by 39 · see citing works
Xia, H., Zhang, W., Zhang, Y., Shang, X., Liu, Y., & Wang, X. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.978069 · Cited by 38 · see citing works
Agnes, A., Lirosi, M. C., Panunzi, S., Santocchi, P., Persiani, R., & D’Ugo, D. (2018) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2018.01.006 · Cited by 37 · see citing works
Improving Causal Inferences in Meta-analyses of Longitudinal Studies: Spanking as an Illustration
Larzelere, R. E., Gunnoe, M. L., & Ferguson, C. J. (2018) · Child Development · doi:10.1111/cdev.13097 · Cited by 37 · see citing works
Ekås, G. R., Ardern, C. L., Grindem, H., & Engebretsen, L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-100956 · Cited by 35 · see citing works
Liu, T., Nie, X., Wu, Z., Zhang, Y., Feng, G., Cai, S., Lv, Y., & Peng, X. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0454-x · Cited by 33 · see citing works
Hemostatic Abnormalities in Dementia: A Systematic Review and Meta-Analysis
Loures, C. D. M. G., Duarte, R. C. F., Silva, M. V. F., Cicarini, W. B., de Souza, L. C., Caramelli, P., Borges, K. B. G., & Ferriani, M. D. G. C. (2019) · Seminars in Thrombosis and Hemostasis · doi:10.1055/s-0039-1688444 · Cited by 31 · see citing works
The delivery of chemotherapy at home: an evidence synthesis
Corbett, M. S., Heirs, M. K., Rose, M., Smith, A. M., Stirk, L., Richardson, G. A., Stark, D., Swinson, D., Craig, D., & Eastwood, A. J. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03140 · Cited by 30 · see citing works
Immunosuppression in the elderly renal allograft recipient: a systematic review
Montero, N., Pérez‐Sáez, M. J., Pascual, J., Abramowicz, D., Budde, K., Dudley, C., Hazzan, M., Klinger, M., Maggiore, U., Oberbauer, R. W., et al. (2016) · Transplantation Reviews · doi:10.1016/j.trre.2016.05.001 · Cited by 30 · see citing works
Personalized Dietary Recommendations Based on Lipid-Related Genetic Variants: A Systematic Review
Pérez-Beltrán, Y. E., Rivera-Iñiguez, I., Gonzalez-Becerra, K., Pérez-Naitoh, N., Tovar, J., Sáyago-Ayerdi, S. G., & Mendivil, E. J. (2022) · Frontiers in Nutrition · doi:10.3389/fnut.2022.830283 · Cited by 28 · see citing works
Hormonal Contraception and the Risk of Breast Cancer in Women of Reproductive Age: A Meta-Analysis
la Roche, L. A. T.-D., Acevedo-Mesa, A., Lizarazo, I. L., Devassy, R., Becker, S., Krentel, H., & Wilde, R. L. D. (2023) · Cancers · doi:10.3390/cancers15235624 · Cited by 26 · see citing works
Tan, J. L., Lee, J. Z., Terrigno, V. R., Saracco, B., Saxena, S., Krathen, J., Hunter, K. M., Cha, Y., & Russo, A. M. (2021) · CJC Open · doi:10.1016/j.cjco.2021.05.019 · Cited by 23 · see citing works
Sikjær, M. G., Pedersen, A. A., Wik, M. S., Stensholt, S. S., Hilberg, O., & Løkke, A. (2023) · European Clinical Respiratory Journal · doi:10.1080/20018525.2023.2168354 · Cited by 23 · see citing works
Verde, P. E., Ohmann, C., Morbach, S., & Icks, A. (2015) · Statistics in Medicine · doi:10.1002/sim.6809 · Cited by 20 · see citing works
He, Y., Liu, Y., May, B. H., Zhang, A. L., Zhang, H., Lu, C., Yang, L., Guo, X., & Xue, C. C. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-018494 · Cited by 19 · see citing works
Gadager, B. B., Tang, L. H., Ravn, M. B., Doherty, P., Harrison, A. S., Christensen, J., Taylor, R. S., Zwisler, A., & Maribo, T. (2022) · BMC Cardiovascular Disorders · doi:10.1186/s12872-022-02723-5 · Cited by 20 · see citing works
Domenech-Briz, V., Gea-Caballero, V., Chover‐Sierra, E., Czapla, M., Fehler, P., Rodríguez-Calvo, A., Ballestar‐Tarín, M. L., Marín-Maicas, P., Cabellos-García, A. C., Pardo-Bosch, M., et al. (2024) · Nursing Reports · doi:10.3390/nursrep14010045 · Cited by 19 · see citing works
Becker, B. J., Aloe, A. M., Duvendack, M., Stanley, T. D., Valentine, J. C., Fretheim, A., & Tugwell, P. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.014 · Cited by 18 · see citing works
Self-citationNew meniscal tears after ACL injury: what is the risk? A systematic review protocol
Ekås, G. R., Ardern, C. L., Grindem, H., & Engebretsen, L. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-097728 · Cited by 18 · see citing works
Tsai, C.-C., Yang, P.-S., Liu, C., Wu, C., Hsu, Y., & Cheng, S. (2017) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2017.03.005 · Cited by 16 · see citing works
Schattenkerk, L. D. E., Musters, G. D., Nijssen, D. J., de Jonge, W. J., de Vries, R., van Heurn, E. L. W., & Derikx, J. P. M. (2021) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2021.01.018 · Cited by 15 · see citing works
Wang, Y., Yan, X., Zhan, J., Zhang, P., Zhang, G., Ge, S., Wen, H., Wang, L., Xu, N., & Lu, L. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.692942 · Cited by 16 · see citing works
Shao, L., Yan, X., Liu, C., Guo, C. J., & Cai, B. (2022) · Medicine · doi:10.1097/md.0000000000031945 · Cited by 15 · see citing works
Pontiroli, A., Centofanti, L., Le Roux, C., Magnani, S., Tagliabue, E., & Folli, F. (2023) · Nutrients · doi:10.3390/nu15040940 · Cited by 13 · see citing works
Pijpers, A. G. H., Schattenkerk, L. D. E., de Vries, R., Broers, C. J. M., Straver, B., van Heurn, E. L. W., Musters, G. D., Gorter, R. R., & Derikx, J. P. M. (2023) · Pediatric Surgery International · doi:10.1007/s00383-023-05449-3 · Cited by 13 · see citing works
Tang, R., Ye, X., Chen, S., Ding, X., Lin, Z., & Zhu, J. (2019) · Journal of Women s Health · doi:10.1089/jwh.2018.7636 · Cited by 13 · see citing works
Pontiroli, A. E., Scovenna, F., Carlini, V., Tagliabue, E., Martín-Delgado, J., Sala, L. L., Tanzi, E., & Zanoni, I. (2024) · Journal of Medical Virology · doi:10.1002/jmv.29343 · Cited by 12 · see citing works
Maxillary Sinus Augmentation with Autogenous Tooth Grafting Material: A Systematic Review
Ghodsian, D., D’Jesús, S., Sánchez‐Labrador, L., Cobo-Vázquez, C. M., Brinkmann, J. C., González, J. M. M., & Méniz-García, C. (2024) · Biomimetics · doi:10.3390/biomimetics9090518 · Cited by 14 · see citing works
Nédélec, E., Foli, E., Shultz, S. J., Swinton, P. A., Dolan, E., Enright, K. J., Piasecki, J., Matthews, J. J., Sale, C., & Elliott-Sale, K. J. (2021) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2021-001170 · Cited by 12 · see citing works
Cheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.02.008 · Cited by 12 · see citing works
Folli, F. B., Centofanti, L., Magnani, S., Tagliabue, E., Bignotto, M., Sala, L. L., & Pontiroli, A. E. (2023) · Journal of Endocrinological Investigation · doi:10.1007/s40618-023-02225-x · Cited by 11 · see citing works
Lee, J.-C., Ahn, S., Paik, K.-H., Kim, H. W., Kang, J., Kim, J., & Hwang, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010491 · Cited by 11 · see citing works
Baritello, O., Salzwedel, A., Sündermann, S., Niebauer, J., & Völler, H. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10194506 · Cited by 11 · see citing works
Larzelere, R. E., Gunnoe, M. L., Pritsker, J., & Ferguson, C. J. (2024) · Marriage & Family Review · doi:10.1080/01494929.2024.2392672 · Cited by 10 · see citing works
Múñoz, Ó., & Morales, Á. J. R. (2018) · Acta Médica Colombiana · doi:10.36104/amc.2018.1095 · Cited by 9 · see citing works
Vancomycin Dosing in Children With Overweight or Obesity: A Systematic Review and Meta-analysis
Khare, M., Azim, A., Kneese, G., Haag, M. B., Weinstein, K., Rhee, K. E., & Foster, B. A. (2020) · Hospital Pediatrics · doi:10.1542/hpeds.2019-0287 · Cited by 9 · see citing works
Schattenkerk, L. D. E., Musters, G. D., Nijssen, D. J., de Jonge, W. J., de Vries, R., van Heurn, E. L. W., & Derikx, J. P. M. (2021) · Innovative Surgical Sciences · doi:10.1515/iss-2020-0042 · Cited by 8 · see citing works
Cholangiocarcinoma Attributed to Occupation: A Systematic Reviews
Seeherunwong, A., Chaiear, N., Khuntikeo, N., & Ekpanyaskul, C. (2022) · Asian Pacific Journal of Cancer Prevention · doi:10.31557/apjcp.2022.23.6.1837 · Cited by 8 · see citing works
Lateral augmentation of the jaw by the split expansion ridge technique. A critical review
Artzi, Z. (2023) · Periodontology 2000 · doi:10.1111/prd.12527 · Cited by 8 · see citing works
Evaluating Treatment Options for Common Bovine Diseases Using Published Data and Clinical Experience
O'Connor, A. M. C., & Fajt, V. R. (2015) · Veterinary Clinics of North America Food Animal Practice · doi:10.1016/j.cvfa.2014.11.001 · Cited by 7 · see citing works
Kågesten, A., Tunçalp, Ö., Ali, M., Chandra‐Mouli, V., Tran, N. T., & Gülmezoglu, A. M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0138647 · Cited by 7 · see citing works
ECMO, ARDS and meta-analyses: Bayes to the rescue?
Graham, P. L., & Moran, J. L. (2020) · Journal of Critical Care · doi:10.1016/j.jcrc.2020.05.009 · Cited by 6 · see citing works
Moran, J. L., & Linden, A. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02215-4 · Cited by 5 · see citing works
Campbell, H., de Jong, V. M. T., Maxwell, L., Jaenisch, T., Debray, T. P. A., & Gustafson, P. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1515 · Cited by 4 · see citing works
Schattenkerk, L. D. E., Musters, G. D., Nijssen, D. J., de Jonge, W. J., de Vries, R., van Heurn, E. L. W., & Derikx, J. P. M. (2020) · Scientific Reports · doi:10.1038/s41598-020-77976-1 · Cited by 5 · see citing works
Global Prevalence of Duodenal Atresia in Trisomy 21: A Systematic Review and Meta-Analysis
Atiyat, D. K., Al-Nusair, D. A., Alhajahjeh, A. A., Al-Awadi, M. A., & Aborajooh, E. A. (2024) · European Journal of Pediatric Surgery · doi:10.1055/a-2471-6435 · Cited by 4 · see citing works
Wu, Q., Zhang, J., Lei, P., Zhu, X., & Huang, C. (2025) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2025.1498636 · Cited by 4 · see citing works
Singh, A. K., Dhungel, S., Kayastha, P. K., Roychoudhury, A., & Jose, A. (2021) · Advances in Oral and Maxillofacial Surgery · doi:10.1016/j.adoms.2021.100014 · Cited by 4 · see citing works
- Authors
- Wells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C.
- Affiliations
- University of Ottawa (CA)
- Institute of Population and Public Health (CA)
- MRC Biostatistics Unit (GB)
- University of Bristol (GB)
- Corresponding author
- Wells, George A
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2013-03
- Volume
- 4
- Issue
- 1
- Pages
- 63-77
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W1803108444
- Cited by
- 141
- References
- 14
- Citations by year
- 2026: 5 · 2025: 2 · 2024: 8 · 2023: 12 · 2022: 15 · 2021: 22 · 2020: 11 · 2019: 10 · 2018: 13 · 2017: 10 · 2016: 9 · 2015: 12 · 2014: 10 · 2013: 2
- FWCI
- 2.25
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- Scottish Government
- National Institute for Health and Care Research (NF-SI-0611-10168)
- Economic and Social Research Council (ES/G007543/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- non-randomized studies, checklists, residual confounding, selective reporting, summary of findings tables, risk of bias assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
