Non‐randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions
Open accessAbstract
The terms applicability, generalizability, external validity and transferability are related, sometimes used interchangeably and have in common that they lack a clear and consistent definition in the classic epidemiological literature. However, all of these terms generally describe one overarching theme: whether or not available research evidence can be directly utilized to answer the healthcare questions at hand, ideally supported by a judgment about the degree of confidence for this utilization. This concept has been called directness. The objectives of this paper were to delineate how non‐randomized studies (NRS) inform judgments in relation to directness and the concepts that it encompasses in the context of systematic reviews. We will briefly review what is known and describe the theoretical and practical issues as well as offer guidance to those tackling the challenges of judging directness and using research evidence to answer healthcare questions with evidence from NRS. In particular, we suggest a framework in which authors can use NRS as a complement, sequence or replacement for randomized controlled trials (RCTs) by focusing on judgments about the population, intervention, comparison and outcomes. Authors of systematic reviews will use NRS to complement judgments about the inconsistencies, the rationale and credibility of subgroup analysis, the baseline risk estimates for the determination of absolute benefits and downsides, and the directness of surrogate outcomes. This evidence includes contextual or supplementary evidence. Authors of systematic review and other summaries of the evidence use NRS as sequential evidence to provide evidence when insufficient evidence is available for an outcome from RCTs, but NRS evidence is available (e.g., long‐term harms). Use of evidence from NRS may also serve to replace RCT evidence when NRS provide equivalent (or potentially higher) confidence in the evidence (i.e. quality) compared to indirect evidence from RCTs. These judgments will be made in the context of other domains that influence the overall quality of the body of evidence, including the risk of bias, publication bias (i.e. limitations in the detailed study design and execution), inconsistency, imprecision and factors that increase our confidence in effects. This article will support systematic reviewers in their interaction with decision makers, that is, those who use the systematic review to develop guidelines, address health policy makers, and make clinical decisions, by making these judgments transparent. Copyright © 2013 John Wiley & Sons, Ltd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most common reason given for writing a protocol was to reduce bias in the selection of the studies by pre-specifying the study selection criteria.
“The most common reason given for writing a protocol was to reduce bias in the selection of the studies by pre-specifying the study selection criteria [37, 38, 40, 42].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [40]
Aspects frequently mentioned as important to address in the research question were the population, exposures or interventions, and outcomes.
“Aspects that were frequently mentioned as important to address were the population, exposures or interventions, and outcomes [38–41, 43, 47, 48, 50, 54].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [40]
Twenty articles (22%) gave recommendations on the key item "research question", with publication dates between 1997 and 2015 (median year 2009).
“Twenty articles (22%) gave recommendations on the key item “research question” [29, 30, 35, 36, 38–43, 45–54], with publication dates between 1997 and 2015 (median year of publication 2009).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [40]
Many authoritative articles providing sound advice exist, such as those by Greenland, Egger et al., and a series in Research Synthesis Methods.
“Although many authoritative articles providing sound advice exist, such as those included in this review by Greenland [85], Egger et al. [29, 97], and a series in Research Synthesis Methods [37, 40, 65, 71, 75], the inexperienced reviewer may find them difficult to identify among the many other papers with contradictory advice.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [40]
Few RCTs reporting type 2 diabetes remission as the primary outcome have been conducted.
“We first included RCTs reporting type 2 diabetes remission as the primary outcome, the design most likely to provide trustworthy evidence. However, as few such RCTs have been conducted, we also evaluated non-randomised studies (NRSs) to capture the totality of the evidence for ‘best available advice’ to inform practice and policy [20].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305 · see citing works
Self-citationGrading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477 · see citing works
Self-citationWhat is ‘quality of evidence’ and why is it important to clinicians?
Guyatt, G. H., Oxman, A. D., Kunz, R., Vist, G. E., Falck-Ytter, Y., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39490.551019.be · Cited by 3,250 · see citing works
Self-citationExternal validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · see citing works
GRADE guidelines: 2. Framing the question and deciding on important outcomes
Guyatt, G. H., Oxman, A. D., Kunz, R., Atkins, D., Brozek, J., Vist, G., Alderson, P., Glasziou, P., Falck-Ytter, Y., & Schünemann, H. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.012 · Cited by 1,911 · see citing works
Self-citationGRADE guidelines: 8. Rating the quality of evidence—indirectness
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Falck-Ytter, Y., Jaeschke, R., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.014 · Cited by 1,843 · see citing works
Self-citationOxman, A. D. (1993) · JAMA · doi:10.1001/jama.270.17.2093 · Cited by 656 · see citing works
Users' Guides to the Medical Literature
Oxman, A. D. (1993) · JAMA · doi:10.1001/jama.1993.03510170083036 · Cited by 475 · see citing works
Interpreting Results and Drawing Conclusions
Schünemann, H. J., Oxman, A. D., Vist, G. E., Higgins, J. P., Deeks, J. J., Glasziou, P., & Guyatt, G. H. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch12 · Cited by 366 · see citing works
Self-citationLetters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
Schünemann, H. J., Best, D., Vist, G. E., & Oxman, A. D. (2003) · PubMed Central · Cited by 289 · see citing works
Self-citationHiggins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1056 · Cited by 183 · see citing works
Self-citationAkl, E. A., Gunukula, S. K., Barba, M., Yosuico, V. E. D., van Doormaal, F. F., Kuipers, S., Middeldorp, S., Dickinson, H. O., Bryant, A., & Schünemann, H. J. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006652.pub3 · Cited by 119 · see citing works
Self-citationReeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 111 · see citing works
Self-citationTreatment Outcomes for Extensively Drug-Resistant Tuberculosis and HIV Co-infection
O’Donnell, M. R., Padayatchi, N., Kvasnovsky, C., Werner, L., Master, I., & Horsburgh, C. R. (2013) · Emerging infectious diseases · doi:10.3201/eid1903.120998 · Cited by 111 · see citing works
Atkins, D. C., Chang, S., Gartlehner, G., Buckley, D., Whitlock, E. P., Berliner, E., & Matchar, D. B. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.021 · Cited by 87 · see citing works
A Community-based Randomized Trial Encouraging Sun Protection for Children
Dietrich, A. J., Olson, A. L., Sox, C. H., Stevens, M. M., Tosteson, T. D., Ahles, T. A., Winchell, C. W., Grant‐Petersson, J., Collison, D. W., & Sanson‐Fisher, R. (1998) · PEDIATRICS · doi:10.1542/peds.102.6.e64 · Cited by 86 · see citing works
Blair, A. (2006) · Biometrics · doi:10.1111/j.1541-0420.2006.00589_4.x · Cited by 61 · see citing works
Applicability and Clinical Relevance of Results in Randomized Controlled Trials
Malmivaara, A. V., Koes, B. W., Bouter, L. M., & van Tulder, M. W. (2006) · Spine · doi:10.1097/01.brs.0000219868.30427.66 · Cited by 56 · see citing works
Uncertainties in baseline risk estimates and confidence in treatment effects
Spencer, F. A., Iorio, A., You, J. J., Murad, M. H., Schünemann, H. J., Vandvik, P. O., Crowther, M. A., Pottie, K. C., Lang, E., Meerpohl, J. J., et al. (2012) · BMJ · doi:10.1136/bmj.e7401 · Cited by 56 · see citing works
Self-citationAkl, E. A., Gunukula, S. K., Barba, M., Yosuico, V. E. D., van Doormaal, F. F., Kuipers, S., Middeldorp, S., Dickinson, H. O., Bryant, A., & Schünemann, H. J. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006652.pub2 · Cited by 31 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Rothman K, Greenland S. 1998. Modern Epidemiology (2nd edition). Philadelphia.
Cited by
Sorted by citations
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · see citing works
Self-citation2015 American College of Rheumatology Guideline for the Treatment of Rheumatoid Arthritis
Singh, J. A., Saag, K. G., Bridges, S. L., Akl, E. A., Bannuru, R. R., Sullivan, M. C., Vaysbrot, E. E., McNaughton, C., Osani, M. C., Shmerling, R. H., et al. (2015) · Arthritis & Rheumatology · doi:10.1002/art.39480 · Cited by 2,806 · see citing works
Self-citationInterventions to improve antibiotic prescribing practices for hospital inpatients
Davey, P. G., Brown, E. M., Charani, E., Fenelon, L. E., Gould, I. M., Holmes, A. H., Ramsay, C. R., Wiffen, P. J., & Wilcox, M. A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003543.pub3 · Cited by 1,872 · see citing works
2015 American College of Rheumatology Guideline for the Treatment of Rheumatoid Arthritis
Singh, J. A., Saag, K. G., Bridges, S. L., Akl, E. A., Bannuru, R. R., Sullivan, M. C., Vaysbrot, E. E., McNaughton, C., Osani, M. C., Shmerling, R. H., et al. (2015) · Arthritis Care & Research · doi:10.1002/acr.22783 · Cited by 1,113 · see citing works
Self-citationInterventions to improve antibiotic prescribing practices for hospital inpatients
Davey, P., Marwick, C. A., Scott, C. L., Charani, E., McNeil, K., Brown, E., Gould, I. M., Ramsay, C. R., & Michie, S. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003543.pub4 · Cited by 1,068 · see citing works
Completing ‘Summary of findings’ tables and grading the certainty of the evidence
Schünemann, H. J., Higgins, J. P., Vist, G. E., Glasziou, P., Akl, E. A., Skoetz, N., Guyatt, G. H., & on behalf of the Cochrane GRADEing Methods Group (formerly Applicability and Recommendations Methods Group) and the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch14 · Cited by 669 · see citing works
Self-citationInterpreting results and drawing conclusions
Schünemann, H. J., Vist, G. E., Higgins, J. P. T., Santesso, N. A., Deeks, J. J., Glasziou, P., Akl, E. A., Guyatt, G. H., & on behalf of the Cochrane GRADEing Methods Group (2019) · doi:10.1002/9781119536604.ch15 · Cited by 595 · see citing works
Self-citationvan der Molen, A. J., Reimer, P., Dekkers, I. A., Bongartz, G. M., Bellin, M., Bertolotto, M., Clémеnt, O., Heinz‐Peer, G., Stacul, F., Webb, J. A. W., et al. (2018) · European Radiology · doi:10.1007/s00330-017-5246-5 · Cited by 456 · see citing works
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Assessing risk of bias in a non‐randomized study
Sterne, J. A., Hernán, M. A., McAleenan, A., Reeves, B. C., & Higgins, J. P. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch25 · Cited by 330 · see citing works
Self-citationUsing systematic reviews in guideline development: The GRADE approach
Zhang, Y., Akl, E. A., & Schünemann, H. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1313 · Cited by 260 · see citing works
Self-citationIlić, D., Evans, S., Allan, C. A., Jung, J. H., Murphy, D. G., & Frydenberg, M. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009625.pub2 · Cited by 238 · see citing works
Händel, M. N., Cardoso, I., von Bülow, C., Rohde, J. F., Ussing, A., Nielsen, S. M., Christensen, R., Body, J., Brandi, M. L., Díez‐Pérez, A., et al. (2023) · BMJ · doi:10.1136/bmj-2021-068033 · Cited by 232 · see citing works
Santesso, N. A., Mustafa, R. A., Wiercioch, W., Kehar, R., Gandhi, S., Chen, Y., Cheung, A., Hopkins, J. P., Khatib, R., Ma, B., et al. (2015) · International Journal of Gynecology & Obstetrics · doi:10.1016/j.ijgo.2015.07.026 · Cited by 210 · see citing works
Self-citationChuruangsuk, C., Hall, J., Reynolds, A., Griffin, S. J., Combet, E., & Lean, M. E. J. (2022) · Diabetologia · doi:10.1007/s00125-021-05577-2 · Cited by 204 · see citing works
Memtsoudis, S. G., Cozowicz, C., Bekeris, J., Bekere, D., Liu, J., Soffin, E. M., Mariano, E. R., Johnson, R. L., Go, G., Hargett, M. J., et al. (2021) · Regional Anesthesia & Pain Medicine · doi:10.1136/rapm-2021-102750 · Cited by 205 · see citing works
Santesso, N. A., Carrasco‐Labra, A., Langendam, M. W., Brignardello‐Petersen, R., Mustafa, R. A., Heus, P., Lasserson, T. J., Opiyo, N., Kunnamo, I., Sinclair, D., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.006 · Cited by 176 · see citing works
Self-citationCuello‐García, C. A., Santesso, N. A., Morgan, R. L., Verbeek, J. H. A. M., Thayer, K. A., Ansari, M., Meerpohl, J. J., Schwingshackl, L., Katikireddi, S. V., Brożek, J., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.026 · Cited by 178 · see citing works
Self-citationCao, L., Yang, Z., Qi, L., & Chen, M. (2019) · Medicine · doi:10.1097/md.0000000000015770 · Cited by 173 · see citing works
Schünemann, H. J., Neumann, I., Hultcrantz, M., Brignardello‐Petersen, R., Zeng, L., Murad, M. H., Izcovich, A., Morgano, G. P., Baldeh, T., Santesso, N. A., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.07.015 · Cited by 171 · see citing works
Self-citationIncluding non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P. T., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Wiley eBooks · doi:10.1002/9781119536604.ch24 · Cited by 167 · see citing works
Self-citationTime to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’
Petticrew, M. P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0027-1 · Cited by 160 · see citing works
Combining randomized and non-randomized evidence in network meta-analysis
Efthimiou, O., Mavridis, D., Debray, T. P. A., Samara, M., Belger, M., Siontis, G. C. M., Leucht, S., Salanti, G., & 4, O. B. O. G. W. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7223 · Cited by 183 · see citing works
Wells, G. A., Shea, B., Higgins, J. P., Sterne, J., Tugwell, P., & Reeves, B. C. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1077 · Cited by 141 · see citing works
Self-citationHypovitaminosis D in bariatric surgery: A systematic review of observational studies
Chakhtoura, M., Nakhoul, N., Shawwa, K., Mantzoros, C. S., & Fuleihan, G. E. (2015) · Metabolism · doi:10.1016/j.metabol.2015.12.004 · Cited by 139 · see citing works
Yepes-Núñez, J. J., Brożek, J., Fiocchi, A. G., Pawankar, R., Cuello‐García, C. A., Zhang, Y., Morgano, G. P., Agarwal, A., Gandhi, S., Terracciano, L., et al. (2017) · Allergy · doi:10.1111/all.13241 · Cited by 135 · see citing works
Self-citationReichow, B., Servili, C., Yasamy, M. T., Barbui, C., & Saxena, S. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001572 · Cited by 135 · see citing works
Ilić, D., Evans, S., Allan, C. A., Jung, J. H., Murphy, D. G., & Frydenberg, M. (2017) · British Journal of Urology · doi:10.1111/bju.14062 · Cited by 129 · see citing works
Daring to draw causal claims from non-randomized studies of primary care interventions
Sourial, N., Longo, C., Vedel, I., & Schuster, T. (2018) · Family Practice · doi:10.1093/fampra/cmy005 · Cited by 126 · see citing works
Verde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123 · see citing works
Sarri, G., Patorno, E., Yuan, H., Guo, J., Bennett, D., Wen, X., Zullo, A. R., Largent, J. A., Panaccio, M. P., Gokhale, M., et al. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111493 · Cited by 120 · see citing works
Partial nephrectomy versus radical nephrectomy for clinical localised renal masses
Kunath, F., Schmidt, S., Krabbe, L., Miernik, A. J., Dahm, P., Cleves, A., Walther, M., & Kroeger, N. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012045.pub2 · Cited by 116 · see citing works
Reeves, B. C., Higgins, J. P. T., Ramsay, C. R., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 111 · see citing works
Self-citationThe ecosystem of health decision making: from fragmentation to synergy
Schünemann, H. J., Reinap, M., Piggott, T., Laidmäe, E., Köhler, K., Põld, M., Ens, B., Irs, A., Akl, E. A., Cuello‐García, C. A., et al. (2022) · The Lancet Public Health · doi:10.1016/s2468-2667(22)00057-3 · Cited by 109 · see citing works
Self-citationLopes‐Júnior, L. C., Bomfim, E. D. O., Olson, K. L., Neves, E. T., da Silveira, D. S. C., Nunes, M. D. R., Nascimento, L. C., Pereira-da-Silva, G., & de Lima, R. A. G. (2020) · BMJ · doi:10.1136/bmj.m4290 · Cited by 106 · see citing works
Frampton, G. K., Whaley, P., Bennett, M. G., Bilotta, G. S., Dorne, J. L. C. M., Eales, J. F., James, K. L., Kohl, C., Land, M., Livoreil, B., et al. (2022) · Environmental Evidence · doi:10.1186/s13750-022-00264-0 · Cited by 103 · see citing works
Intravesical gemcitabine for non-muscle invasive bladder cancer
Jones, G., Cleves, A., Wilt, T. J., MASON, M. D., Kynaston, H., & Shelley, M. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009294.pub2 · Cited by 88 · see citing works
Septoplasty for nasal obstruction due to a deviated nasal septum in adults: a systematic review
van Egmond, M. M. H. T., Rovers, M. M., Tillema, A. H. J., & van Heerbeek, N. (2018) · Rhinology Journal · doi:10.4193/rhin18.016 · Cited by 82 · see citing works
Peinemann, F., & Labeit, A. M. (2018) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12324 · Cited by 76 · see citing works
Lassi, Z. S., Kedzior, S. G. E., Tariq, W., Jadoon, Y., Das, J. K., & Bhutta, Z. A. (2020) · Nutrients · doi:10.3390/nu12030606 · Cited by 68 · see citing works
Arditi, C., Burnand, B., & Peytremann‐Bridevaux, I. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1816-5 · Cited by 68 · see citing works
Restrictions and their reporting in systematic reviews of effectiveness: an observational study
Helbach, J., Pieper, D., Mathes, T., Rombey, T., Zeeb, H., Allers, K., & Hoffmann, F. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01710-w · Cited by 75 · see citing works
Rojas, M. X., Granados, C., & Charry-Anzola, L. P. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005975.pub3 · Cited by 65 · see citing works
The TRANSFER Approach for assessing the transferability of systematic review findings
Munthe‐Kaas, H. M., Nøkleby, H., Lewin, S. A., & Glenton, C. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0834-5 · Cited by 64 · see citing works
Gershon, A. S., Lindenauer, P. K., Wilson, K. C., Rose, L., Walkey, A. J., Sadatsafavi, M., Anstrom, K. J., Au, D. H., Bender, B. G., Brookhart, M. A., et al. (2021) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202010-3943st · Cited by 63 · see citing works
Tarp, S., Furst, D. E., Boers, M., Luta, G., Bliddal, H., Tarp, U., Asmussen, K. H., Brock, B., Døssing, A., Jørgensen, T. S., et al. (2016) · Lara D. Veeken · doi:10.1093/rheumatology/kew442 · Cited by 63 · see citing works
Busenhart, D. M., Erb, J., Rigakos, G., Eliades, T., & Papageorgiou, S. N. (2018) · Oral Oncology · doi:10.1016/j.oraloncology.2018.06.001 · Cited by 58 · see citing works
A bibliometric analysis of global research output on network meta-analysis
Shi, J., Gao, Y., Liu, M., Yang, K., Sun, Y., Chen, J., Shi, S., Geng, J., Li, L., Wu, J., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01470-5 · Cited by 59 · see citing works
Single-dose intravesical chemotherapy after nephroureterectomy for upper tract urothelial carcinoma
Hwang, E. C., Sathianathen, N. J., Jung, J. H., Kim, M. H., Dahm, P., & Risk, M. C. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013160.pub2 · Cited by 53 · see citing works
O’Keeffe, C., & McNally, S. (2021) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-021-00286-3 · Cited by 76 · see citing works
Observational evidence and strength of evidence domains: case examples
O'Neil, M. E., Berkman, N. D., Hartling, L., Chang, S., Anderson, J., Motu’apuaka, M. L., Guise, J., & McDonagh, M. S. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-35 · Cited by 51 · see citing works
Interventions to enhance maternal awareness of decreased fetal movement: a systematic review
Winje, B., Wojcieszek, A. M., Gonzalez‐Angulo, L., Teoh, Z., Norman, J. E., Frøen, J. F., & Flenady, V. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13802 · Cited by 49 · see citing works
Amiodarone or lidocaine for cardiac arrest: A systematic review and meta-analysis
Sanfilippo, F., Corredor, C., Santonocito, C., Panarello, G., Arcadipane, A. F., Ristagno, G., & Pellis, T. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.07.235 · Cited by 48 · see citing works
External Validity: The Next Step for Systematic Reviews?
Avellar, S., Thomas, J., Kleinman, R., Sama-Miller, E., Woodruff, S. E., Coughlin, R. L., & Westbrook, T. R. (2016) · Evaluation Review · doi:10.1177/0193841x16665199 · Cited by 45 · see citing works
Jung, J. H., Reddy, B., McCutcheon, K. A., Borofsky, M. S., Narayan, V. M., Kim, M. H., & Dahm, P. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012832.pub2 · Cited by 43 · see citing works
Sanfilippo, F., Johnson, C., Bellavia, D., Morsolini, M., Romano, G., Santonocito, C., Centineo, L., Pastore, F., Pilato, M., & Arcadipane, A. (2017) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2017.02.046 · Cited by 30 · see citing works
Palmowski, A., Buttgereit, T., Palmowski, Y., Nielsen, S. M., Boers, M., Christensen, R., & Buttgereit, F. (2018) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2018.10.017 · Cited by 43 · see citing works
Dekkers, I. A., & van der Molen, A. J. (2018) · American Journal of Roentgenology · doi:10.2214/ajr.17.19499 · Cited by 42 · see citing works
Ijaz, S., Verbeek, J. H. A. M., Tikka, C., & Ruotsalainen, J. H. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.01.001 · Cited by 41 · see citing works
Mansell, R., & Purssell, E. (2017) · Journal of Advanced Nursing · doi:10.1111/jan.13476 · Cited by 41 · see citing works
Establishment of a typology for startups 4.0
Kaczam, F., Siluk, J. C. M., Guimarães, G. E., de Moura, G. L., da Silva, W. V., & da Veiga, C. P. (2021) · Review of Managerial Science · doi:10.1007/s11846-021-00463-y · Cited by 40 · see citing works
Systematic mapping of checklists for assessing transferability
Munthe‐Kaas, H. M., Nøkleby, H., & Nguyen, L. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0893-4 · Cited by 41 · see citing works
Zeraatkar, D., Bhasin, A. A., Morassut, R. E., Churchill, I. F., Gupta, A. A., Lawson, D. O., Miroshnychenko, A., Sirotich, E., Aryal, K., Mikhail, D., et al. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab002 · Cited by 39 · see citing works
Gardner, H., Albarqouni, L., Elfeky, A., Gillies, K., & Treweek, S. P. (2020) · F1000Research · doi:10.12688/f1000research.22182.1 · Cited by 37 · see citing works
Intravesical gemcitabine for non-muscle invasive bladder cancer
Han, M. A., Maisch, P., Jung, J. H., Hwang, J., Narayan, V. M., Cleves, A., Hwang, E. C., & Dahm, P. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009294.pub3 · Cited by 39 · see citing works
Surrogate endpoints: a key concept in clinical epidemiology
Christensen, R., Ciani, O., Manyara, A. M., & Taylor, R. S. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111242 · Cited by 38 · see citing works
Kietpeerakool, C., Aue-aungkul, A., Galaal, K. A., Ngamjarus, C., & Lumbiganon, P. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012828.pub2 · Cited by 38 · see citing works
Agnes, A., Lirosi, M. C., Panunzi, S., Santocchi, P., Persiani, R., & D’Ugo, D. (2018) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2018.01.006 · Cited by 37 · see citing works
Pottie, K. C., Lotfi, T., Kilzar, L., Howeiss, P., Rizk, N., Akl, E. A., Dias, S., Biggs, B., Christensen, R., Rahman, P., et al. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15081700 · Cited by 37 · see citing works
Self-citationEvaluation of root resorption following orthodontic intrusion: a systematic review and meta-analysis
Bellini‐Pereira, S. A., de Almeida, J. C., Castillo, A. A.-D., dos Santos, C. C. O., Henriques, J. F. C., & Janson, G. (2020) · European Journal of Orthodontics · doi:10.1093/ejo/cjaa054 · Cited by 37 · see citing works
Medical and surgical interventions for the treatment of urinary stones in children
Barreto, L., Jung, J. H., Abdelrahim, A., Ahmed, M., Dawkins, G. P. C., & Kaźmierski, M. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010784.pub2 · Cited by 36 · see citing works
Pereira, S. A. D. S., Cicareli, Á. J., Idalgo, F. A., Nunes, A. G., Kassis, E. N., Henriques, J. F. C., & Bellini‐Pereira, S. A. (2020) · Journal of Esthetic and Restorative Dentistry · doi:10.1111/jerd.12695 · Cited by 36 · see citing works
Biologics in severe asthma: the role of real-world evidence from registries
Paoletti, G., Pepys, J., Casini, M., Bona, D. D., Heffler, E. M., Goh, C. Y. Y., Price, D. B., & Canonica, G. W. (2022) · European Respiratory Review · doi:10.1183/16000617.0278-2021 · Cited by 36 · see citing works
Grillo-Ardila, C. F., Tibavizco-Palacios, D., Triana, L. C., Rugeles, S. J., Vallejo-Ortega, M. T., Calderón-Franco, C. H., & Ramírez-Mosquera, J. J. (2024) · Nutrients · doi:10.3390/nu16111560 · Cited by 35 · see citing works
Minozzi, S., Cinquini, M., Gianola, S., Castellini, G., Gerardi, C., & Banzi, R. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.001 · Cited by 35 · see citing works
Gordon, I., Rutherford, C., Makarounas‐Kirchmann, K., & Kirchmann, M. (2020) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2020.108158 · Cited by 36 · see citing works
Immediate sequential bilateral surgery versus delayed sequential bilateral surgery for cataracts
Dickman, M. M., Spekreijse, L. S., Winkens, B., Schouten, J., Simons, R. W. P., Dirksen, C. D., & Nuijts, R. M. M. A. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013270.pub2 · Cited by 34 · see citing works
Jung, J. H., Shin, T. Y., McCutcheon, K. A., Borofsky, M. S., Narayan, V. M., Young, S., Golzarian, J., Kim, M. H., Reddy, B., & Dahm, P. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012867 · Cited by 34 · see citing works
Cuello‐García, C. A., Morgan, R. L., Brożek, J., Santesso, N. A., Verbeek, J. H. A. M., Thayer, K. A., Guyatt, G. H., & Schünemann, H. J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.010 · Cited by 32 · see citing works
Self-citationAlvimopan for recovery of bowel function after radical cystectomy
Sultan, S., Coles, B., & Dahm, P. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012111.pub2 · Cited by 31 · see citing works
Using GRADE to respond to health questions with different levels of urgency.
Thayer, K., & Schunemann, H. (2016) · Environment International · doi:10.1016/j.envint.2016.03.027 · Cited by 32 · see citing works
Self-citationHwang, E. C., Jung, J. H., Borofsky, M. S., Kim, M. H., & Dahm, P. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013143.pub2 · Cited by 30 · see citing works
Kang, T. W., Jung, J. H., Hwang, E. C., Borofsky, M. S., Kim, M. H., & Dahm, P. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013251.pub2 · Cited by 29 · see citing works
van Die, M. D., Bone, K. M., Visvanathan, K., Kyrø, C., Aune, D., Ee, C., & Paller, C. J. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad104 · Cited by 29 · see citing works
Hidalgo, L. H., de Alarcón, A., López-Cortés, L. E., Márquez, R. L., López-Cortés, L. E., Gutiérrez‐Valencia, A., & Gil-Navarro, M. V. (2020) · Antibiotics · doi:10.3390/antibiotics9100657 · Cited by 29 · see citing works
Bröckelmann, N., Balduzzi, S., Harms, L., Beyerbach, J., Πετροπούλου, Μ., Kubiak, C., Wolkewitz, M., Meerpohl, J. J., & Schwingshackl, L. (2022) · BMC Medicine · doi:10.1186/s12916-022-02369-2 · Cited by 29 · see citing works
Jung, J. H., McCutcheon, K. A., Reddy, B., Borofsky, M. S., Narayan, V. M., Kim, M. H., & Dahm, P. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012832 · Cited by 28 · see citing works
Jung, J. H., McCutcheon, K. A., Borofsky, M. S., Young, S., Golzarian, J., Kim, M. H., Dahm, P., & Narayan, V. M. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012867.pub2 · Cited by 29 · see citing works
Using Systematic Reviews in Guideline Development
Schünemann, H. J. (2022) · doi:10.1002/9781119099369.ch22 · Cited by 27 · see citing works
Self-citationBerre, M. L., Presse, N., Morin, M., Larouche, M., Campeau, L., Hu, Y., Reid, I., & Dumoulin, C. L. (2020) · Neurourology and Urodynamics · doi:10.1002/nau.24344 · Cited by 29 · see citing works
Iheozor‐Ejiofor, Z., Lakunina, S., Gordon, M. A., Akintelure, D., Sinopoulou, V., & Akobeng, A. K. (2020) · United European Gastroenterology Journal · doi:10.1177/2050640620967899 · Cited by 27 · see citing works
Via, L. L., Astuto, M., Bignami, E. G., Busalacchi, D., Dezio, V., Girardis, M., Lanzafame, B., Ristagno, G., Pelosi, P. P., & Sanfilippo, F. (2022) · Minerva Anestesiologica · doi:10.23736/s0375-9393.22.16449-7 · Cited by 26 · see citing works
A meta-analysis of cranial electrotherapy stimulation in the treatment of depression
Price, L. R., Briley, J., Haltiwanger, S., & Hitching, R. (2020) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2020.12.043 · Cited by 27 · see citing works
Girolamo, N. D., & Reynders, R. M. (2016) · Draw Science · doi:10.7717/peerj.1649 · Cited by 25 · see citing works
Methodology of the Second Pediatric Acute Lung Injury Consensus Conference
Iyer, N. P., Khemani, R. G., Émériaud, G., Lopez-Fernandez, Y. M., Korang, S. K., Steffen, K. M., Barbaro, R. P., Bembea, M. M., & on behalf of the Second Pediatric Acute Lung Injury Consensus Conference (PALICC-2) Group on behalf and the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network (2023) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003160 · Cited by 25 · see citing works
Langendam, M. W., Carrasco‐Labra, A., Santesso, N. A., Mustafa, R. A., Brignardello‐Petersen, R., Ventresca, M., Heus, P., Lasserson, T. J., Moustgaard, R., Brożek, J., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.008 · Cited by 25 · see citing works
Self-citationMedical and surgical interventions for the treatment of urinary stones in children
Barreto, L., Jung, J. H., Abdelrahim, A., Ahmed, M., Dawkins, G. P. C., & Kaźmierski, M. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010784.pub3 · Cited by 25 · see citing works
Jung, A., Balzer, J., Braun, T., & Luedtke, K. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01561-5 · Cited by 22 · see citing works
Network Meta-Analysis for Complex Social Interventions: Problems and Potential
Grant, E. S., & Calderbank-Batista, T. (2013) · Journal of the Society for Social Work and Research · doi:10.5243/jsswr.2013.25 · Cited by 24 · see citing works
Shen, Y., Zhou, J., Zhang, S., Wang, X., Jia, Y., He, S., Wang, Y. Y., Li, W., Shao, J., Zhuang, X., et al. (2018) · Clinical Drug Investigation · doi:10.1007/s40261-018-0717-2 · Cited by 23 · see citing works
- Authors
- Schünemann, H. J., Tugwell, P., Reeves, B. C., Akl, E. A., Santesso, N., Spencer, F. A., Shea, B., Wells, G., & Helfand, M.
- Affiliations
- McMaster University (CA)
- Ottawa Hospital (CA)
- Bristol Royal Infirmary (GB)
- Oregon Health & Science University (US)
- Corresponding author
- Schünemann, Holger J.
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2013-03
- Volume
- 4
- Issue
- 1
- Pages
- 49-62
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2126629865
- Cited by
- 248
- References
- 20
- Citations by year
- 2026: 14 · 2025: 17 · 2024: 17 · 2023: 16 · 2022: 32 · 2021: 30 · 2020: 25 · 2019: 22 · 2018: 20 · 2017: 19 · 2016: 14 · 2015: 8 · 2014: 6 · 2013: 7 · 2012: 1
- FWCI
- 2.36
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- non-randomized studies, generalizability, applicability, transferability, surrogate endpoints, subgroup analysis, contextual evidence, risk of bias, publication bias, inconsistency, imprecision, quality of evidence, systematic review methodology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
