Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence
Open accessThe DOI matches its Crossref record; 95 of the 96 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). In the first paper, we reported findings for the four initial steps of an overview (specification of purpose, objectives and scope; eligibility criteria; search methods; data extraction). In this paper, we report the remaining steps: assessing risk of bias; synthesis, presentation and summary of the findings; and assessing certainty of the evidence arising from the overview. METHODS: In stage I, we identified cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews. Based on these studies, we developed a framework of possible methods for overviews, categorised by the steps in conducting an overview. Multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified studies evaluating methods and mapped these evaluations to the framework. RESULTS: Forty-two stage I studies described methods relevant to one or more of the latter steps of an overview. Six studies evaluating methods were included in stage II. These mapped to steps involving (i) the assessment of risk of bias (RoB) in SRs (two SRs and three primary studies, all reporting evaluation of RoB tools) and (ii) the synthesis, presentation and summary of the findings (one primary study evaluating methods for measuring overlap). CONCLUSION: Many methods have been described for use in the latter steps in conducting an overview; however, evaluation and guidance for applying these methods is sparse. The exception is RoB assessment, for which a multitude of tools exist-several with sufficient evaluation and guidance to recommend their use. Evaluation of other methods is required to provide a comprehensive evidence map.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The data extraction approaches outlined in the MOoR framework to manage discrepant data across systematic reviews involve retrieving either published or registry reports of the primary studies, or contacting systematic review or primary study authors, or both, for clarification regarding discrepancies [7, 16].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [16]
Experimental — accuracy not yet measuredIn the citing paper
“A systematic map of all methods used in overviews, called the MOoR framework, identified nine methods to manage the overlap used across four steps in the conduct of an overview [7, 16].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [16]
In the citing paper
“Discrepant and overlapping risk of bias assessments across systematic reviews can be resolved by: (a) extracting the risk of bias assessment of primary studies from the included systematic reviews, using data extraction approaches to manage missing, flawed assessments, or discrepant assessments of the same primary study; or (b) re-assessing all primary studies using a common risk of bias tool.”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [16]
In the citing paper
“However, approaches for assessing reporting bias in overviews of reviews have been catalogued.16”
From Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. (2022) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636 · cited 1×
“One recent example is the publication of AMSTAR 2 [41].”
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000055 · Cited by 2,587 · cited 3×
“…how to present the RoB assessments for SRs 1.3.1 Display assessments in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each SR) Aromataris 2015 [39, 59]; Becker [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77] 1.3.2 Display assessments…”
InfluentialROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
Systematic review finds overlapping reviews were not mentioned in every other overview
Pieper, D., Antoine, S.-L., Mathes, T., Neugebauer, E. A. M., & Eikermann, M. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.007 · Cited by 1,318 · cited 3×
“Of the four primary studies evaluating methods [15, 19, 34, 35]: (i) none referred to a study protocol or noted the existence of one, (ii) three used convenience samples as a method to select the sample of SRs to which the tool/measure was applied, (iii) the three studies that evaluated RoB tools either used a conveni…”
Meta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,250
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227 · cited 21×
“…in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each SR) Aromataris 2015 [39, 59]; Becker [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77] 1.3.2 Display assessments graphically Crick 2015 [48] ▪ ROBIS RoB graph depicting authors’…”
InfluentialThe Medical Review Article: State of the Science
MULROW, C. D. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-106-3-485 · Cited by 856
Validation of an index of the quality of review articles
Oxman, A. D., & Guyatt, G. H. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90160-b · Cited by 786
Users' Guides to the Medical Literature: VI. How to Use an Overview
Oxman, A. D. (1994) · JAMA · doi:10.1001/jama.1994.03520170077040 · Cited by 775
Guidelines for Meta-analyses Evaluating Diagnostic Tests
Irwig, L., Tosteson, A. N. A., Gatsonis, C., Lau, J., Colditz, G., Chalmers, T. C., & Mosteller, F. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-8-199404150-00008 · Cited by 753
Ioannidis, J. P. A. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081086 · Cited by 716 · cited 2×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the included SR(s), noting mis…”
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses)
Greenhalgh, T. (1997) · BMJ · doi:10.1136/bmj.315.7109.672 · Cited by 578
External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569
Guidelines for reading literature reviews.
Oxman, A. D., & Guyatt, G. H. (1988) · Europe PMC (PubMed Central) · Cited by 549
Oxman, A. D. (1994) · JAMA · doi:10.1001/jama.272.17.1367 · Cited by 486
Systematic Reviews: Checklists for review articles
Oxman, A. D. (1994) · BMJ · doi:10.1136/bmj.309.6955.648 · Cited by 480
The Science of Reviewing Researcha
OXMAN, A. D., & GUYATT, G. H. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26342.x · Cited by 472 · cited 1×
“Name of the included tools or measures (unnamed tools are identified by first author name and year of publication) AMSTAR [22, 23], R-AMSTAR [36] Named tools: AMSTAR [22, 23], CASP [83], FOCUS [84], MAC [85], NHMRC [86], OQAQ [28], SIGN [87], RAPiD [88] Unnamed tools: Assendelft 1995 [89], Auperin 1997 [90], Crombie 1…”
A guide to interpreting discordant systematic reviews.
Jadad, A., Cook, D. J., & Browman, G. P. (1997) · Europe PMC (PubMed Central) · Cited by 444
Kung, J. (2010) · The Open Dentistry Journal · doi:10.2174/1874210601004020084 · Cited by 329 · cited 2×
“Pieper [17] reviewed evidence of the reliability and construct validity of the AMSTAR [22, 23] and R-AMSTAR (revised-AMSTAR [36]) tools.”
Hunt, H., Pollock, A., Campbell, P., Estcourt, L., & Brunton, G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0695-8 · Cited by 314
Mokkink, L. B., Terwee, C. B., Gibbons, E., Stratford, P. W., Alonso, J., Patrick, D. L., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-82 · Cited by 285
Grading the strength of a body of evidence when assessing health care interventions: an EPC update
Berkman, N. D., Lohr, K. N., Ansari, M. T., Balk, E. M., Kane, R., McDonagh, M., Morton, S. C., Viswanathan, M., Bass, E. B., Butler, M., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.023 · Cited by 279
Assessment of the quality of meta-analysis in agronomy
Philibert, A., Loyce, C., & Makowski, D. (2012) · Agriculture, Ecosystems & Environment · doi:10.1016/j.agee.2011.12.003 · Cited by 259
A descriptive analysis of overviews of reviews published between 2000 and 2011.
Hartling, L., Chisholm, A., Thomson, D., & Dryden, D. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049667 · Cited by 241 · cited 8×
“3 (Re)-assess RoB of some or all primary studies CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Jadad 1997 [29]; Moja 2012 [63]; Thomson 2010 [26] ▪ When two different tools are used, then assess the primary studies using one tool ▪ When two different tools are used (e.”
InfluentialPatnode, C. D., Henderson, J. T., Thompson, J. H., Senger, C. A., Fortmann, S. P., & Whitlock, E. P. (2015) · Annals of Internal Medicine · doi:10.7326/m15-0171 · Cited by 238 · cited 2×
“Venn diagrams as referenced in Patnode [82]) 5.”
InfluentialBecker, L. A., & Oxman, A. D. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch22 · Cited by 218 · cited 19×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the…”
InfluentialOverviews of reviews often have limited rigor: a systematic review
Pieper, D., Buechter, R., Jerinic, P., & Eikermann, M. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.015 · Cited by 213 · cited 6×
“A second-order subgroup analysis was identified as a potential method Table 4 Synthesis, presentation and summary of the findings (Continued) Step Sub-step Methods/approaches Sources (first author, year) ▪ Examples 5.3.2 Use statistical approaches to deal with overlap Cooper 2012 [6]; Tang 2013 [78] ▪ Identify meta-an…”
InfluentialCooper, H., & Koenka, A. C. (2012) · American Psychologist · doi:10.1037/a0027119 · Cited by 196 · cited 43×
“…Determine the summary approach 1.2.1 Describe and/or tabulate the characteristics of the included SRs in terms of PICO elements Becker 2008 [4]; Cooper 2012 [6]; JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69–72]; Ryan 2009 [25]; Smith 2011 [77]; Thomson 2010 [26] ▪ Matrix of…”
InfluentialPollock, M., Fernandes, R. M., Becker, L. A., Featherstone, R., & Hartling, L. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0367-5 · Cited by 195
Mokkink, L. B., Terwee, C. B., Stratford, P. W., Alonso, J., Patrick, D. L., Riphagen, I., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9451-9 · Cited by 186
Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0617-1 · Cited by 184
Self-citationPollock, A., Farmer, S. E., Brady, M. C., Langhorne, P., Mead, G. E., Mehrholz, J., van Wijck, F., & Wiffen, P. J. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.08.013 · Cited by 181 · cited 4×
“For example, Pollock 2015 incorporated the limitations of included SRs in their GRADE assessment by rating down the certainty of evidence for SRs that did not meet criteria deemed to indicate important sources of bias [31, 32].”
InfluentialSystematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties
Pieper, D., Buechter, R. B., Li, L., Prediger, B., & Eikermann, M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.009 · Cited by 161 · cited 9×
“low, moderate, high quality) [12, 17] Adaptation of the step from SRs to overviews.”
InfluentialMethods for second order meta-analysis and illustrative applications
Schmidt, F. L., & Oh, I.-S. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.03.002 · Cited by 151 · cited 5×
“…meta-analysis of effect estimates (meta-analysis of the primary study effect estimates)a Becker 2008 [4]; Chen 2014 [46]; Cooper 2012 [6]; Pieper 2014a [17]; Robinson 2015 [24, 69–72]; Schmidt 2013 [74]; Tang 2013 [78]; Thomson 2010 [26] ▪ May re-extract data from the primary studies, or use the…”
InfluentialUsing Existing Systematic Reviews in Complex Systematic Reviews
Whitlock, E. P., Lin, J. S., Chou, R., Shekelle, P., & Robinson, K. A. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-10-200805200-00010 · Cited by 144 · cited 1×
“…of an odds ratio 2.2.4 Need to limit or expand the MAs into a new MA that meets the population, intervention and comparator elements of the overview Thomson 2010 [26]; Whitlock 2008 [24, 69–72] ▪ Extracting the subset of trials that include only children and adolescents from a MA that…”
InfluentialBallard, M., & Montgomery, P. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1229 · Cited by 136
Overviews of systematic reviews: great promise, greater challenge
McKenzie, J. E., & Brennan, S. E. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0582-8 · Cited by 134
Self-citationThe Relationship Between Methodological Quality and Conclusions in Reviews of Spinal Manipulation
Assendelft, W. J. J. (1995) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1995.03530240052041 · Cited by 132
The assessment of systematic reviews in dentistry
Glenny, A. M., Esposito, M., Coulthard, P., & Worthington, H. V. (2003) · European Journal of Oral Sciences · doi:10.1034/j.1600-0722.2003.00013.x · Cited by 121
The evolution of a new publication type: Steps and challenges of producing overviews of reviews
Thomson, D., Russell, K., Becker, L., Klassen, T., & Hartling, L. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.30 · Cited by 109 · cited 30×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the included SR(s), noting mis…”
InfluentialMethodology for jbi umbrella reviews
Aromataris, E., Fernandez, R. S., Godfrey, C. M., Holly, C. M., Khalil, H., & Tungpunkom, P. (2014) · Research Online (University of Wollongong) · Cited by 95
The Pocket Guide to Critical Appraisal: A Handbook for Health Care Professionals
Crombie, I. K. (1996) · Medical Entomology and Zoology · Cited by 89
A tool to assess the quality of a meta‐analysis
Higgins, J. P. T., Lane, P. W., Anagnostelis, B., Anzures‐Cabrera, J., Baker, N. F., Cappelleri, J. C., Haughie, S., Hollis, S., Lewis, S. C., Moneuse, P., et al. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1092 · Cited by 83
Caird, J., Sutcliffe, K., Kwan, I., Dickson, K., & Thomas, J. (2015) · Evidence & Policy · doi:10.1332/174426514x13988609036850 · Cited by 81 · cited 27×
“…intervals, study design, number of study participants (O’Mara 2011 [64]) 1.2.4 Summarise and/or tabulate RoB assessments of SRs and primary studies Becker 2008 [4]; Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59]; Li 2012 [62]; Ryan 2009 [25]; Robinson 2015 [24, 69–72];…”
InfluentialLi, T., Vedula, S. S., Scherer, R., & Dickersin, K. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-5-201203060-00009 · Cited by 78 · cited 6×
“…Summarise and/or tabulate RoB assessments of SRs and primary studies Becker 2008 [4]; Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59]; Li 2012 [62]; Ryan 2009 [25]; Robinson 2015 [24, 69–72]; Smith 2011 [77] ▪ For example, summarise the RoB/quality assessment methods…”
InfluentialThe Pocket Guide to Critical Appraisal: A Handbook for Health Care Professionals
Harvey, B. J. (1997) · Europe PMC (PubMed Central) · Cited by 74
Twelve recommendations for integrating existing systematic reviews into new reviews: EPC guidance
Robinson, K. A., Chou, R., Berkman, N. D., Newberry, S. J., Fu, R. F., Hartling, L., Dryden, D. M., Butler, M. E., Foisy, M., Anderson, J., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.035 · Cited by 67
Hartling, L., Vandermeer, B., & Fernandes, R. M. (2014) · Evidence-Based Child Health: A Cochrane Review Journal · doi:10.1002/ebch.1968 · Cited by 67 · cited 3×
“Becker 2008 [4]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; JBI 2014 [39, 59]; Kramer 2009 [61]; Pieper 2014c [66]; Robinson 2015 [24, 69–72]; Ryan 2009 [25]; Silva 2014 [75] ▪ Report assessments of the certainty of the evidence for each comparison and outcome directly from the…”
InfluentialLunny, C., McKenzie, J. E., & McDonald, S. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.002 · Cited by 66
Self-citationUp-to-dateness of reviews is often neglected in overviews: a systematic review
Pieper, D., Antoine, S.-L., Neugebauer, E. A. M., & Eikermann, M. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.08.008 · Cited by 65 · cited 2×
“1 Select an existing RoB assessment tool for SRs Baker 2014 [43]; Becker 2008 [4]; Bolland 2014 [5]; Büchter 2011 [45, 65]; Caird 2015 [1]; Chen 2014 [46]; CMIMG 2012 [47]; Cooper 2012 [6]; Flodgren 2011 [49]; Foisy 2011 [50]; Foisy 2014 [52]; Hartling 2012 [53]; Hartling 2013 [54]; Jadad 1997 [29]; James 2014 [58]; J…”
Quality and transparency of overviews of systematic reviews
Li, L., Tian, J., Tian, H., Sun, R., Liu, Y., & Yang, K. (2012) · Journal of Evidence-Based Medicine · doi:10.1111/j.1756-5391.2012.01185.x · Cited by 62 · cited 6×
“…Summarise and/or tabulate RoB assessments of SRs and primary studies Becker 2008 [4]; Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59]; Li 2012 [62]; Ryan 2009 [25]; Robinson 2015 [24, 69–72]; Smith 2011 [77] ▪ For example, summarise the RoB/quality assessment methods…”
InfluentialMailis, A., & Taenzer, P. (2012) · Pain Research and Management · doi:10.1155/2012/794325 · Cited by 56
Baker, P. R. A., Costello, J. T., Dobbins, M., & B. Waters, E. (2014) · Journal of Public Health · doi:10.1093/pubmed/fdu050 · Cited by 53 · cited 4×
“…the process for assessing RoB 3.1 Determine the number of overview authors required to assess studiesa 3.1.1 Independent assessment by 2 or more authors Baker 2014 [43]; Becker 2008 [4]; Cooper 2012 [6]; JBI 2014 [39]; Li 2012 [62]; Ryan 2009 [25] 3.1.2 One author assessment Inferred 3.1.3 One…”
InfluentialLunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E. (2016) · Systematic Reviews · doi:10.1186/s13643-015-0178-0 · Cited by 53
Self-citationDifferences in Overlapping Meta-Analyses of Vitamin D Supplements and Falls
Bolland, M. J., Grey, A., & Reid, I. R. (2014) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2014-2562 · Cited by 49 · cited 7×
“…SRs§ 1.1 Determine how to assess RoB in the included SRs 1.1.1 Select an existing RoB assessment tool for SRs Baker 2014 [43]; Becker 2008 [4]; Bolland 2014 [5]; Büchter 2011 [45, 65]; Caird 2015 [1]; Chen 2014 [46]; CMIMG 2012 [47]; Cooper 2012 [6]; Flodgren 2011 [49]; Foisy 2011 [50];…”
Influential10. Meta‐analysis: Part 2: assessing the quality of published meta‐analyses
Wilson, A. J., & Henry, D. A. (1992) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1992.tb139704.x · Cited by 49 · cited 1×
“Name of the included tools or measures (unnamed tools are identified by first author name and year of publication) AMSTAR [22, 23], R-AMSTAR [36] Named tools: AMSTAR [22, 23], CASP [83], FOCUS [84], MAC [85], NHMRC [86], OQAQ [28], SIGN [87], RAPiD [88] Unnamed tools: Assendelft 1995 [89], Auperin 1997 [90], Crombie 1…”
Popovich, I., Windsor, B., Jordan, V., Showell, M. G., Shea, B., & Farquhar, C. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0050403 · Cited by 46
Ryan, R. E., Kaufman, C. A., & Hill, S. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-16 · Cited by 44 · cited 28×
“…Summarise assessments across items or domains by using a scoring system§§ JBI 2014 [39, 59]; Pieper 2014a [17]; Robinson 2015 [24, 69–72]; Whiting 2013 [12]; Ryan 2009 [25]; Silva 2014 [75] ▪ Sum items, assigning equal or unequal weight to each (JBI 2014 [39, 59]) ▪ Calculate the mean score…”
InfluentialThe Quality of Meta-Analyses of Genetic Association Studies: A Review With Recommendations
Minelli, C., Thompson, J. R., Abrams, K. R., Thakkinstian, A., & Attia, J. R. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp350 · Cited by 44
Overview of systematic reviews - a new type of study. Part II
Silva, V., Grande, A. J., Carvalho, A. P. V. D., Martimbianco, A. L. C., & Riera, R. (2014) · Sao Paulo Medical Journal · doi:10.1590/1516-3180.2013.8150015 · Cited by 41
Methodological approaches in conducting overviews: current state in HTA agencies
Pieper, D., Antoine, S., Morfeld, J., Mathes, T., & Eikermann, M. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1107 · Cited by 39 · cited 39×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the included SR(s), noting mis…”
InfluentialAuperin, A., Pignon, J., & Poynard, T. (1997) · Alimentary Pharmacology & Therapeutics · doi:10.1046/j.1365-2036.1997.131302000.x · Cited by 38 · cited 1×
“Name of the included tools or measures (unnamed tools are identified by first author name and year of publication) AMSTAR [22, 23], R-AMSTAR [36] Named tools: AMSTAR [22, 23], CASP [83], FOCUS [84], MAC [85], NHMRC [86], OQAQ [28], SIGN [87], RAPiD [88] Unnamed tools: Assendelft 1995 [89], Auperin 1997 [90], Crombie 1…”
A flood tide of systematic reviews on endodontic posts: methodological assessment using of R-AMSTAR
Schmitter, M., Sterzenbach, G., Faggion, C. M., & Krastl, G. (2013) · Clinical Oral Investigations · doi:10.1007/s00784-013-0945-z · Cited by 35
(2008) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2008.04.009 · Cited by 29
Chen, Y., Hemming, K., Chilton, P. J., Gupta, K. K., Altman, D. G., & Lilford, R. J. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.08.007 · Cited by 28 · cited 21×
“…SRs 1.1.1 Select an existing RoB assessment tool for SRs Baker 2014 [43]; Becker 2008 [4]; Bolland 2014 [5]; Büchter 2011 [45, 65]; Caird 2015 [1]; Chen 2014 [46]; CMIMG 2012 [47]; Cooper 2012 [6]; Flodgren 2011 [49]; Foisy 2011 [50]; Foisy 2014 [52]; Hartling 2012 [53]; Hartling 2013 [54];…”
InfluentialA Statistical Method for Synthesizing Meta-Analyses
Tang, L. L., Caudy, M., & Taxman, F. (2013) · Computational and Mathematical Methods in Medicine · doi:10.1155/2013/732989 · Cited by 27 · cited 8×
“4 Determine the method to convert effect metrics (either from primary studies or meta-analyses) to the same scale Cooper 2012 [6]; Tang 2013 [78]; Thomson 2010 [26]”
InfluentialMultiple systematic reviews: methods for assessing discordances of results
Moja, L., Fernandez del Rio, M. P., Banzi, R., Cusi, C., D’Amico, R., Liberati, A., Lodi, G., Lucenteforte, E., Minozzi, S., Pecoraro, V., et al. (2012) · Internal and Emergency Medicine · doi:10.1007/s11739-012-0846-1 · Cited by 27 · cited 6×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the included SR(s), noting mis…”
InfluentialCritical reading of the meta-analysis of clinical trials.
Nony, P., Cucherat, M., Haugh, M., & Boissel, J. (1995) · PubMed · Cited by 27
Metaanalysis for the obstetrician-gynecologist
Thacker, S. B., Peterson, H. B., & Stroup, D. F. (1996) · American Journal of Obstetrics and Gynecology · doi:10.1016/s0002-9378(96)70580-x · Cited by 25
A review of the methodological features of systematic reviews in maternal medicine
Sheikh, L., Johnston, S., Thangaratinam, S., Kilby, M. D., & Khan, K. S. (2007) · BMC Medicine · doi:10.1186/1741-7015-5-10 · Cited by 22
Thomson, D., Foisy, M., Oleszczuk, M., Wingert, A., Chisholm, A., & Hartling, L. (2013) · Evidence-Based Child Health: A Cochrane Review Journal · doi:10.1002/ebch.1897 · Cited by 22
Pooling systematic reviews of systematic reviews: a Bayesian panoramic meta‐analysis
Hemming, K., Bowater, R. J., & Lilford, R. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4372 · Cited by 22 · cited 4×
“…from primary studies or meta-analyses) to the same scalea Cooper 2012 [6]; Tang 2013 [78]; Thomson 2010 [26] 2.5 Determine the meta-analysis model and estimation methodsa Cooper 2012 [6]; Hemming 2012 [56]; Schmidt 2013 [74] ▪ For example, second order meta-analysis: fixed or random effects…”
InfluentialImpact of choice of quality appraisal tool for systematic reviews in overviews
Pieper, D., Mathes, T., & Eikermann, M. (2014) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12097 · Cited by 21 · cited 39×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the included SR(s), noting mis…”
InfluentialSantaguida, P. L., Oremus, M., Walker, K. L., Wishart, L. R., Siegel, K. L., & Raina, P. S. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.012 · Cited by 20
An analysis of review articles published in four anaesthesia journals
Smith, A. F. (1997) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03014462 · Cited by 19
Use of Meta-analysis as a Teaching Strategy in Nursing Research Courses
Beck, C. T. (1997) · Journal of Nursing Education · doi:10.3928/0148-4834-19970201-11 · Cited by 18 · cited 2×
“Low risk of bias AMSTAR A Measurement Tool to Assess Systematic Reviews; CASP Critical Appraisal Skills Programme; IRR Inter-rater reliability; OQAQ Overview Quality Assessment Questionnaire; MAC Meta-analysis Appraisal Checklist; NHMRC National Health and Medical Research Council; RAPiD Rapid Appraisal Protocol inter…”
Crick, K., Wingert, A., Williams, K. J., Fernandes, R. M., Thomson, D., & Hartling, L. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0084-0 · Cited by 17 · cited 15×
“Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26] ▪ Report RoB assessments of primary studies from the included SR(s), noting mis…”
InfluentialRating the quality of evidence is by necessity a matter of judgment
Murad, M. H., Mustafa, R. A., Morgan, R. L., Sultan, S., Falck-Ytter, Y. T., & Dahm, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.11.018 · Cited by 17
Meta-analysis of clinical trials: a consumer's guide
Geller, N. L., & Proschan, M. A. (1996) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543409608835151 · Cited by 16
Quality of systematic reviews of observational nontherapeutic studies.
Shamliyan, T., Kane, R. L., & Jansen, S. (2010) · Europe PMC (PubMed Central) · Cited by 15
How to critically appraise and apply meta‐analyses in clinical practice
Ho, R., ONG, H. S., Kundadak, G. K., Cheung, M. W. ‐., & Mak, A. (2010) · International Journal of Rheumatic Diseases · doi:10.1111/j.1756-185x.2010.01560.x · Cited by 12
Zambon, M., Biondi‐Zoccai, G., Elena, B., Ruggeri, L., Zangrillo, A., & Landoni, G. (2012) · Journal of Anesthesia · doi:10.1007/s00540-012-1372-z · Cited by 10
Development of the Metareview Assessment of Reporting Quality (MARQ) Checklist
Singh, J. P. (2012) · Revista de la Facultad de Medicina · Cited by 10
Integration of Existing Systematic Reviews
Robinson, K. A., Whitlock, E. P., O'Neil, M. E., Anderson, J., Hartling, L., Dryden, D. M., Butler, M. E., Newberry, S. J., McPheeters, M. L., Berkman, N. D., et al. (2014) · Cited by 8
A review of the methodological features of systematic reviews in fetal medicine
Knox, E. M., Thangaratinam, S., Kilby, M. D., & Khan, K. S. (2009) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2009.05.006 · Cited by 7
Robinson, K. A., Whitlock, E. P., O’Neil, M. K., Anderson, J. L., Hartling, L. M., Dryden, D., Butler, M. K., Newberry, S., McPheeters, M. L., Berkman, N. D., et al. (2014) · Cited by 3
“Overviews” should meet the methodological standards of systematic reviews
Kovacs, F. M., Urrútia, G., & Alarcón, J. D. (2013) · European Spine Journal · doi:10.1007/s00586-013-2996-x · Cited by 2 · cited 1×
“…Flodgren 2011 [49]; Foisy 2011 [50]; Foisy 2014 [52]; Hartling 2012 [53]; Hartling 2013 [54]; Jadad 1997 [29]; James 2014 [58]; JBI 2014 [39, 59]; Kovacs 2014 [60]; Kramer 2009 [61]; Li 2012 [62]; Pieper 2012 [3]; Pieper 2014c [66]; Pieper 2014d [68]; Pieper 2014a [17]; Robinson 2015 [24,…”
James, B. M., Baker, P., Costello, J. T., & Francis, D. P. (2014) · QUT ePrints (Queensland University of Technology) · Cited by 2
Smith KE, Hill S and Bambra C (eds.). Health inequalities, critical perspectives.
Koupil, I. (2016) · European Journal of Public Health · doi:10.1093/eurpub/ckw012 · Cited by 1
Research works printed but not linked: 17
- Whiting P Davies P Savović J Caldwell D Churchill R Chapter 4. Phase 2: review of existing quality assessment tools for systematic reviews. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews 2013 22 35
- Bai A Shukla VK Bak G Wells G Chapter 4: tools selected through QAT project. In: quality assessment tools project report 2012 Ottawa Canadian Agency for Drugs and Technologies in Health
- Whiting P Davies P Savović J Caldwell D Churchill R Chapter 5. Phase 3: review of studies that have used the AMSTAR tool. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews 2013
- Parmelli E, Banzi R, Fernandez Del Rio MDP, Minozzi S, Moja L, Pecoraro V, Liberati A: Using AMSTAR to assess the methodological quality of systematic reviews: an external validation study. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain [abstract]. In Cochrane Database Syst Rev, Supplement, vol. Suppl. pp. 139; 2011:139.
- Cochrane Methods Group. About the Cochrane Methodology Register: Cochrane; 2012. http://www.cochranelibrary.com/help/the-cochrane-methodology-register-july-issue-2012.html
- CMIMG C Review Type & Methodological Considerations --Background Paper for the First Part of the Paris CMIMG Discussion 2012
- Foisy MFR Dryden DM Hartling L Grading the quality of evidence in existing systematic reviews: challenges and considerations 22nd Cochrane Colloquium 2014 Hyderabad Wiley
- Büchter R Pieper D Jerinic P Overviews of systematic reviews often do not assess methodological quality of included reviews. Poster 19th Cochrane Colloquium 2011 Madrid Cochrane Database Syst Rev 105 106
- Robinson KA Chou R Berkman ND Newberry SJ Fu R Hartling L Dryden D Butler M Foisy M Anderson J Motu’apuaka ML Relevo R Guise JM Chang S Integrating bodies of evidence: existing systematic reviews and primary studies Methods Guide for Effectiveness and Comparative Effectiveness Reviews 2008 Rockville Agency for Healthcare Research and Quality (US) 25834891
- White CM Ip S McPheeters MC Tim S Chou R Lohr KN Robinson K McDonald K Whitlock EP Using existing systematic reviews to replace de novo processes in conducting comparative effectiveness reviews. In Methods Guide for Comparative Effectiveness Reviews 2009 Rockville Agency for Healthcare Research and. Quality 21433402
- Salanti G Becker L Caldwell D Churchill R Higgins J Li T Schmid C Evolution of Cochrane Intervention Reviews and Overviews of Reviews to better accommodate comparisons among multiple interventions Report from a meeting of the Cochrane Comparing Multiple Interventions Methods Groups: Cochrane Comparing Multiple Interventions Methods Groups 2011
- Wagner S, White M, Schultz I, Iverson R, Hsu V, McGuire L, Schultz W. Assessing a systematic review of systematic reviews: developing a criteria. In: Innovation in worker health and safety: Annual Conference, Canadian Association for Research on Work and Health, June 1-2, 2012. Vancouver; 2012. https://www.wwdpi.org/SiteCollectionDocuments/CIRPD-Research/CARWH2012/P3_MethodologicalCriteria.pdf.
- McMaster University Health systems evidence 2011
- Unit PHR Critical Appraisal Skills Programme (CASP). 10 questions to help you make sense of reviews 2006 Oxford Public Health Resource Unit
- FOCUS FOCUS critical appraisal tool 2001 London The Royal College of Psychiatrists
- Joanna Briggs Institute RAPid: Rapid Appraisal protocol internet database 2006 Adelaide The Joanna Briggs Institute
- Task Force on Systematic Review and Guidelines Assessing the quality and applicability of systematic reviews (AQASR) 2011 Austin National Center for the Dissemination of Disability Research
Cited by
Gates, M., Gates, A., Pieper, D., Fernandes, R. M., Tricco, A. C., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070849 · Cited by 856
Self-citationHuman health impacts of exposure to phthalate plasticizers: An overview of reviews
Eales, J., Bethel, A., Galloway, T., Hopkinson, P., Morrissey, K., Short, R. E., & Garside, R. (2022) · Environment International · doi:10.1016/j.envint.2021.106903 · Cited by 367
Prehabilitation in adult patients undergoing surgery: an umbrella review of systematic reviews
McIsaac, D. I., Gill, M., Boland, L., Hutton, B., Branje, K., Shaw, J., Grudzinski, A. L., Barone, N., Gillis, C., Akhtar, S., et al. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.11.014 · Cited by 312
Graphical Representation of Overlap for OVErviews: GROOVE tool
Bracchiglione, J., Meza, N., Bangdiwala, S. I., Niño de Guzmán, E., Urrútia, G., Bonfill, X., & Madrid, E. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1557 · Cited by 299
“10,17,18 Matrices of evidence are grids that link SRs with their included primary studies.”
BackgroundPollock, M., Fernandes, R. M., Pieper, D., Tricco, A. C., Gates, M., Gates, A., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1252-9 · Cited by 228
Metadata
- Authors
- Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E.
- Affiliations
- Monash University (AU)
- Corresponding author
- McKenzie, Joanne E.
- Venue
- Systematic Reviews
- Published
- 2018-10-12
- Volume
- 7
- Issue
- 1
- Pages
- 159-159
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6186052
- OpenAlex
- W2898047957
- Cited by
- 139
- References
- 96
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 13 · 2023: 29 · 2022: 27 · 2021: 22 · 2020: 15 · 2019: 10
- FWCI
- 2.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (acceptedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Australian Postgraduate Award, International Postgraduate Research Scholarship
- National Health and Medical Research Council (1143429)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- overviews of systematic reviews, risk of bias assessment, quality of evidence, evidence map, methodological evaluation, reporting guidelines
- MeSH terms
- Systematic Reviews as Topic, Humans, Research Design, Review Literature as Topic, Bias, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


