Ten simple rules for conducting umbrella reviews
Open accessAbstract
Objective: Evidence syntheses such as systematic reviews and meta-analyses provide a rigorous and transparent knowledge base for translating clinical research into decisions, and thus they represent the basic unit of knowledge in medicine. Umbrella reviews are reviews of previously published systematic reviews or meta-analyses. Therefore, they represent one of the highest levels of evidence synthesis currently available, and are becoming increasingly influential in biomedical literature. However, practical guidance on how to conduct umbrella reviews is relatively limited. Methods: We present a critical educational review of published umbrella reviews, focusing on the essential practical steps required to produce robust umbrella reviews in the medical field. Results: The current manuscript discusses 10 key points to consider for conducting robust umbrella reviews. The points are: ensure that the umbrella review is really needed, prespecify the protocol, clearly define the variables of interest, estimate a common effect size, report the heterogeneity and potential biases, perform a stratification of the evidence, conduct sensitivity analyses, report transparent results, use appropriate software and acknowledge the limitations. We illustrate these points through recent examples from umbrella reviews and suggest specific practical recommendations. Conclusions: The current manuscript provides a practical guidance for conducting umbrella reviews in medical areas. Researchers, clinicians and policy makers might use the key points illustrated here to inform the planning, conduction and reporting of umbrella reviews in medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The review followed recommended practice including a published protocol with predefined inclusion criteria, standardised data extraction, description of overlap, estimations of common effect sizes where applicable, and assessments of study quality and heterogeneity.
“We also followed recommended practice which include the following features: a published protocol with predefined inclusion criteria; a standardised data extraction technique; a description of any overlap in meta-analyses of the same outcomes; estimations of common effect sizes where applicable; and assessments of study quality and heterogeneity (Aromataris et al., 2015; Fusar-Poli and Radua, 2018;”
From The benefits and harms of community treatment orders for people diagnosed with psychiatric illnesses: A rapid umbrella review of systematic reviews and meta-analyses (2024) · cites this paper as Fusar-Poli and Radua, 2018
The sections on prespecifying the protocol, estimating a common effect size, and reporting heterogeneity support most of the cited recommendations; the protocol section also calls for defined data-extraction methods and predefined inclusion criteria. The section on defining variables of interest discusses overlapping reviews and recommends selecting one, but does not specifically recommend describing the overlap, so that feature is not fully supported by this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe level of quantitative evidence was assessed using criteria adapted from Fusar-Poli and Radua.
“Finally, we assessed the level of quantitative evidence using criteria adapted from Fusar-Poli and Radua (2018):”
From The benefits and harms of community treatment orders for people diagnosed with psychiatric illnesses: A rapid umbrella review of systematic reviews and meta-analyses (2024) · cites this paper as Fusar-Poli and Radua (2018)
The review was guided by the World Health Organization practical guide for Rapid Reviews to Strengthen Health Policy and Systems and adhered to PRISMA guidelines and umbrella review guidance.
“A rapid umbrella review was conducted guided by the World Health Organization practical guide for Rapid Reviews to Strengthen Health Policy and Systems [14] and adhering to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [15] and umbrella review guidance [16].”
From Implementation of Telemental Health Services Before COVID-19: Rapid Umbrella Review of Systematic Reviews (2021) · cites this paper as [16]
The strength of evidence was graded using umbrella review methodology.
“graded the strength of the evidence as using umbrella review methodology.”
From Mobile phone-based interventions for mental health: A systematic meta-review of 14 meta-analyses of randomized controlled trials (2022) · cites this paper as [22]
We failed to find convincing evidence in support of any mobile phone-based intervention on any outcome.
“Applying standards drawn from umbrella review methodology [22, 24], we failed to find convincing evidence in support of any mobile phone-based intervention on any outcome.”
From Mobile phone-based interventions for mental health: A systematic meta-review of 14 meta-analyses of randomized controlled trials (2022) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623 · see citing works
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.005 · Cited by 19,575 · see citing works
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1× · see citing works
“…of the calculations of an umbrella review with usual meta-analytical software, such as ‘meta’, ‘metafor’ or ‘metansue’ packages for R. 27–29 Some software includes better estimation methods of the between-study heterogeneity than others, 30–32 but this may probably represent a minor difference.”
MethodMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · see citing works
External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569 · see citing works
What causes psychosis? An umbrella review of risk and protective factors
Radua, J., Ramella‐Cravaro, V., Ioannidis, J. P. A., Reichenberg, A., Phiphopthatsanee, N., Amir, T., Yenn Thoo, H., Oliver, D., Davies, C., Morgan, C., et al. (2018) · World Psychiatry · doi:10.1002/wps.20490 · Cited by 553 · see citing works
Self-citationFusar-Poli, P., Cappucciati, M., Borgwardt, S., Woods, S. W., Addington, J., Nelson, B., Nieman, D. H., Stahl, D. R., Rutigliano, G., Riecher-Rössler, A., et al. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.2324 · Cited by 440 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 26
- Ioannidis JP, Khoury MJ, Km J. Improving validation practices in "omics" research. Science 2011;334:1230-2. · DOI
- Szucs d, Ioannidis JP, Ijp A. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol 2017;15:e2000797. · DOI · Paper page
- Ioannidis JP. Evolution and translation of research findings: from bench to where? PLoS Clin Trials 2006;1:e36. · DOI · Paper page
- Ioannidis JP, Munafò MR, Fusar-Poli P, et al. Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends Cogn Sci 2014;18:235-41. · DOI · Paper page
- Ioannidis JP. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ 2009;181:488-93. · DOI · Paper page
- elliott J, Kelly SE, Bai Z, et al. Optimal duration of dual antiplatelet therapy following percutaneous coronary intervention: protocol for an umbrella review. BMJ Open 2017;7:e015421. · DOI
- catalá-lópez F, Hutton B, Driver JA, et al. Cancer and central nervous system disorders: protocol for an umbrella review of systematic reviews and updated meta- analyses of observational studies. Syst Rev 2017;6:69. · DOI
- dinsdale S, Azevedo LB, Shucksmith J, et al. Effectiveness of weight management, smoking cessation and alcohol reduction interventions in changing behaviors during pregnancy: an umbrella review protocol. JBI Database System Rev Implement Rep 2016;14:29-47. · DOI
- Jadczak Ad, Makwana N, Luscombe-Marsh ND, et al. Effectiveness of exercise interventions on physical function in community-dwelling frail older people: an umbrella review protocol. JBI Database System Rev Implement Rep 2016;14:93-102. · DOI
- chai lK, Burrows T, May C, et al. Effectiveness of family-based weight management interventions in childhood obesity: an umbrella review protocol. JBI Database System Rev Implement Rep 2016;14:32-9. · DOI
- thomson K, Bambra C, McNamara C, et al. The effects of public health policies on population health and health inequalities in European welfare states: protocol for an umbrella review. Syst Rev 2016;5:57. · DOI
- Krause d, Roupas P. Dietary interventions as a neuroprotective therapy for the delay of the onset of cognitive decline in older adults: an umbrella review protocol. JBI Database System Rev Implement Rep 2015;13:74-83. · DOI
- Schmidt A, Cappucciati M, Radua J, et al. Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta- analytical sequential testing simulation. Schizophr Bull 2017;43:375-88. · DOI
- Fusar-Poli P, Cappucciati M, Bonoldi I, et al. Prognosis of brief psychotic episodes: a meta-analysis. JAMA Psychiatry 2016;73:211-20. · DOI · Paper page
- Siontis Kc, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: survey of published studies. BMJ 2013;347:f4501. · DOI
- Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q 2016;94:485-514. · DOI · Paper page
- Stroup dF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology -a proposal for reporting. JAMA 2000;283:2008-12.
- Shea bJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. · DOI · Paper page
- chinn S. A simple method for converting an odds ratio to effect size for use in meta- analysis. Stat Med 2000;19:3127-31. · DOI · Paper page
- bellou V, Belbasis L, Tzoulaki I, et al. Systematic evaluation of the associations between environmental risk factors and dementia: An umbrella review of systematic reviews and meta-analyses. Alzheimers Dement 2017;13. · DOI
- bellou V, Belbasis L, Tzoulaki I, et al. Environmental risk factors and Parkinson's disease: an umbrella review of meta-analyses. Parkinsonism Relat Disord 2016;23:1-9. · DOI
- belbasis l, Bellou V, Evangelou E, et al. Environmental risk factors and multiple sclerosis: an umbrella review of systematic reviews and meta-analyses. Lancet Neurol 2015;14:263-73. · DOI
- Schwarzer G. meta: an R package for meta-analysis. R News 2007;7:40-5.
- Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw 2010;36:1-48. · DOI · Paper page
- radua J, Schmidt A, Borgwardt S, et al. Ventral striatal activation during reward processing in psychosis: a neurofunctional meta-analysis. JAMA Psychiatry 2015;72:1243-51. · DOI
- Petropoulou M, Mavridis D. A comparison of 20 heterogeneity variance estimators in statistical synthesis of results from studies: a simulation study. Stat Med 2017;36:4266-80. copyright. on August 16, 2023 by guest. Protected by http://mentalhealth.bmj.com/ Evid Based Mental Health: first published as 10.1136/ebmental-2018-300014 on 13 July 2018. Downloaded from · DOI
Cited by
Sorted by citations
Gates, M., Gates, A., Pieper, D., Fernandes, R. M., Tricco, A. C., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070849 · Cited by 856 · see citing works
Self-citationPrevention of Psychosis: Advances in Detection, Prognosis, and Intervention
Fusar-Poli, P., Salazar de Pablo, G., Correll, C. U., Meyer-Lindenberg, A., Millan, M. J., Borgwardt, S., Galderisi, S., Bechdolf, A., Pfennig, A., Kessing, L. V., et al. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.4779 · Cited by 476 · see citing works
Self-citationDavies, C., Segre, G., Estradé, A., Radua, J., De Micheli, A., Provenzani, U., Oliver, D., Salazar de Pablo, G., Ramella-Cravaro, V., Besozzi, M., et al. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30057-2 · Cited by 301 · see citing works
Self-citationKim, J. Y., Son, M. J., Son, C. Y., Radua, J., Eisenhut, M., Gressier, F., Koyanagi, A., Carvalho, A. F., Stubbs, B., Solmi, M., et al. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30181-6 · Cited by 271 · see citing works
Self-citationvan IJzendoorn, M. H., Bakermans‐Kranenburg, M. J., Coughlan, B., & Reijman, S. (2020) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13147 · Cited by 262 · see citing works
Goldberg, S. B., Lam, S. U., Simonsson, O., Torous, J., & Sun, S. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000002 · Cited by 237 · see citing works
“We evaluated the certainty of the evidence using criteria drawn from umbrella review methodology [22, 24].”
BackgroundMethodInfluentialHowes, O. D., Whitehurst, T., Shatalina, E., Townsend, L., Onwordi, E. C., Mak, T. L. A., Arumuham, A., O’Brien, O., Lobo, M., Vano, L., et al. (2021) · World Psychiatry · doi:10.1002/wps.20822 · Cited by 243 · see citing works
Task shifting in primary care to tackle healthcare worker shortages: An umbrella review
Leong, S. L., Teoh, S. L., Fun, W. H., & Lee, S. W. H. (2021) · European Journal of General Practice · doi:10.1080/13814788.2021.1954616 · Cited by 228 · see citing works
Evidence-based umbrella review of 162 peripheral biomarkers for major mental disorders
Carvalho, A. F., Solmi, M., Sanches, M., Machado, M. O., Stubbs, B., Ajnakina, O., Sherman, C., Sun, Y. R., Liu, C. S., Brunoni, A. R., et al. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0835-5 · Cited by 204 · see citing works
Self-citationThe Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197 · see citing works
Self-citationProviding Human Support for the Use of Digital Mental Health Interventions: Systematic Meta-review
Werntz, A., Amado, S., Jasman, M., Ervin, A., & Rhodes, J. E. (2023) · Journal of Medical Internet Research · doi:10.2196/42864 · Cited by 158 · see citing works
Association between anxiety and hypertension in adults: A systematic review and meta-analysis
Lim, L.-F., Solmi, M., & Cortese, S. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.08.031 · Cited by 134 · see citing works
Barnett, P., Goulding, L., Casetta, C., Jordan, H., Sheridan-Rains, L., Steare, T., Williams, J., Wood, L., Gaughran, F., & Johnson, S. (2021) · Journal of Medical Internet Research · doi:10.2196/26492 · Cited by 136 · see citing works
Dragioti, E., Radua, J., Solmi, M., Gosling, C. J., Oliver, D., Lascialfari, F., Ahmed, M., Cortese, S., Estradé, A., Arrondo, G., et al. (2023) · World Psychiatry · doi:10.1002/wps.21068 · Cited by 130 · see citing works
Self-citationPreventive Treatments for Psychosis: Umbrella Review (Just the Evidence)
Fusar-Poli, P., Davies, C., Solmi, M., Brondino, N., De Micheli, A., Kotlicka-Antczak, M., Shin, J. I., & Radua, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00764 · Cited by 108 · see citing works
Self-citationCircular Economy: A Review of Review Articles
Hossain, M., Park, S., Suchek, N., & Pansera, M. (2024) · Business Strategy and the Environment · doi:10.1002/bse.3867 · Cited by 77 · see citing works
Autoimmune diseases and adverse pregnancy outcomes: an umbrella review
Singh, M., Wambua, S., Lee, S. I., Okoth, K., Wang, Z., Fayaz, F. F. A., Eastwood, K.-A., Nelson-Piercy, C., Reynolds, J. A., Nirantharakumar, K., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03309-y · Cited by 71 · see citing works
Impact of organized activities on mental health in children and adolescents: An umbrella review
Boelens, M., Smit, M. S., Raat, H., Bramer, W. M., & Jansen, W. (2022) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2021.101687 · Cited by 65 · see citing works
Bougioukas, K. I., Vounzoulaki, E., Mantsiou, C. D., Papanastasiou, G. D., Savvides, E. D., Ntzani, E. E., & Haidich, A.-B. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.019 · Cited by 46 · see citing works
Diagnostic biomarkers for obsessive-compulsive disorder: A reasonable quest or ignis fatuus?
Fullana, M. A., Abramovitch, A., Via, E., López-Sola, C., Goldberg, X., Reina, N., Fortea, L., Solanes, A., Buckley, M. J., Ramella-Cravaro, V., et al. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.08.008 · Cited by 44 · see citing works
Self-citationImpact of Exercise Training Interventions on Flow-Mediated Dilation in Adults: An Umbrella Review
Shivgulam, M. E., Liu, H., Schwartz, B. D., Langley, J. E., Bray, N. W., Kimmerly, D. S., & O’Brien, M. W. (2023) · Sports Medicine · doi:10.1007/s40279-023-01837-w · Cited by 42 · see citing works
Kong, F., Zhao, Y., Zhang, W., Wang, X., Wu, T., Zhou, Z., Xu, Y., Xia, L., & Sun, T. (2026) · Nature Communications · doi:10.1038/s41467-025-67701-9 · Cited by 29 · see citing works
Kisely, S., Zirnsak, T., Corderoy, A., Ryan, C. J., & Brophy, L. (2024) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/00048674241246436 · Cited by 25 · see citing works
“Finally, we assessed the level of quantitative evidence using criteria adapted from Fusar-Poli and Radua (2018):”
MethodByrne, S., Cobham, V., Richardson, M., & Imuta, K. (2023) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-023-00436-5 · Cited by 23 · see citing works
Solanes, A., Albajes-Eizagirre, A., Fullana, M. A., Fortea, L., Fusar-Poli, P., Torrent, C., Solé, B., Bonnín, C. M., Shin, J. I., Vieta, E., et al. (2021) · Revista de Psiquiatría y Salud Mental · doi:10.1016/j.rpsm.2020.08.002 · Cited by 19 · see citing works
Self-citationGates, M., Gates, A., Pieper, D., Fernandes, R., Tricco, A., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · doi:10.31222/osf.io/23b6j · Cited by 3 · see citing works
Self-citationCommon issues of systematic reviews in the sports and exercise medicine field.
Memon, A. R., Owen, P. J., Anderson, N., Verhagen, E., Mundell, N. L., & Belavy, D. L. (2024) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2023-001784 · Cited by 3 · see citing works
- Authors
- Fusar-Poli, P., & Radua, J.
- Affiliations
- Early Psychosis: Interventions and Clinical-detection (EPIC) Lab, Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK
- Venue
- Evidence Based Mental Health
- Publisher
- BMJ
- Published
- 2018-07-13
- Volume
- 21
- Issue
- 3
- Pages
- 95-100
- ISSN
- 1362-0347 (ISSN-L)
- 1468-960X
- PMC
- PMC10270421
- OpenAlex
- W2884633725
- Cited by
- 724
- References
- 8
- Citations by year
- 2026: 129 · 2025: 117 · 2024: 122 · 2023: 110 · 2022: 108 · 2021: 66 · 2020: 50 · 2019: 20 · 2018: 2
- FWCI
- 26.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Evidence-Based Mental Health (publishedVersion, other-oa)
- PubMed Central (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Portal (King's College London) (submittedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- Funders
- King's College London
- Medical Research Council (MC_PC_16048)
- Instituto de Salud Carlos III
- European Regional Development Fund
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- umbrella review, systematic reviews, meta-analysis, evidence synthesis, heterogeneity, bias assessment, sensitivity analysis, reporting transparency, research methodology guidelines
- MeSH terms
- Systematic Reviews as Topic, Humans, Statistics as Topic, Meta-Analysis as Topic, Guidelines as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
