Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews (SRs) can help decision makers interpret the deluge of published biomedical literature. However, a SR may be of limited use if the methods used to conduct the SR are flawed, and reporting of the SR is incomplete. To our knowledge, since 2004 there has been no cross-sectional study of the prevalence, focus, and completeness of reporting of SRs across different specialties. Therefore, the aim of our study was to investigate the epidemiological and reporting characteristics of a more recent cross-section of SRs. METHODS AND FINDINGS: We searched MEDLINE to identify potentially eligible SRs indexed during the month of February 2014. Citations were screened using prespecified eligibility criteria. Epidemiological and reporting characteristics of a random sample of 300 SRs were extracted by one reviewer, with a 10% sample extracted in duplicate. We compared characteristics of Cochrane versus non-Cochrane reviews, and the 2014 sample of SRs versus a 2004 sample of SRs. We identified 682 SRs, suggesting that more than 8,000 SRs are being indexed in MEDLINE annually, corresponding to a 3-fold increase over the last decade. The majority of SRs addressed a therapeutic question and were conducted by authors based in China, the UK, or the US; they included a median of 15 studies involving 2,072 participants. Meta-analysis was performed in 63% of SRs, mostly using standard pairwise methods. Study risk of bias/quality assessment was performed in 70% of SRs but was rarely incorporated into the analysis (16%). Few SRs (7%) searched sources of unpublished data, and the risk of publication bias was considered in less than half of SRs. Reporting quality was highly variable; at least a third of SRs did not report use of a SR protocol, eligibility criteria relating to publication status, years of coverage of the search, a full Boolean search logic for at least one database, methods for data extraction, methods for study risk of bias assessment, a primary outcome, an abstract conclusion that incorporated study limitations, or the funding source of the SR. Cochrane SRs, which accounted for 15% of the sample, had more complete reporting than all other types of SRs. Reporting has generally improved since 2004, but remains suboptimal for many characteristics. CONCLUSIONS: An increasing number of SRs are being published, and many are poorly conducted and reported. Strategies are needed to help reduce this avoidable waste in research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many systematic review reports include narrative summaries of the characteristics and risk of bias across all included studies.36”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even for those studies which explicitly reference PRISMA, there is only slight, statistically non-significant evidence of improved reporting, as found by Page et al. [15].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The studies included in a systematic review will generally vary with respect to design, study populations, and risk of bias [1].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“in a survey of 300 systematic reviews, 64% of the reviews included observational studies [7].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although few published evaluations exist, evidence suggests that the endorsement of PRISMA by journals is associated with more complete reporting of systematic reviews than in journals not endorsing the reporting guideline.50 51 52”
From Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. (2022) · cites this paper as [51]
Experimental — accuracy not yet measured
23 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Self-citationPreferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Self-citationPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Self-citationThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationPRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citation
Research works printed but not linked: 2
- Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. AHRQ Publication No. 10(14)-EHC063-EF. 2014 Jan [cited 21 Apr 2016]. Available: https://www.effectivehealthcare.ahrq.gov/ehc/products/60/318/CER-Methods-Guide-140109.pdf. 21433403
- Nordic Cochrane Centre. Review Manager (RevMan), version 5.1. Copenhagen: Nordic Cochrane Centre; 2011.
Cited by
Items 101–200 of 793
Eke, A. C., Eleje, G. U., Eke, U. A., Xia, Y. F., & Liu, J. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008545.pub2 · Cited by 89
Gates, M., Gates, A., Duarte, G. S., Cary, M., Becker, M., Prediger, B., Vandermeer, B., Fernandes, R. M., Pieper, D., & Hartling, L. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.04.026 · Cited by 79
Sheyholislami, H., & Connor, K. L. (2021) · Nutrients · doi:10.3390/nu13072382 · Cited by 75
Nguyen, P.-Y., Kanukula, R., McKenzie, J. E., Alqaidoom, Z., Brennan, S. E., Haddaway, N. R., Hamilton, D. G., Karunananthan, S., McDonald, S., Moher, D., et al. (2022) · BMJ · doi:10.1136/bmj-2022-072428 · Cited by 68
Self-citationTime-to-update of systematic reviews relative to the availability of new evidence
Bashir, R., Surian, D., & Dunn, A. G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0856-9 · Cited by 72
Tierney, J. F., Fisher, D. J., Burdett, S., Stewart, L. A., & Parmar, M. K. B. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003019 · Cited by 76
“From recent data [1], we estimate that approximately 1,200 oncology intervention reviews are published each year, which may be of variable quality, so we expect our findings to be of widespread use.”
BackgroundTsujimoto, Y., Tsujimoto, H., Kataoka, Y., Kimachi, M., Shimizu, S., Ikenoue, T., Fukuma, S., Yamamoto, Y., & Fukuhara, S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.008 · Cited by 71
Meta-analyses indexed in PsycINFO had a better completeness of reporting when they mention PRISMA
Leclercq, V., Beaudart, C., Ajamieh, S., Rabenda, V., Tirelli, E., & Bruyère, O. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.014 · Cited by 69
Meta-analyses frequently pooled different study types together: a meta-epidemiological study
Bun, R. S., Scheer, J., Guillo, S., Tubach, F., & Dechartres, A. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.10.013 · Cited by 70
“,[16] 21% and 7% of therapeutic non-Cochrane reviews published in 2016 considered cohort and case–control studies, respectively, in addition to RCTs, a rate that has”
Backgroundde Vries, B. B. L. P., van Smeden, M., Rosendaal, F. R., & Groenwold, R. H. H. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.009 · Cited by 72
Rethlefsen, M. L., Brigham, T., Price, C. L., Moher, D., Bouter, L. M., Kirkham, J. J., Schroter, S., & Zeegers, M. P. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111229 · Cited by 77
Self-citationBoutron, I., Créquit, P., Williams, H. C., Meerpohl, J. J., Craig, J. C., & Ravaud, P. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.024 · Cited by 70
“The methods used to conduct systematic reviews are frequently flawed [15].”
BackgroundMethodInfluentialThe effect of librarian involvement on the quality of systematic reviews in dental medicine
Schellinger, J., Sewell, K. B., Bloss, J., Ebron, T., & Forbes, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256833 · Cited by 73
“The methodological quality of SRs can be highly variable across health professions [2, 5, 6], despite the widespread use of international guidelines on SRs such as those from the Cochrane Collaboration, the National Academy of Medicine, AHRQ, MOOSE, and others.”
BackgroundThe score after 10 years of registration of systematic review protocols
van der Braak, K., Ghannad, M., Orelio, C. C., Heus, P., Damen, J. A., Spijker, R., Robinson, K. A., Lund, H., & Hooft, L. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02053-9 · Cited by 69
Gates, A., Gates, M., Duarte, G. S., Cary, M., Becker, M., Prediger, B., Vandermeer, B., Fernandes, R. M., Pieper, D., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0746-1 · Cited by 70
“Discussion The methodological and reporting quality of SRs can vary [4], and when their conduct is poor, the results can be biased.”
BackgroundMethodInfluentialText mining to support abstract screening for knowledge syntheses: a semi-automated workflow
Pham, B., Jovanovic, J., Bagheri, E., Antony, J., Ashoor, H., Nguyen, T. T., Rios, P., Robson, R., Thomas, S. M., Watt, J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01700-x · Cited by 63
“Information from a study examining characteristics of a representative sample of the population of SRs is used to guide the selection of k relative to the median number of 15 included studies (interquartile range: 8 – 25) [36].”
Self-citationStudying review articles in scientometrics and beyond: a research agenda
Blümel, C. D., & Schniedermann, A. (2020) · Scientometrics · doi:10.1007/s11192-020-03431-7 · Cited by 72
Barčot, O., Borić, M., Peričić, T. P., Ćavar, M., Došenović, S., Vuka, I., & Puljak, L. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0804-y · Cited by 65
Protocol registration improves reporting quality of systematic reviews in dentistry
dos Santos, M. B. F., Agostini, B. A., Bassani, R., Pereira, G. K. R., & Sarkis-Onofre, R. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00939-7 · Cited by 61
Self-citationFiorillo, V., & Amico, B. M. (2024) · Dairy · doi:10.3390/dairy5030031 · Cited by 72
“Numerous observational studies have shown that the use of the PRISMA method is associated with more comprehensive literature reviews [30,32–34].”
MethodDechartres, A., Atal, I., Riveros, C., Meerpohl, J., & Ravaud, P. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1517 · Cited by 59
Büttner, F., Winters, M., Delahunt, E., Elbers, R., Lura, C. B., Khan, K. M., Weir, A., & Ardern, C. L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-101675 · Cited by 55
Can systematic reviews with sparse data be trusted?
Afshari, A., Wetterslev, J., & Smith, A. F. (2016) · Anaesthesia · doi:10.1111/anae.13730 · Cited by 60
“Editors, writers and readers should exercise great caution when interpreting conclusions of reviews that only include very few trials, even when they fulfill the required standards [3].”
BackgroundRefining Boolean queries to identify relevant studies for systematic review updates
Alharbi, A., & Stevenson, M. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa148 · Cited by 68
“With over eight thousand reviews being produced per year [6], keeping them up to date represents a formidable challenge.”
BackgroundHelbach, J., Hoffmann, F., Pieper, D., & Allers, K. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.12.019 · Cited by 70
Page, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S., Haddaway, N., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 63
“Cochrane Database of Systematic Reviews, Campbell Systematic Reviews, JBI Evidence Synthesis) versus published elsewhere; between the 2020 reviews of health interventions and a previously examined sample of 110 systematic reviews of health interventions indexed in MEDLINE in February 2014 [25, 39]; between reviews wit…”
BackgroundMethodInfluentialSelf-citationHealth Effects of Electronic Cigarettes: An Umbrella Review and Methodological Considerations
Travis, N., Knoll, M., Cadham, C. J., Cook, S., Warner, K. E., Fleischer, N. L., Douglas, C. E., Sánchez‐Romero, L. M., Mistry, R., Meza, R., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159054 · Cited by 62
“These findings are largely in line with existing evidence that despite several guidelines for conducting and reporting systematic reviews, less than a third of systematic reviews report having adhered to these guidelines [35].”
ResultGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00139 · Cited by 63
Hoffmann, F., Eggers, D., Pieper, D., Zeeb, H., & Allers, K. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.003 · Cited by 56
A retrospective comparison of systematic reviews with same-topic rapid reviews
Reynen, E., Robson, R., Ivory, J., Hwee, J., Straus, S. E., Pham, B., & Tricco, A. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.001 · Cited by 51
“who highlighted poor reporting of a 299 large cross-section of systematic reviews in 2016 [23].”
BackgroundSelf-citationA survey of prevalence of narrative and systematic reviews in five major medical journals
Faggion, C. M., Bakas, N., & Wasiak, J. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0453-y · Cited by 61
“For example, two definitions [14, 20] report “reproducibility” as key characteristics of a systematic review.”
BackgroundMethodThe influence of the team in conducting a systematic review
Uttley, L., & Montgomery, P. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0548-x · Cited by 54
“Additionally, Page and colleagues [18] found when analysing 682 systematic reviews (both Cochrane and non-Cochrane) that at least a third”
BackgroundSystematic reviews in dentistry: Current status, epidemiological and reporting characteristics
Bassani, R., Pereira, G. K. R., Page, M. J., Tricco, A. C., Moher, D., & Sarkis-Onofre, R. (2019) · Journal of Dentistry · doi:10.1016/j.jdent.2019.01.014 · Cited by 53
Self-citationHutton, B., Wolfe, D., Moher, D., & Shamseer, L. (2017) · Evidence Based Mental Health · doi:10.1136/eb-2017-102666 · Cited by 53
“Evidence is accumulating that shows that the endorsement and use of reporting guidelines in the editorial process, peer review and by authors are associated with more completely reported research.(6) 16– 21 This article provides an overview of the most widely used reporting guidelines for reporting of protocols and co…”
BackgroundInfluentialSelf-citationAyala, A. M. C., Parker, K. E., Mazzoli, E., Lander, N., Ridgers, N. D., Timperio, A. F., Lubans, D. R., Abbott, G., Koorts, H., & Salmon, J. (2024) · Sports Medicine - Open · doi:10.1186/s40798-024-00688-7 · Cited by 62
“To assess potential small-study effects and publication bias for meta-analysis with at least 10 studies (i.e. MVPA, BMI, and WC) [30], funnel plots were produced and the Egger regression asymmetry test [31] was conducted.”
MethodEfficacy and safety of medical cannabinoids in children: a systematic review and meta-analysis
Treves, N., Mor, N., Allegaert, K., Bassalov, H., Berkovitch, M., Stolar, O. E., & Matok, I. (2021) · Scientific Reports · doi:10.1038/s41598-021-02770-6 · Cited by 58
Pollock, M., Fernandes, R. M., Newton, A. S., Scott, S. D., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0914-3 · Cited by 52
Khouri, C., Revol, B., Lepelley, M., Mouffak, A., Bernardeau, C., Salvo, F., Pariente, A., Roustit, M., & Cracowski, J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.07.014 · Cited by 56
Ferraro, M. C., Cashin, A. G., Wand, B. M., Smart, K. M., Berryman, C. F., Marston, L., Moseley, G. L., McAuley, J. H., & O'Connell, N. E. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009416.pub3 · Cited by 76
Contributions of Modifiable Risk Factors to Dementia Incidence: A Bayesian Network Analysis
Liang, J., Lü, L., Li, J., Qu, X.-Y., Li, J., Qian, S., Wang, Y., Jia, R., Wang, C., & Xu, Y. (2020) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2020.04.006 · Cited by 55
Heart Rate Variability and Cognition: A Narrative Systematic Review of Longitudinal Studies
Nicolini, P., Malfatto, G., & Lucchi, T. A. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13010280 · Cited by 61
“Although a systematic review need not contain a meta-analysis [136,137,139] and many systematic reviews are narrative [142–144], including some on HRV and cognition (e.g., [22]), we do recognize that meta-analyses provide the highest level of evidence for healthcare decision-making [139].”
BackgroundFaggion, C. M., Huivin, R., Aranda, L., Pandis, N., & Alarcon, M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.02.011 · Cited by 49
Identification of problems in search strategies in Cochrane Reviews
Franco, J. V. A., Garrote, V. L., Escobar Liquitay, C. M., & Vietto, V. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1302 · Cited by 48
Colder Carras, M., Shi, J., Hard, G., & Saldanha, I. J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0240032 · Cited by 43
“Recent decades have witnessed a surge in the number of systematic reviews conducted [8].”
BackgroundWayant, C., Page, M. J., & Vassar, M. (2019) · JAMA Oncology · doi:10.1001/jamaoncol.2019.2564 · Cited by 46
Self-citationPriorities in Bioeconomy Strategies: A Systematic Literature Review
Papadopoulou, C.-I., Loizou, E., & Chatzitheodoridis, F. (2022) · Energies · doi:10.3390/en15197258 · Cited by 50
“Although numerous studies report that PRISMA analysis is associated with more complete reporting of systematic reviews [24–27], there is a counter argument that corrective actions could be taken to improve it [28].”
BackgroundEffectiveness of smoking cessation interventions among adults: an overview of systematic reviews
Hersi, M., Beck, A., Hamel, C., Esmaeilisaraji, L., Pussegoda, K., Austin, B., Ahmadzai, N., Pratt, M., Thuku, M., Yazdi, F., et al. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02570-9 · Cited by 52
“First, we limited inclusion to Cochrane systematic reviews as evidence suggests that review findings from this source are more likely to have GRADE ratings than non-Cochrane reviews [49].”
BackgroundSelf-citationThe Global Research Collaboration of Network Meta-Analysis: A Social Network Analysis
Li, L., Catalá-López, F., Alonso‐Arroyo, A., Tian, J., Aleixandre‐Benavent, R., Pieper, D., Ge, L., Yao, L., Wang, Q., & Yang, K. (2016) · PLoS ONE · doi:10.1371/journal.pone.0163239 · Cited by 50
“Some countries, particularly China in the last years, have developed a research culture that places a strong emphasis on the production of systematic reviews and meta-analyses [30].”
BackgroundSelf-citationFunnell, S., Jull, J., Mbuagbaw, L. C. E., Welch, V. A., Dewidar, O., Wang, X., Lesperance, M., Ghogomu, E. T., Rizvi, A., Akl, E. A., et al. (2023) · International Journal for Equity in Health · doi:10.1186/s12939-023-01854-1 · Cited by 45
“Our team members have used such methodological assessments to inform the development of other reporting guidelines, such as for randomized cluster trials [70] and systematic reviews [71].”
MethodSelf-citationSaldanha, I. J., Smith, B. T., Ntzani, E., Jap, J., Balk, E. M., & Lau, J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1250-y · Cited by 44
“Our finding that 44% of eligible systematic reviews were registered in PROSPERO is a considerable improvement from the 4% that was reported in a random sample of 300 systematic reviews published in 2014 [11].”
BackgroundInfluentialImproving the conduct of systematic reviews: a process mining perspective
Pham, B., Bagheri, E., Rios, P., Pourmasoumi, A., Robson, R. C., Hwee, J., Isaranuwatchai, W., Darvesh, N., Page, M. J., & Tricco, A. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.011 · Cited by 40
Self-citationNascimento, D. P., Gonzalez, G. Z., Araújo, A. C., Moseley, A. M., Maher, C. G., & Costa, L. O. P. (2019) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2020.8962 · Cited by 46
The methodological rigour of systematic reviews in environmental health
Menon, J. M. L., Struijs, F., & Whaley, P. (2022) · Critical Reviews in Toxicology · doi:10.1080/10408444.2022.2082917 · Cited by 45
“The findings of these studies echo empirical evidence of widespread limitations in the conduct and reporting of SRs in healthcare (Ioannidis 2016; Page et al. 2016; Pussegoda et al. 2017; Gao et al. 2020) and socioeconomic research (Wang et al. 2021).”
BackgroundResultContinued citation of retracted radiation oncology literature—Do we have a problem?
Hamilton, D. G. (2019) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2018.11.014 · Cited by 42
Karampatakis, T., Tsergouli, K., & Lowrie, K. (2023) · Microbial Pathogenesis · doi:10.1016/j.micpath.2023.106090 · Cited by 43
Adoption of the Internet of Things in higher education: opportunities and challenges
Fernández‐Batanero, J. M., Rueda, M. M., Cerero, J. F., & Menéses, E. L. (2023) · Interactive Technology and Smart Education · doi:10.1108/itse-01-2023-0025 · Cited by 42
Cumpston, M., Brennan, S., Ryan, R. E., & McKenzie, J. E. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.02.003 · Cited by 43
Model Checking in Meta-Analysis
Viechtbauer, W. (2020) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315119403-11 · Cited by 53
Brady, R., McSharry, L., Lawson, S., & Regan, J. (2021) · European Journal of Cancer Care · doi:10.1111/ecc.13549 · Cited by 52
Munce, S., Gray, C. S., Pomeroy, B. C., Bayley, M. T., Kokorelias, K. M., Luong, D., Biddiss, E. A., Cave, T., Bragge, P., Chew-Graham, C. A., et al. (2023) · JMIR Research Protocols · doi:10.2196/50463 · Cited by 45
Self-citationCiapponi, A. (2018) · Evidencia actualizacion en la práctica ambulatoria · doi:10.51987/evidencia.v21i1.6834 · Cited by 68
Travis, N., Knoll, M., Cook, S., Oh, H., Cadham, C. J., Sánchez‐Romero, L. M., & Levy, D. T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20031908 · Cited by 50
“This finding is even more staggering compared to a study that found that only one-third of systematic reviews published in MEDLINE in 2014 reported working from a previously developed protocol [48].”
ResultMiao, Y., Luo, Y., Zhao, Y., Liu, M., Wang, H., & Wu, Y. X. (2024) · Journal of Medical Internet Research · doi:10.2196/58013 · Cited by 46
“SUCRA scores range from 0 to 1, with a higher score indicating a higher likelihood that an intervention is the most effective [32].”
Zeraatkar, D., Bhasin, A. A., Morassut, R. E., Churchill, I. F., Gupta, A. A., Lawson, D. O., Miroshnychenko, A., Sirotich, E., Aryal, K., Mikhail, D., et al. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab002 · Cited by 45
“(6), for example, found methodological and reporting limitations to be common in a sample of systematic reviews indexed in MEDLINE.”
BackgroundGusenbauer, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1520 · Cited by 43
“Thus, while quantitative data indicates an increase in the quality of information supply – that is, more evidence-syntheses with overall more rigour in reporting and conduct – the quality of individual studies varies however.(12)”
BackgroundDevelopment of an efficient search filter to retrieve systematic reviews from PubMed
Oliván, J. A. S., Cuenca, G. M., & Avilés, R. A. (2021) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2021.1223 · Cited by 47
Li, T., Hua, F., Dan, S., Zhong, Y., Levey, C., & Song, Y. (2020) · Journal of Dentistry · doi:10.1016/j.jdent.2020.103471 · Cited by 38
Canellas, J. V. D. S., Ritto, F. G., Rodolico, A., Aguglia, E., Fernandes, G. V. D. O., Figueredo, C. M. D. S., & Vettore, M. V. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1135853 · Cited by 38
“Additionally, these organizations produce only a minority of all systematic reviews (Page et al., 2016).”
BackgroundGarcía, L. M., Hernandez, H. P., Superchi, C., de Guzmán, E. N., Ballesteros, M., Ibargoyen-Roteta, N., McFarlane, E., Posso, M., Figuls, M. R. I., del Campo, R. R., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.008 · Cited by 38
Mapping of global scientific research in comorbidity and multimorbidity: A cross-sectional analysis
Catalá-López, F., Alonso‐Arroyo, A., Page, M. J., Hutton, B., Tabarés‐Seisdedos, R., & Aleixandre‐Benavent, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0189091 · Cited by 45
Self-citationEpidemiology and reporting characteristics of preclinical systematic reviews
Hunniford, V. T., Montroy, J. R., Fergusson, D. A., Avey, M. T., Wever, K. E., McCann, S., Foster, M., Fox, G., Lafreniere, M., Ghaly, M., et al. (2021) · PLoS Biology · doi:10.1371/journal.pbio.3001177 · Cited by 44
“Our findings are similar to an evaluation of 300 clinical systematic reviews indexed in February 2014, which found that at least a third of the reviews did not report use of a review protocol, years of coverage of the search, methods for data extraction, and methods for study risk of bias assessment [18].”
ResultSelf-citationSalameh, J.-P., McInnes, M. D. F., Moher, D., Thombs, B. D., McGrath, T. A., Frank, R., Dehmoobad Sharifabadi, A., Kraaijpoel, N., Levis, B., & Bossuyt, P. M. (2019) · Clinical Chemistry · doi:10.1373/clinchem.2018.292987 · Cited by 37
Self-citationHalf of Cochrane reviews were published more than 2 years after the protocol
Andersen, M. Z., Gülen, S., Fonnes, S., Andresen, K., & Rosenberg, J. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.05.011 · Cited by 42
McGrath, T. A., Moher, D., & McInnes, M. D. F. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1090-9 · Cited by 42
Self-citationHan, J. L., Gandhi, S., Bockoven, C. G., Narayan, V. M., & Dahm, P. (2016) · British Journal of Urology · doi:10.1111/bju.13653 · Cited by 36
Meta-analyses Can Be Credible and Useful: A New Standard
Ioannidis, J. P. A. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0035 · Cited by 36
Assessment of Language and Indexing Biases Among Chinese-Sponsored Randomized Clinical Trials
Jia, Y., Huang, D., Wen, J., Wang, Y., Rosman, L. M., Chen, Q., Robinson, K. A., Gagnier, J. J., Ehrhardt, S., & Celentano, D. D. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.5894 · Cited by 36
Lunny, C., Neelakant, T., Chen, A., Shinger, G., Stevens, A., Tasnim, S., Sadeghipouya, S., Adams, S., Zheng, Y. W., Lin, L., et al. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1530 · Cited by 40
“The number of systematic reviews has reached epidemic proportions(10); from none in 1987(11); 7 per day in 2003(5); 11 a day in 2010(12); and 22 per day in 2014.(13) Overviews have grown in number in tandem with the growth in systematic reviews, and they aim to appraise and synthesise results of systematic reviews.”
BackgroundSelf-citationGrey matters; on the importance of publication bias in systematic reviews
Winters, M., & Weir, A. (2016) · British Journal of Sports Medicine · doi:10.1136/bjsports-2016-096679 · Cited by 36
Guidance to best tools and practices for systematic reviews1
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Journal of Pediatric Rehabilitation Medicine · doi:10.3233/prm-230019 · Cited by 49
“Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [28].”
InfluentialNissen, T. E., Wayant, C., Wahlstrom, A., Sinnett, P., Fugate, C. S., Herrington, J., & Vassar, M. B. (2017) · Clinical Obesity · doi:10.1111/cob.12174 · Cited by 36
“To improve the body of available evidence, stakeholders – funders, research institutions and publishers – could incentivize and equip investigators to use highest-quality methodological and reporting practices (44,47), perhaps through the methods recommended by Moher and Altman (48).”
BackgroundKamioka, H., Tsutani, K., Origasa, H., Yoshizaki, T., Kitayuguchi, J., Shimada, M., Wada, Y., & Takano‐Ohmuro, H. (2019) · Nutrients · doi:10.3390/nu11071583 · Cited by 38
“Additionally, an SR may be of limited use if the methods used to conduct the SR are flawed and reporting of the SR was incomplete [6].”
MethodIdentifying the Research and Trends in STEM Education in Pakistan: A Systematic Literature Review
Aslam, S., Saleem, A., Kennedy, T. J., Kumar, T., Parveen, K., Akram, H., & Zhang, B. (2022) · SAGE Open · doi:10.1177/21582440221118545 · Cited by 47
“It is apparent from observational studies that systematic reviews are more likely to be reported in full compliance with the PRISMA 2009 statement when it is implemented (Agha et al., 2016; Page et al., 2016).”
BackgroundShi, J. Y., Hu, Z.-Y., Wen, Y.-R., Wang, Y.-F., Lin, Y., Zhao, H., Lin, Y.-T., & Wang, Y. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1046518 · Cited by 38
Methodological and Reporting Quality of Systematic Reviews and Meta-analyses in Endodontics
Nagendrababu, V., Pulikkotil, S. J., Sultan, O. S., Jayaraman, J., & Peters, O. A. (2018) · Journal of Endodontics · doi:10.1016/j.joen.2018.02.013 · Cited by 37
“Strategies such as Methodological Expectations of Cochrane Intervention Reviews in the editorial process are other reasons for good quality (88).”
BackgroundPawliuk, C., Cheng, S. S. Y., Zheng, A., & Siden, H. B. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111237 · Cited by 41
Torres, G. C. S., Ledbetter, L., Cantrell, S. E., Alomo, A. R. L., Blodgett, T. J., Bongar, M. V. V., Hatoum, S., Hendren, S., Loa, R. F., Montaña, S., et al. (2024) · Journal of Nursing Scholarship · doi:10.1111/jnu.12969 · Cited by 44
Scoping review search practices in the social sciences: A scoping review
Logan, J. R., Webb, J., Singh, N. K., Tanner, N., Barrett, K., Wall, M., Walsh, B., & Ayala, A. P. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1742 · Cited by 36
Puljak, L. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.002 · Cited by 36
Justesen, T. F., Freyberg, J., & Schultz, A. N. Ø. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01281-2 · Cited by 41
“The majority of SRs searched reference lists of the included articles, similar to previous findings [28].”
ResultBüchter, R. B., Weise, A., & Pieper, D. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01438-z · Cited by 39
“It is unsurprising that the Cochrane reviews in our sample had a higher quality of reporting than the nonCochrane reviews, which has been shown for other aspects of reporting previously [10].”
ResultPetersen, J. M., Barrett, M., Ahrens, K. A., Murray, E. J., Bryant, A., HOGUE, C. J. R., Mumford, S. L., Gadupudi, S. S., Fox, M. P., & Trinquart, L. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1544 · Cited by 36
“In a survey of 300 reviews from 2014, 74 focused on exposure-outcome associations and among them, all but one comprised observational studies.(2) A major threat to the validity of these reviews is residual confounding in the component studies.”
BackgroundDosenovic, S., Jelicic Kadic, A., Vucic, K., Markovina, N., Pieper, D., & Puljak, L. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0493-y · Cited by 36
“An updated version of this study, published in 2016, indicated that the number of SRs being published is increasing, but the majority of them are still poorly conducted and reported [9].”
BackgroundRombey, T., Allers, K., Mathes, T., Hoffmann, F., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0698-8 · Cited by 34
“In a random sample of 300 reviews published in 2014, only 4% mentioned that the SR has been registered [27].”
BackgroundRyan, R. E., & Hill, S. (2019) · Health Research Policy and Systems · doi:10.1186/s12961-019-0435-0 · Cited by 34
“This is particularly the case in Cochrane, an international organisation responsible for 15% of all of the world’s systematic reviews [13].”
BackgroundZhang, J., Han, L., Shields, L. E., Tian, J., & Wang, J. (2019) · Medicine · doi:10.1097/md.0000000000018099 · Cited by 35
“Panic reported that only 4 of 90 SRs in gastroenterology and hepatology journals,([23]) 15% in radiology journals,([24]) 27% in orthodontic journals,([10]) and one-third in biomedical research([17]) mentioned research protocols.”
BackgroundMethodResultInfluentialLiang, J.-H., Li, J., Wu, R.-K., Li, J.-Y., Qian, S., Jia, R.-X., Wang, Y.-Q., Qian, Y., & Xu, Y. (2020) · European Child and Adolescent Psychiatry · doi:10.1007/s00787-020-01492-w · Cited by 23
Methodological quality of systematic reviews on treatments for osteoporosis: A cross-sectional study
Tsoi, A. K., Ho, L. T. F., Wu, I. X. Y., Wong, C. H. L., Ho, R. S. T., Lim, J. Y. Y., Mao, C., Lee, E. K. P., & Chung, V. C. (2020) · Bone · doi:10.1016/j.bone.2020.115541 · Cited by 32
Santis, K. K. D., Lorenz, R. C., Lakeberg, M., & Matthias, K. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1532 · Cited by 35
Zhou, X., Zhao, P., Guo, X., Wang, J., & Wang, R. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.935176 · Cited by 35
“SUCRA with a higher value denotes a greater likelihood of a given treatment being in the top rank or highly effective, while zero indicates that the treatment is definitely the worst (Page et al., 2016).”
Background
Metadata
- Authors
- Page, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R., & Moher, D.
- Affiliations
- University of Bristol (GB)
- University of Ottawa (CA)
- University of Oxford (GB)
- Ottawa Hospital Research Institute (CA)
- Children's Hospital of Eastern Ontario (CA)
- St. Michael's Hospital (CA)
- Universitat de València (ES)
- Lanzhou University (CN)
- Vancouver General Hospital (CA)
- Corresponding author
- Moher, David
- ORCID
- Venue
- PLOS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-05-24
- Volume
- 13
- Issue
- 5
- Pages
- e1002028
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4878797
- OpenAlex
- W2396846174
- Semantic Scholar
- 363d54e2baf229adfd585079bfd7c18ae8feed26
- Cited by
- 768
- References
- 41
- Citations by year
- 2026: 25 · 2025: 43 · 2024: 85 · 2023: 73 · 2022: 85 · 2021: 146 · 2020: 95 · 2019: 83 · 2018: 58 · 2017: 60 · 2016: 15
- FWCI
- 24.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Explore Bristol Research (publishedVersion, other-oa)
- Monash University Research Portal (Monash University) (acceptedVersion, other-oa)
- Lanzhou University Institutional Repository (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, other-oa)
- Funders
- National Institute for Health and Care Research
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
- Governo Brasil
- Generalitat Valenciana (PROMETEOII/2015/021)
- Canadian Institutes of Health Research (1088535)
- Medical Research Council (MR/J00488X/1)
- National Health and Medical Research Council (1088535)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- systematic reviews, reporting quality, meta-analysis, risk of bias assessment, publication bias, research funding disclosure, biomedical literature, research quality improvement, research integrity, research transparency, research accountability, research reproducibility, research credibility, research trustworthiness
- MeSH terms
- Cross-Sectional Studies, Epidemiologic Methods, Humans, Research Design, Review Literature as Topic, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.