Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews (SRs) can help decision makers interpret the deluge of published biomedical literature. However, a SR may be of limited use if the methods used to conduct the SR are flawed, and reporting of the SR is incomplete. To our knowledge, since 2004 there has been no cross-sectional study of the prevalence, focus, and completeness of reporting of SRs across different specialties. Therefore, the aim of our study was to investigate the epidemiological and reporting characteristics of a more recent cross-section of SRs. METHODS AND FINDINGS: We searched MEDLINE to identify potentially eligible SRs indexed during the month of February 2014. Citations were screened using prespecified eligibility criteria. Epidemiological and reporting characteristics of a random sample of 300 SRs were extracted by one reviewer, with a 10% sample extracted in duplicate. We compared characteristics of Cochrane versus non-Cochrane reviews, and the 2014 sample of SRs versus a 2004 sample of SRs. We identified 682 SRs, suggesting that more than 8,000 SRs are being indexed in MEDLINE annually, corresponding to a 3-fold increase over the last decade. The majority of SRs addressed a therapeutic question and were conducted by authors based in China, the UK, or the US; they included a median of 15 studies involving 2,072 participants. Meta-analysis was performed in 63% of SRs, mostly using standard pairwise methods. Study risk of bias/quality assessment was performed in 70% of SRs but was rarely incorporated into the analysis (16%). Few SRs (7%) searched sources of unpublished data, and the risk of publication bias was considered in less than half of SRs. Reporting quality was highly variable; at least a third of SRs did not report use of a SR protocol, eligibility criteria relating to publication status, years of coverage of the search, a full Boolean search logic for at least one database, methods for data extraction, methods for study risk of bias assessment, a primary outcome, an abstract conclusion that incorporated study limitations, or the funding source of the SR. Cochrane SRs, which accounted for 15% of the sample, had more complete reporting than all other types of SRs. Reporting has generally improved since 2004, but remains suboptimal for many characteristics. CONCLUSIONS: An increasing number of SRs are being published, and many are poorly conducted and reported. Strategies are needed to help reduce this avoidable waste in research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many systematic review reports include narrative summaries of the characteristics and risk of bias across all included studies.36”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even for those studies which explicitly reference PRISMA, there is only slight, statistically non-significant evidence of improved reporting, as found by Page et al. [15].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The studies included in a systematic review will generally vary with respect to design, study populations, and risk of bias [1].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“in a survey of 300 systematic reviews, 64% of the reviews included observational studies [7].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although few published evaluations exist, evidence suggests that the endorsement of PRISMA by journals is associated with more complete reporting of systematic reviews than in journals not endorsing the reporting guideline.50 51 52”
From Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. (2022) · cites this paper as [51]
Experimental — accuracy not yet measured
23 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Self-citationPreferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Self-citationPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Self-citationThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationPRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citation
Research works printed but not linked: 2
- Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. AHRQ Publication No. 10(14)-EHC063-EF. 2014 Jan [cited 21 Apr 2016]. Available: https://www.effectivehealthcare.ahrq.gov/ehc/products/60/318/CER-Methods-Guide-140109.pdf. 21433403
- Nordic Cochrane Centre. Review Manager (RevMan), version 5.1. Copenhagen: Nordic Cochrane Centre; 2011.
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
Self-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationThe PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Self-citationShea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
Self-citationThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Self-citationThe PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003583 · Cited by 5,148
“completeness of published systematic reviews,[17, 21, 36, 37] reviewed the items included in other documents providing guidance for systematic reviews,[38] surveyed systematic review methodologists and journal editors for their views on how to revise the original PRISMA statement,[35] discussed the findings at an in-p…”
BackgroundMethodSelf-citationSynthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline
Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6890 · Cited by 4,628
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.001 · Cited by 4,265
Self-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Self-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Self-citationUpdating guidance for reporting systematic reviews: development of the PRISMA 2020 statement
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., & Moher, D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.003 · Cited by 3,490
Self-citationHaddaway, N. R., Page, M. J., Pritchard, C. C., & McGuinness, L. A. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1230 · Cited by 3,486
Self-citationDeclaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Revista Española de Cardiología · doi:10.1016/j.recesp.2021.06.016 · Cited by 2,095
Self-citationThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Declaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Revista Española de Cardiología (English Edition) · doi:10.1016/j.rec.2021.07.010 · Cited by 1,313
“We evaluated the reporting completeness of published systematic reviews [17 , 21 , 36 , 37] , reviewed the items included in other documents providing guidance for systematic reviews [38] , surveyed systematic review methodologists and journal editors for their views on how to revise the original”
MethodSelf-citationHow to properly use the PRISMA Statement
Sarkis-Onofre, R., Catalá-López, F., Aromataris, E., & Lockwood, C. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01671-z · Cited by 1,104
Self-citationSmith, M., Hosking, J., Woodward, A., Witten, K., MacMillan, A., Field, A., Baas, P., & Mackie, H. (2017) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-017-0613-9 · Cited by 888
“, identification of articles, data extraction, quality assessment) were duplicated by a coauthor (JH) with a random selection of 10% of each dataset [47, 48].”
MethodPage, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Self-citationGates, M., Gates, A., Pieper, D., Fernandes, R. M., Tricco, A. C., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070849 · Cited by 856
Self-citationPage, M. J., & Moher, D. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0663-8 · Cited by 769
Self-citationA declaração PRISMA 2020: diretriz atualizada para relatar revisões sistemáticas
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2022) · Revista Panamericana de Salud Pública · doi:10.26633/rpsp.2022.112 · Cited by 433
Self-citationMOOSE Reporting Guidelines for Meta-analyses of Observational Studies
Brooke, B. S., Schwartz, T. A., & Pawlik, T. M. (2021) · JAMA Surgery · doi:10.1001/jamasurg.2021.0522 · Cited by 742
Salameh, J.-P., Bossuyt, P. M., McGrath, T. A., Thombs, B. D., Hyde, C. J., Macaskill, P., Deeks, J. J., Leeflang, M., Korevaar, D. A., Whiting, P., et al. (2020) · BMJ · doi:10.1136/bmj.m2632 · Cited by 719
Self-citationDekkers, O. M., Vandenbroucke, J. P., Cevallos, M., Renehan, A. G., Altman, D. G., & Egger, M. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002742 · Cited by 525
Self-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & Group, P.-S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.962 · Cited by 409
“Post-publication peer review of search methods, even amongst Cochrane reviews, which generally have superior reporting compared to nonCochrane reviews [14], has shown that reporting that appears complete may still pose challenges for those wishing to reproduce searches [19–23].”
ResultSelf-citationRegistration of systematic reviews in PROSPERO: 30,000 records and counting
Page, M. J., Shamseer, L., & Tricco, A. C. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0699-4 · Cited by 509
“Yet these organisations produce a minority of all published SRs [1, 2], so most would only become known at the time the SR was completed and published.”
BackgroundMethodResultInfluentialSelf-citationCumpston, M. S., McKenzie, J. E., Welch, V. A., & Brennan, S. E. (2022) · Journal of Public Health · doi:10.1093/pubmed/fdac036 · Cited by 464
Ardern, C. L., Büttner, F., Andrade, R., Weir, A., Ashe, M. C., Holden, S., Impellizzeri, F. M., Delahunt, E., Dijkstra, H. P., Mathieson, S., et al. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2021-103987 · Cited by 458
Self-citationNussbaumer-Streit, B., Klerings, I., Dobrescu, A. I., Persad, E., Stevens, A., Garritty, C., Kamel, C., Affengruber, L., King, V. J., & Gartlehner, G. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.10.011 · Cited by 440
Missed nursing care: An overview of reviews
Chaboyer, W., Harbeck, E., Lee, B., & Grealish, L. (2021) · The Kaohsiung Journal of Medical Sciences · doi:10.1002/kjm2.12308 · Cited by 381
“There has been an exponential rise in the numbers of systematic reviews with one group estimating about 22 new systematic reviews are published each day.(18) As a result, overviews of reviews have emerged as another methodology to systematically collate, appraise, and synthesize the results of several systematic revie…”
BackgroundAssessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402
Self-citationMueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
“In light of the increasing use of observational data in systematic reviews and meta-analyses [7], it is crucial for reviewers to have access to sound methodological advice on how to conduct systematic reviews and meta-analyses of observational data.”
BackgroundSystematic review adherence to methodological or reporting quality
Pussegoda, K., Turner, L., Garritty, C., Mayhew, A., Skidmore, B., Stevens, A., Boutron, I., Sarkis-Onofre, R., Bjerre, L. M., Hróbjartsson, A., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0527-2 · Cited by 322
“SR evidence has become increasingly important in clinical decision-making and for informing clinical guidelines and health policy [2, 3].”
Self-citationGe, L., Tian, J.-H., Li, Y.-N., Pan, J.-X., Li, G., Wei, D., Xing, X., Pan, B., Chen, Y.-L., Song, F.-J., et al. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.012 · Cited by 322
“It was estimated that more than 8,000 systematic reviews were indexed in MEDLINE annually , corresponding to a 3-fold increase over the last decade [3].”
BackgroundResultInfluentialHunt, H., Pollock, A., Campbell, P., Estcourt, L., & Brunton, G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0695-8 · Cited by 314
Stop this waste of people, animals and money
Moher, D., Shamseer, L., Cobey, K. D., Lalu, M. M., Galipeau, J., Avey, M. T., Ahmadzai, N., Alabousi, M., Barbeau, P., Beck, A., et al. (2017) · Nature · doi:10.1038/549023a · Cited by 281
Self-citationPravila PRISMA 2020.: ažurirane smjernice za izvještavanje u sustavnim pregledima
Page, M. J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McGuinness, L., McDonald, S., Stewart, L. A., et al. (2021) · Medicina Fluminensis · doi:10.21860/medflum2021_264903 · Cited by 163
Self-citationPage, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278
“The risk of bias due to selective non-reporting in the included studies was assessed in only 24% of reviews.(7) Another study showed that authors of Cochrane reviews routinely record whether any outcomes that were measured were not reported in the included trials, yet rarely consider if such non-reporting could have b…”
BackgroundSelf-citationHoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T., & Pieper, D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.022 · Cited by 278
Krnic Martinic, M., Pieper, D., Glatt, A., & Puljak, L. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0855-0 · Cited by 240
“A study published in 2016 indicated that more than 8000 systematic reviews were being indexed annually in MEDLINE, corresponding to a three-fold increase over the last decade [5].”
BackgroundMethodMcKeown, S., & Mir, Z. M. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01583-y · Cited by 251
“Unable to retrieve full-text publication for 10 false-positive duplicate references within EndNote X9 Unable to retrieve full-text publication for 2 false-positive duplicate references within Rayyan McKeown and Mir Systematic Reviews (2021) 10:38 Page 5 of 8”
BackgroundNaci, H., Davis, C., Savović, J., Higgins, J. P. T., Sterne, J. A. C., Gyawali, B., Romo-Sandoval, X., Handley, N., & Booth, C. M. (2019) · BMJ · doi:10.1136/bmj.l5221 · Cited by 243
Artificial Intelligence in the Tourism Industry: An Overview of Reviews
García-Madurga, M.-Á., & Grilló-Méndez, A.-J. (2023) · Administrative Sciences · doi:10.3390/admsci13080172 · Cited by 166
The methodological and reporting quality of systematic reviews from China and the USA are similar
Tian, J., Zhang, J., Ge, L., Yang, K., & Song, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.12.004 · Cited by 248
“It should be emphasized that the improvement in methodological quality is also relevant to authors from the USA and likely to be relevant to systematic reviews by authors from any countries [13].”
BackgroundResultInfluentialKennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0925-0 · Cited by 212
“in 2016, this estimate was updated to 25 systematic reviews published each day [2].”
BackgroundPage, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T., Mulrow, C. D., Shamseer, L., & Moher, D. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.11.010 · Cited by 211
“Six meta-research studies (21, 23, 24, 31-33) in the lead author’s personal collection which included an evaluation of the reporting quality of SRs using a tailored reporting checklist were also reviewed.”
BackgroundMethodSelf-citationGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 237
“As a result, information that may be relevant to the critical appraisal of non-Cochrane reviews is often removed or is relegated to online-only supplements that may not be readily or fully accessible [28].”
InfluentialSMEs Must Go Online—E-Commerce as an Escape Hatch for Resilience and Survivability
Costa, J., & Castro, R. (2021) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer16070166 · Cited by 249
Errors in search strategies used in systematic reviews and their effects on information retrieval
Salvador-Oliván, J. A., Marco-Cuenca, G., & Arquero-Avilés, R. (2019) · Journal of the Medical Library Association · doi:10.5195/jmla.2019.567 · Cited by 199
Hoffmann, T. C., Oxman, A. D., Ioannidis, J. P., Moher, D., Lasserson, T. J., Tovey, D. I., Stein, K., Sutcliffe, K., Ravaud, P., Altman, D. G., et al. (2017) · BMJ · doi:10.1136/bmj.j2998 · Cited by 194
Self-citationFrench, D. P., Cameron, E., Benton, J. S., Deaton, C., & Harvie, M. (2017) · Annals of Behavioral Medicine · doi:10.1007/s12160-017-9895-z · Cited by 180
“This review quality is encouraging, given that a recent systematic survey of systematic reviews found many examples of poor quality, including at least a third omitting descriptions of search strategies or methods of quality appraisal [35].”
BackgroundA declaração PRISMA 2020: diretriz atualizada para relatar revisões sistemáticas
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T., Mulrow, C. D., Shamseer, L., Tetzlaff, J., Akl, E. A., Brennan, S., et al. (2022) · Epidemiologia e Serviços de Saúde · doi:10.5123/s1679-49742022000200033 · Cited by 387
Self-citationGusenbauer, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1729 · Cited by 196
“12 Citation searching is particularly relevant in fields that use diverse, non-standardized language that might be difficult to capture with language-based keyword searching alone.”
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171
Self-citationConduct and reporting of citation searching in Cochrane systematic reviews: A cross‐sectional study
Briscoe, S., Bethel, A., & Rogers, M. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1355 · Cited by 150
“Of the 45 Cochrane reviews, 84% reported backward citation searching and 18% reported other search methods including forward citation searching.(10) These findings are similar to the prevalence of backward (87%) and forward (9%) citation searching in our sample of Cochrane reviews.”
BackgroundResultInfluentialCastellini, G., Bruschettini, M., Gianola, S., Gluud, C., & Moja, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0770-1 · Cited by 161
Mental health during and after protests, riots and revolutions: A systematic review
Ni, M. Y., Kim, Y., McDowell, I., Wong, S., Qiu, H., Wong, I. O., Galea, S., & Leung, G. M. (2020) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867419899165 · Cited by 143
“Due to the heterogeneity of the identified studies, a meta-analysis may not be appropriate (Higgins and Green, 2012b; Page et al., 2016).”
BackgroundGates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148
“As the standards for the conduct and reporting of SRs have become more stringent [4], they have become more labor-intensive to produce.”
BackgroundTawfik, G. M., Giang, H. T. N., Ghozy, S. A., Altibi, A. M. A., Kandil, H., Le, H., Eid, P. S., Radwan, I., Makram, O. M., Hien, T. T. T., et al. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01094-9 · Cited by 154
Pieper, D., & Puljak, L. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.027 · Cited by 159
Hamedani, S. S., Aslam, S., Oraibi, B.-A. M., Wah, Y. B., & Hamedani, S. S. (2024) · Education Sciences · doi:10.3390/educsci14101041 · Cited by 171
Ikeda, E., Hinckson, E. A., Witten, K., & Smith, M. (2018) · Health & Place · doi:10.1016/j.healthplace.2018.09.009 · Cited by 148
Veroniki, A. A., Hutton, B., Stevens, A., McKenzie, J. E., Page, M. J., Moher, D., McGowan, J., Straus, S. E., Li, T., Munn, Z., et al. (2025) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00308 · Cited by 110
Self-citationGianola, S., Jesus, T. S., Bargeri, S., & Castellini, G. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240123 · Cited by 143
“However, no reviews to date have focused on the epidemiology and the reporting characteristics of publications related to the COVID-19 pandemic [11, 12] or the advances in research during the pandemic [13–15].”
BackgroundOverviews of systematic reviews: great promise, greater challenge
McKenzie, J. E., & Brennan, S. E. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0582-8 · Cited by 134
“In 2014, it was estimated that more than 8000 systematic reviews were indexed annually on MEDLINE, a threefold increase over the last decade [3].”
BackgroundPollock, M., Fernandes, R. M., Newton, A. S., Scott, S. D., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0768-8 · Cited by 132
Wolfenden, L., Barnes, C., Lane, C., McCrabb, S., Brown, H., Gerritsen, S., Barquera, S., Véjar-Rentería, L. S., Munguía, A., & Yoong, S. L. (2021) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-01046-y · Cited by 142
“Umbrella reviews provide a means of rapidly and efficiently synthesising evidence from a broad body of research, [27, 28] and can assist decision-makers in choosing from different intervention strategies; thus, they are frequently Wolfenden et al.”
BackgroundThe “realist search”: A systematic scoping review of current practice and reporting
Booth, A., Briscoe, S., & Wright, J. M. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1386 · Cited by 141
Tricco, A. C., Zarin, W., Ghassemi, M., Nincic, V., Lillie, E., Page, M. J., Shamseer, L., Antony, J., Rios, P., Hwee, J., et al. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.014 · Cited by 122
Self-citationAyorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · PLoS ONE · doi:10.1371/journal.pone.0227580 · Cited by 127
“Another study which examined a random sample of 300 systematic reviews of biomedical research indexed in MEDLINE in February 2014 found that 31% had formally assessed publication bias [9].”
BackgroundCumpston, M., McKenzie, J. E., Thomas, J., & Brennan, S. (2020) · F1000Research · doi:10.12688/f1000research.24469.1 · Cited by 134
The contribution of databases to the results of systematic reviews: a cross-sectional study
Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0232-1 · Cited by 118
“The CDSR was chosen for this analysis as Cochrane reviews: 1) provide tabulated data from the component trials allowing for re-analysis; 2) provide a detailed list of references for all relevant trials; and, 3) have been reported to be of higher methodological quality [26, 27], which may translate into more comprehens…”
MethodResistance training and health in adults: an overview of systematic reviews
El-Kotob, R., Ponzano, M., Chaput, J., Janssen, I., Kho, M. E., Poitras, V. J., Ross, R., Ross‐White, A., Saunders, T. J., & Giangregorio, L. (2020) · Applied Physiology, Nutrition, and Metabolism · doi:10.1139/apnm-2020-0245 · Cited by 136
Pharmacological treatments for low back pain in adults: an overview of Cochrane Reviews
Cashin, A. G., Wand, B. M., O'Connell, N. E., Lee, H., Rizzo, R. R. N., Bagg, M. K., O’Hagan, E. T., Maher, C. G., Furlan, A. D., van Tulder, M. W., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013815.pub2 · Cited by 158
Reproducible research practices are underused in systematic reviews of biomedical interventions
Page, M. J., Altman, D. G., Shamseer, L., McKenzie, J. E., Ahmadzai, N., Wolfe, D., Yazdi, F., Catalá-López, F., Tricco, A. C., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.017 · Cited by 115
“a systematic search and screening by two authors independently) [17].”
BackgroundMethodInfluentialSelf-citationGates, M., Gates, A., Guitard, S., Pollock, M., & Hartling, L. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01509-0 · Cited by 115
“It is estimated that 8000 systematic reviews were published in 2014 [2], more than three times the yearly publication rate recorded 10 years earlier [3].”
BackgroundPollock, M., Fernandes, R. M., & Hartling, L. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0325-5 · Cited by 107
Chaabna, K., Cheema, S., Abraham, A., Maisonneuve, P., Lowenfels, A. B., & Mamtani, R. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01552-x · Cited by 110
Network meta‐analysis of rare events using the Mantel‐Haenszel method
Efthimiou, O., Rücker, G., Schwarzer, G., Higgins, J. P. T., Egger, M., & Salanti, G. (2019) · Statistics in Medicine · doi:10.1002/sim.8158 · Cited by 107
“, (Nd) ) T and a matrix V = diag(V (1),V (2),.”
BackgroundMethodBerberine and health outcomes: An umbrella review
Li, Z., Yang, W., Xu, Q., Ma, J., Li, X., Yan, J., Tian, Y., Wen, Y., & Chen, T. (2023) · Phytotherapy Research · doi:10.1002/ptr.7806 · Cited by 119
“Systematic reviews and meta-analyses can guide clinical decisions and provide evidence to support clinical guidelines and health policies (Page et al., 2016; Pussegoda et al., 2017).”
BackgroundKoffel, J. B., & Rethlefsen, M. L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0163309 · Cited by 107
Baudard, M., Yavchitz, A., Ravaud, P., Perrodeau, E., & Boutron, I. (2017) · BMJ · doi:10.1136/bmj.j448 · Cited by 107
“31 A previous study by Hart in 2012 aimed to reanalyse meta-analyses by adding unpublished outcome data from trials obtained from the US Food and Drug Administration to published meta-analyses.”
MethodRathore, A., & Lom, B. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0514-7 · Cited by 106
“Finally, with respect to systematic reviews, recent evidence has indicated that despite the increased utility and popularity of this kind of format a significant proportion of published reviews are both poorly conducted and reported [37].”
Assessment of Publication Trends of Systematic Reviews and Randomized Clinical Trials, 1995 to 2017
Niforatos, J. D., Weaver, M., & Johansen, M. E. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2019.3013 · Cited by 99
Hansen, C., Lundh, A., Rasmussen, K., & Hróbjartsson, A. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000047.pub2 · Cited by 96
Interventions for postoperative pain in children: An overview of systematic reviews
Borić, K., Došenović, S., Kadić, A. J., Batinić, M., Ćavar, M., Urlić, M., Markovina, N., & Puljak, L. (2017) · Pediatric Anesthesia · doi:10.1111/pan.13203 · Cited by 102
Systematic reviews with published protocols compared to those without: more effort, older search
Allers, K., Hoffmann, F., Mathes, T., & Pieper, D. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.005 · Cited by 98
Quality of flow diagram in systematic review and/or meta-analysis
Vu-Ngoc, H., Elawady, S. S., Mehyar, G. M., Abdelhamid, A. H., Mattar, O. M., Halhouli, O., Vuong, N. L., Ali, C. D. M., Hassan, U. H., Kien, N. D., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195955 · Cited by 99
Reproducibility of individual effect sizes in meta-analyses in psychology
Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., Olsson-Collentine, A., & Wicherts, J. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0233107 · Cited by 82
“When these decisions and calculations are not carefully undertaken and specifically reported, the methodological quality of the meta-analysis cannot be assessed [6,7].”
BackgroundRapid review method series: interim guidance for the reporting of rapid reviews
Stevens, A., Hersi, M., Garritty, C., Hartling, L., Shea, B., Stewart, L. A., Welch, V. A., & Tricco, A. C. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-112899 · Cited by 101
Self-citationMarshall, Z., Welch, V. A., Minichiello, A., Swab, M., Brunger, F., & Kaposy, C. (2019) · Transgender Health · doi:10.1089/trgh.2018.0020 · Cited by 86
Associations and risk factors for dental trauma: A systematic review of systematic reviews
Magno, M. B., Nadelman, P., Leite, K. L. D. F., Ferreira, D. M. T. P., Pithon, M. M., & Maia, L. C. (2020) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.12574 · Cited by 100
Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review
Abou-Setta, A. M., Jeyaraman, M., Attia, A., Al-Inany, H. G., Ferri, M., Ansari, M. T., Garritty, C. M., Bond, K., & Norris, S. L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0165903 · Cited by 83
Goldkuhle, M., Narayan, V. M., Weigl, A., Dahm, P., & Skoetz, N. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-020869 · Cited by 88
Restrictions and their reporting in systematic reviews of effectiveness: an observational study
Helbach, J., Pieper, D., Mathes, T., Rombey, T., Zeeb, H., Allers, K., & Hoffmann, F. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01710-w · Cited by 94
“Another development over recent years has been the rapid increase in the number of SRs [1, 2, 5], which may have led to research waste [5, 6].”
BackgroundResultInfluentialPage, M. J., Altman, D. G., McKenzie, J. E., Shamseer, L., Ahmadzai, N., Wolfe, D., Yazdi, F., Catalá-López, F., Tricco, A. C., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.022 · Cited by 82
“, dentistry [8]), or to Cochrane reviews, which represent only 15% of all SRs of biomedical research [14].”
BackgroundSelf-citationEpistemonikos: a comprehensive database of systematic reviews for health decision-making
on behalf of Epistemonikos project, Rada, G., Pérez, D., Araya‐Quintanilla, F., Ávila, C., Bravo‐Soto, G., Bravo-Jeria, R., Cánepa, A., Capurro, D., Castro-Gutiérrez, V., et al. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01157-x · Cited by 87
Mass Production of Systematic Reviews and Meta‐analyses: An Exercise in Mega‐silliness?
Page, M. J., & Moher, D. (2016) · Milbank Quarterly · doi:10.1111/1468-0009.12211 · Cited by 86
Self-citationHow is AMSTAR applied by authors – a call for better reporting
Pieper, D., Könsgen, N., Breuing, J., Ge, L., & Wegewitz, U. E. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0520-z · Cited by 85
“Nevertheless, some of our sample characteristics are comparable to those of other studies in terms of quality/risk of bias assessment [1] and number of included studies per review/overview [2, 3, 16].”
BackgroundResult
Metadata
- Authors
- Page, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R., & Moher, D.
- Affiliations
- University of Bristol (GB)
- University of Ottawa (CA)
- University of Oxford (GB)
- Ottawa Hospital Research Institute (CA)
- Children's Hospital of Eastern Ontario (CA)
- St. Michael's Hospital (CA)
- Universitat de València (ES)
- Lanzhou University (CN)
- Vancouver General Hospital (CA)
- Corresponding author
- Moher, David
- ORCID
- Venue
- PLOS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-05-24
- Volume
- 13
- Issue
- 5
- Pages
- e1002028
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4878797
- OpenAlex
- W2396846174
- Semantic Scholar
- 363d54e2baf229adfd585079bfd7c18ae8feed26
- Cited by
- 768
- References
- 41
- Citations by year
- 2026: 25 · 2025: 43 · 2024: 85 · 2023: 73 · 2022: 85 · 2021: 146 · 2020: 95 · 2019: 83 · 2018: 58 · 2017: 60 · 2016: 15
- FWCI
- 24.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Explore Bristol Research (publishedVersion, other-oa)
- Monash University Research Portal (Monash University) (acceptedVersion, other-oa)
- Lanzhou University Institutional Repository (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, other-oa)
- Funders
- National Institute for Health and Care Research
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
- Governo Brasil
- Generalitat Valenciana (PROMETEOII/2015/021)
- Canadian Institutes of Health Research (1088535)
- Medical Research Council (MR/J00488X/1)
- National Health and Medical Research Council (1088535)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- systematic reviews, reporting quality, meta-analysis, risk of bias assessment, publication bias, research funding disclosure, biomedical literature, research quality improvement, research integrity, research transparency, research accountability, research reproducibility, research credibility, research trustworthiness
- MeSH terms
- Cross-Sectional Studies, Epidemiologic Methods, Humans, Research Design, Review Literature as Topic, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.