Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study
Open accessThe DOI matches its Crossref record; 44 of the 47 references it cites were found in a registry or an index.
Abstract
Objective: The academic and scientific community has reacted at pace to gather evidence to help and inform about COVID-19. Concerns have been raised about the quality of this evidence. The aim of this review was to map the nature, scope and quality of evidence syntheses on COVID-19 and to explore the relationship between review quality and the extent of researcher, policy and media interest. Design and setting: A meta-research: systematic review of reviews. Information sources: PubMed, Epistemonikos COVID-19 evidence, the Cochrane Library of Systematic Reviews, the Cochrane COVID-19 Study Register, EMBASE, CINAHL, Web of Science Core Collection and the WHO COVID-19 database, searched between 10 June 2020 and 15 June 2020. Eligibility criteria: Any peer-reviewed article reported as a systematic review, rapid review, overview, meta-analysis or qualitative evidence synthesis in the title or abstract addressing a research question relating to COVID-19. Articles described as meta-analyses but not undertaken as part of a systematic or rapid review were excluded. Study selection and data extraction: Abstract and full text screening were undertaken by two independent reviewers. Descriptive information on review type, purpose, population, size, citation and attention metrics were extracted along with whether the review met the definition of a systematic review according to six key methodological criteria. For those meeting all criteria, additional data on methods and publication metrics were extracted. Risk of bias: For articles meeting all six criteria required to meet the definition of a systematic review, AMSTAR-2 ((A MeaSurement Tool to Assess systematic Reviews, version 2.0) was used to assess the quality of the reported methods. Results: 2334 articles were screened, resulting in 280 reviews being included: 232 systematic reviews, 46 rapid reviews and 2 overviews. Less than half reported undertaking critical appraisal and a third had no reproducible search strategy. There was considerable overlap in topics, with discordant findings. Eighty-eight of the 280 reviews met all six systematic review criteria. Of these, just 3 were rated as of moderate or high quality on AMSTAR-2, with the majority having critical flaws: only a third reported registering a protocol, and less than one in five searched named COVID-19 databases. Review conduct and publication were rapid, with 52 of the 88 systematic reviews reported as being conducted within 3 weeks, and a half published within 3 weeks of submission. Researcher and media interest, as measured by altmetrics and citations, was high, and was not correlated with quality. Discussion: This meta-research of early published COVID-19 evidence syntheses found low-quality reviews being published at pace, often with short publication turnarounds. Despite being of low quality and many lacking robust methods, the reviews received substantial attention across both academic and public platforms, and the attention was not related to the quality of review methods. Interpretation: Flaws in systematic review methods limit the validity of a review and the generalisability of its findings. Yet, by being reported as ‘systematic reviews’, many readers may well regard them as high-quality evidence, irrespective of the actual methods undertaken. The challenge especially in times such as this pandemic is to provide indications of trustworthiness in evidence that is available in ‘real time’. PROSPERO registration number: CRD42020188822.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Finally, poor reporting has been commonly investigated in literature 37 38 ; owing to the limited information of the data extraction process reported by review authors, we could not fully investigate the association between good practice recommendations and the likelihood of data extraction.”
From Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study (2022) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
The psychological impact of quarantine and how to reduce it: Rapid review of the evidence
Brooks, S. K., Webster, R. K., Smith, L. E., Woodland, L., Wessely, S., Greenberg, N., & Rubin, G. J. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30460-8 · Cited by 16,378
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636 · cited 1×
“We chose to concentrate on seven key AMSTAR-2 domains to assess methodological quality, as recommended by Shea et al , 17 and, therefore, did not rate the quality on all aspects of the review methods.”
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J. P. T., Green, S. R., & Higgins, J. P. T. (2008) · Cited by 7,171
Chu, D. K., Akl, E. A., Duda, S., Solo, K., Yaacoub, S., Schünemann, H. J., Chu, D. K., Akl, E. A., El-harakeh, A., Bognanni, A., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)31142-9 · Cited by 3,843
Clinical, laboratory and imaging features of COVID-19: A systematic review and meta-analysis
Rodriguez-Morales, A. J., Cardona-Ospina, J. A., Gutiérrez-Ocampo, E., Villamizar-Peña, R., Holguin-Rivera, Y., Escalera-Antezana, J. P., Alvarado-Arnez, L. E., Bonilla-Aldana, D. K., Franco-Paredes, C., Henao-Martinez, A. F., et al. (2020) · Travel Medicine and Infectious Disease · doi:10.1016/j.tmaid.2020.101623 · Cited by 2,542
Cited by
Infodemics and health misinformation: a systematic review of reviews
Borges do Nascimento, I. J., Beatriz Pizarro, A., Almeida, J., Azzopardi-Muscat, N., André Gonçalves, M., Björklund, M., & Novillo-Ortiz, D. (2022) · Bulletin of the World Health Organization · doi:10.2471/blt.21.287654 · Cited by 605
The rapid, massive growth of COVID-19 authors in the scientific literature
Ioannidis, J. P. A., Salholz-Hillel, M., Boyack, K. W., & Baas, J. (2020) · doi:10.1101/2020.12.15.422900 · Cited by 107
The rapid, massive growth of COVID-19 authors in the scientific literature
Ioannidis, J. P. A., Salholz-Hillel, M., Boyack, K. W., & Baas, J. (2021) · Royal Society Open Science · doi:10.1098/rsos.210389 · Cited by 107
Massive covidization of research citations and the citation elite
Ioannidis, J. P. A., Bendavid, E., Salholz‐Hillel, M., Boyack, K. W., & Baas, J. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2204074119 · Cited by 92
Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study
Xu, C., Yu, T., Furuya-Kanamori, L., Lin, L., Zorzela, L., Zhou, X., Dai, H., Loke, Y., & Vohra, S. (2022) · BMJ · doi:10.1136/bmj-2021-069155 · Cited by 82
Metadata
- Authors
- Abbott, R., Bethel, A., Rogers, M., Whear, R., Orr, N., Shaw, L., Stein, K., & Thompson Coon, J.
- Affiliations
- University of Exeter (GB)
- Corresponding author
- Abbott, Rebecca
- ORCID
- Venue
- BMJ Evidence-Based Medicine
- Publisher
- BMJ
- Published
- 2021-06-03
- Volume
- 27
- Issue
- 3
- Pages
- 169-177
- ISSN
- 1356-5524 (ISSN-L)
- 2515-446X
- 2515-4478
- PMC
- PMC9132873
- OpenAlex
- W3164392163
- Cited by
- 56
- References
- 47
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 12 · 2023: 13 · 2022: 16 · 2021: 5
- FWCI
- 0.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ Evidence-Based Medicine (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institute for Health Research Applied Research Collaboration South West Peninsula (Not applicable)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- COVID-19 pandemic, evidence synthesis, review quality, infodemic, AMSTAR 2, critical appraisal, rapid review, media coverage of research, publication bias, research ethics committees, research governance
- MeSH terms
- Systematic Reviews as Topic, COVID-19, Infodemic, Humans, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. https://app.cochat.ai/papers/10.1136/bmjebm-2021-111710
- The paper
- Abbott, R., Bethel, A., Rogers, M., Whear, R., Orr, N., Shaw, L., Stein, K., & Thompson Coon, J. (2022). Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. BMJ Evidence-Based Medicine. https://doi.org/10.1136/bmjebm-2021-111710
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.