CoChat

Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study

Open access
Abbott, Rebecca, Bethel, Alison, Rogers, Morwenna, Whear, Rebecca, Orr, Noreen, Shaw, Liz, Stein, Ken, Thompson Coon, Jo · 2022 · BMJ Evidence-Based Medicine

The DOI matches its Crossref record; 44 of the 47 references it cites were found in a registry or an index.

Abstract

Objective: The academic and scientific community has reacted at pace to gather evidence to help and inform about COVID-19. Concerns have been raised about the quality of this evidence. The aim of this review was to map the nature, scope and quality of evidence syntheses on COVID-19 and to explore the relationship between review quality and the extent of researcher, policy and media interest. Design and setting: A meta-research: systematic review of reviews. Information sources: PubMed, Epistemonikos COVID-19 evidence, the Cochrane Library of Systematic Reviews, the Cochrane COVID-19 Study Register, EMBASE, CINAHL, Web of Science Core Collection and the WHO COVID-19 database, searched between 10 June 2020 and 15 June 2020. Eligibility criteria: Any peer-reviewed article reported as a systematic review, rapid review, overview, meta-analysis or qualitative evidence synthesis in the title or abstract addressing a research question relating to COVID-19. Articles described as meta-analyses but not undertaken as part of a systematic or rapid review were excluded. Study selection and data extraction: Abstract and full text screening were undertaken by two independent reviewers. Descriptive information on review type, purpose, population, size, citation and attention metrics were extracted along with whether the review met the definition of a systematic review according to six key methodological criteria. For those meeting all criteria, additional data on methods and publication metrics were extracted. Risk of bias: For articles meeting all six criteria required to meet the definition of a systematic review, AMSTAR-2 ((A MeaSurement Tool to Assess systematic Reviews, version 2.0) was used to assess the quality of the reported methods. Results: 2334 articles were screened, resulting in 280 reviews being included: 232 systematic reviews, 46 rapid reviews and 2 overviews. Less than half reported undertaking critical appraisal and a third had no reproducible search strategy. There was considerable overlap in topics, with discordant findings. Eighty-eight of the 280 reviews met all six systematic review criteria. Of these, just 3 were rated as of moderate or high quality on AMSTAR-2, with the majority having critical flaws: only a third reported registering a protocol, and less than one in five searched named COVID-19 databases. Review conduct and publication were rapid, with 52 of the 88 systematic reviews reported as being conducted within 3 weeks, and a half published within 3 weeks of submission. Researcher and media interest, as measured by altmetrics and citations, was high, and was not correlated with quality. Discussion: This meta-research of early published COVID-19 evidence syntheses found low-quality reviews being published at pace, often with short publication turnarounds. Despite being of low quality and many lacking robust methods, the reviews received substantial attention across both academic and public platforms, and the attention was not related to the quality of review methods. Interpretation: Flaws in systematic review methods limit the validity of a review and the generalisability of its findings. Yet, by being reported as ‘systematic reviews’, many readers may well regard them as high-quality evidence, irrespective of the actual methods undertaken. The challenge especially in times such as this pandemic is to provide indications of trustworthiness in evidence that is available in ‘real time’. PROSPERO registration number: CRD42020188822.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjebm-2021-111710

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram.
Figure 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram.CC BY-NC · PubMed Central
Figure 2 Geographical distribution of the production of COVID-19 systematic reviews.
Figure 2 Geographical distribution of the production of COVID-19 systematic reviews.CC BY-NC · PubMed Central
Figure 3 Relationship between AMSTAR-2 category and Altmetric Attention Score, number of Web of Science citations and number of Google Scholar citations.
Figure 3 Relationship between AMSTAR-2 category and Altmetric Attention Score, number of Web of Science citations and number of Google Scholar citations.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Cited by

See all 63

Metadata

Authors
Abbott, R., Bethel, A., Rogers, M., Whear, R., Orr, N., Shaw, L., Stein, K., & Thompson Coon, J.
Affiliations
University of Exeter (GB)
Corresponding author
Abbott, Rebecca
Venue
BMJ Evidence-Based Medicine
Publisher
BMJ
Published
2021-06-03
Volume
27
Issue
3
Pages
169-177
ISSN
  • 1356-5524 (ISSN-L)
  • 2515-446X
  • 2515-4478
OpenAlex
W3164392163
Cited by
56
References
47
Citations by year
2026: 3 · 2025: 7 · 2024: 12 · 2023: 13 · 2022: 16 · 2021: 5
FWCI
0.48
Type
journal article
Language
en
Open access
Locations
  • BMJ Evidence-Based Medicine (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
National Institute for Health Research Applied Research Collaboration South West Peninsula (Not applicable)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
COVID-19 pandemic, evidence synthesis, review quality, infodemic, AMSTAR 2, critical appraisal, rapid review, media coverage of research, publication bias, research ethics committees, research governance
MeSH terms
Systematic Reviews as Topic, COVID-19, Infodemic, Humans, Pandemics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. https://app.cochat.ai/papers/10.1136/bmjebm-2021-111710
The paper
Abbott, R., Bethel, A., Rogers, M., Whear, R., Orr, N., Shaw, L., Stein, K., & Thompson Coon, J. (2022). Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. BMJ Evidence-Based Medicine. https://doi.org/10.1136/bmjebm-2021-111710

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.