Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength

Open access

Published version of a preprint

Abstract

Introduction: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. Methods: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. Results: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. Discussion: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2021-053820

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 PRISMA diagram of systematic review process. This chart shows the PRISMA diagram for the process of screening the literature from search to the full review phase. PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Figure 1 PRISMA diagram of systematic review process. This chart shows the PRISMA diagram for the process of screening the literature from search to the full review phase. PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-AnalysesCC BY-NC · PubMed Central
Figure 2 Descriptive sample statistics (n=36). This chart shows descriptive statistics of the 36 studies entered into our systematic evidence review.
Figure 2 Descriptive sample statistics (n=36). This chart shows descriptive statistics of the 36 studies entered into our systematic evidence review.CC BY-NC · PubMed Central
Figure 3 Main consensus results summary for key and overall questions. This chart shows the final overall ratings (left) and the key design question ratings for the consensus review of the 36 included studies, answering the degree to which the articles met the given key design question criteria. The key design question ratings were not asked for the nine included articles which selected methods assumed by the guidance to be non-appropriate. The question prompt in the figure is shortened for clarity, where the full prompt for each key question is available in the Methods section.
Figure 3 Main consensus results summary for key and overall questions. This chart shows the final overall ratings (left) and the key design question ratings for the consensus review of the 36 included studies, answering the degree to which the articles met the given key design question criteria. The key design question ratings were not asked for the nine included articles which selected methods assumed by the guidance to be non-appropriate. The question prompt in the figure is shortened for clarity, where the full prompt for each key question is available in the Methods section.CC BY-NC · PubMed Central
Figure 4 Comparison of independent reviews, weakest link and direct consensus review. This chart shows the final overall ratings by three different possible metrics. The first column contains all of the independent review ratings for the 27 studies which were eventually included in our sample, noting that reviewers who either selected them as not meeting inclusion criteria or selected a method that did not receive the full review did not contribute. The middle column contains the final consensus reviews among the 27 articles which received full review. The last column contains the weakest link rating, as described in the Methods section. The question prompt in the figure is shortened for clarity, where the full prompt for each key question is available in the Methods section.
Figure 4 Comparison of independent reviews, weakest link and direct consensus review. This chart shows the final overall ratings by three different possible metrics. The first column contains all of the independent review ratings for the 27 studies which were eventually included in our sample, noting that reviewers who either selected them as not meeting inclusion criteria or selected a method that did not receive the full review did not contribute. The middle column contains the final consensus reviews among the 27 articles which received full review. The last column contains the weakest link rating, as described in the Methods section. The question prompt in the figure is shortened for clarity, where the full prompt for each key question is available in the Methods section.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 66 references

Research works printed but not linked: 4

  1. Chapter 8: Assessing risk of bias in a randomized trial. Available: https://training.cochrane.org/handbook/current/chapter-08 [Accessed 8 Sep 2021].
  2. Krippendorff KH . Content analysis: an introduction to its methodology. SAGE Publications, 1980.
  3. About Google Scholar. Available: https://scholar.google.com/intl/en/scholar/about.html [Accessed 15 Jan 2021].
  4. 10.1136/bmj.m1328 · doi:10.1136/bmj.m1328 · Paper page

Cited by

See all 36

Metadata

Authors
Haber, N. A., Clarke-Deelder, E., Feller, A., Smith, E. R., Salomon, J. A., MacCormack-Gelles, B., Stone, E. M., Bolster-Foucault, C., Daw, J. R., Hatfield, L. A., Fry, C. E., Boyer, C. B., Ben-Michael, E., Joyce, C. M., Linas, B. S., Schmid, I., Au, E. H., Wieten, S. E., Jarrett, B., Axfors, C., Nguyen, V. T., Griffin, B. A., Bilinski, A., & Stuart, E. A.
Affiliations
  • Stanford University (US)
  • Department of Global Health and Population, Harvard University T H Chan School of Public Health, Boston, Massachusetts, USA
  • University of California, Berkeley (US)
  • George Washington University (US)
  • Harvard University (US)
  • Johns Hopkins University (US)
  • Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Québec, Canada
  • Columbia University (US)
  • Vanderbilt University (US)
  • Institute for Quantitative Social Science, Harvard University, Cambridge, MA, USA
  • McGill University (CA)
  • The University of Sydney (AU)
  • RAND Corporation (US)
Corresponding author
Haber, Noah A
Venue
BMJ Open
Publisher
BMJ
Published
2022-01
Volume
12
Issue
1
Pages
e053820
ISSN
2044-6055 (ISSN-L)
OpenAlex
W4205346926
Cited by
36
References
66
Citations by year
2026: 1 · 2025: 4 · 2024: 6 · 2023: 9 · 2022: 7
FWCI
0.69
Type
journal article
Language
en
Open access
Locations
  • BMJ Open (publishedVersion, cc-by-nc)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Durham Research Online (Durham University) (acceptedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • The Sydney eScholarship Repository (The University of Sydney) (submittedVersion)
Funders
  • Laura and John Arnold Foundation
  • Knut och Alice Wallenbergs Stiftelse (2019.0561)
  • National Institutes of Health
  • National Heart, Lung, and Blood Institute (T32HL098048)
  • National Institute of Mental Health (F31MH121128, T32MH122357, T32MH109436)
  • National Institute on Drug Abuse (P50DA046351)
Topics
  • Physical Sciences › Mathematics › Modeling and Simulation › COVID-19 epidemiological studies
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
Keywords
health policy evaluation, research design, causal inference, quality of evidence, COVID-19 outcomes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. https://app.cochat.ai/papers/10.1136/bmjopen-2021-053820
The paper
Haber, N. A., Clarke-Deelder, E., Feller, A., Smith, E. R., Salomon, J. A., MacCormack-Gelles, B., Stone, E. M., Bolster-Foucault, C., Daw, J. R., Hatfield, L. A., Fry, C. E., Boyer, C. B., Ben-Michael, E., Joyce, C. M., Linas, B. S., Schmid, I., Au, E. H., Wieten, S. E., Jarrett, B., . . . Stuart, E. A. (2022). Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. BMJ Open. https://doi.org/10.1136/bmjopen-2021-053820

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.