The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength
Open accessPublished version of a preprint
Abstract
Introduction: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. Methods: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. Results: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. Discussion: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Where formal theory is well-developed, researchers may be able to translate the parallel trends assumption into domain-specific conditions and use context to help select functional form or scale of the outcome, to which the parallel trends assumption is usually sensitive [17, 32, 66].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [66]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
The effect of large-scale anti-contagion policies on the COVID-19 pandemic
Hsiang, S., Allen, D., Annan-Phan, S., Bell, K., Bolliger, I., Chong, T., Druckenmiller, H., Huang, L. Y., Hultgren, A., Krasovich, E., et al. (2020) · Nature · doi:10.1038/s41586-020-2404-8 · Cited by 1,379
Identifying airborne transmission as the dominant route for the spread of COVID-19
Zhang, R., Li, Y., Zhang, A. L., Wang, Y., & Molina, M. J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2009637117 · Cited by 1,236
Courtemanche, C., Garuccio, J., Le, A., Pinkston, J., & Yelowitz, A. (2020) · Health Affairs · doi:10.1377/hlthaff.2020.00608 · Cited by 773
Lyu, W., & Wehby, G. L. (2020) · Health Affairs · doi:10.1377/hlthaff.2020.00818 · Cited by 721
Islam, N., Sharp, S. J., Chowell, G., Shabnam, S., Kawachi, I., Lacey, B., Massaro, J. M., D’Agostino, R. B., & White, M. (2020) · BMJ · doi:10.1136/bmj.m2743 · Cited by 601
irr: Various Coefficients of Interrater Reliability and Agreement
Matthias Gamer <m.gamer@uke.uni-hamburg.de>, J. L. <. C. A. (2005) · CRAN: Contributed Packages · doi:10.32614/cran.package.irr · Cited by 540
The Efficacy of Lockdown Against COVID-19: A Cross-Country Panel Analysis
Alfano, V., & Ercolano, S. (2020) · Applied Health Economics and Health Policy · doi:10.1007/s40258-020-00596-3 · Cited by 481
Association Between Statewide School Closure and COVID-19 Incidence and Mortality in the US
Auger, K. A., Shah, S. S., Richardson, T., Hartley, D., Hall, M., Warniment, A., Timmons, K., Bosse, D., Ferris, S. A., Brady, P. W., et al. (2020) · JAMA · doi:10.1001/jama.2020.14348 · Cited by 470
Research works printed but not linked: 4
- Chapter 8: Assessing risk of bias in a randomized trial. Available: https://training.cochrane.org/handbook/current/chapter-08 [Accessed 8 Sep 2021].
- Krippendorff KH . Content analysis: an introduction to its methodology. SAGE Publications, 1980.
- About Google Scholar. Available: https://scholar.google.com/intl/en/scholar/about.html [Accessed 15 Jan 2021].
- 10.1136/bmj.m1328 · doi:10.1136/bmj.m1328 · Paper page
Cited by
Policy evaluation during a pandemic
Callaway, B., & Li, T. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.009 · Cited by 38
Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods
Feng, S., Ganguli, I., Lee, Y., Poe, J., Ryan, A., & Bilinski, A. (2025) · Statistics in Medicine · doi:10.1002/sim.70247 · Cited by 16
Self-citation
Metadata
- Authors
- Haber, N. A., Clarke-Deelder, E., Feller, A., Smith, E. R., Salomon, J. A., MacCormack-Gelles, B., Stone, E. M., Bolster-Foucault, C., Daw, J. R., Hatfield, L. A., Fry, C. E., Boyer, C. B., Ben-Michael, E., Joyce, C. M., Linas, B. S., Schmid, I., Au, E. H., Wieten, S. E., Jarrett, B., Axfors, C., Nguyen, V. T., Griffin, B. A., Bilinski, A., & Stuart, E. A.
- Affiliations
- Stanford University (US)
- Department of Global Health and Population, Harvard University T H Chan School of Public Health, Boston, Massachusetts, USA
- University of California, Berkeley (US)
- George Washington University (US)
- Harvard University (US)
- Johns Hopkins University (US)
- Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Québec, Canada
- Columbia University (US)
- Vanderbilt University (US)
- Institute for Quantitative Social Science, Harvard University, Cambridge, MA, USA
- McGill University (CA)
- The University of Sydney (AU)
- RAND Corporation (US)
- Corresponding author
- Haber, Noah A
- ORCID
- Haber, Noah A
- Feller, Avi
- Smith, Emily R
- Salomon, Joshua A.
- Stone, Elizabeth M
- Daw, Jamie R
- Hatfield, Laura Anne
- Fry, Carrie E
- Boyer, Christopher B
- Joyce, Caroline M
- Linas, Beth S
- Schmid, Ian
- Au, Eric H
- Wieten, Sarah E
- Jarrett, Brooke
- Axfors, Cathrine
- Nguyen, Van Thu
- Griffin, Beth Ann
- Bilinski, Alyssa
- Stuart, Elizabeth A
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2022-01
- Volume
- 12
- Issue
- 1
- Pages
- e053820
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC8753111
- OpenAlex
- W4205346926
- Cited by
- 36
- References
- 66
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 6 · 2023: 9 · 2022: 7
- FWCI
- 0.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Durham Research Online (Durham University) (acceptedVersion)
- PubMed Central (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- The Sydney eScholarship Repository (The University of Sydney) (submittedVersion)
- Funders
- Laura and John Arnold Foundation
- Knut och Alice Wallenbergs Stiftelse (2019.0561)
- National Institutes of Health
- National Heart, Lung, and Blood Institute (T32HL098048)
- National Institute of Mental Health (F31MH121128, T32MH122357, T32MH109436)
- National Institute on Drug Abuse (P50DA046351)
- Topics
- Physical Sciences › Mathematics › Modeling and Simulation › COVID-19 epidemiological studies
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- health policy evaluation, research design, causal inference, quality of evidence, COVID-19 outcomes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. https://app.cochat.ai/papers/10.1136/bmjopen-2021-053820
- The paper
- Haber, N. A., Clarke-Deelder, E., Feller, A., Smith, E. R., Salomon, J. A., MacCormack-Gelles, B., Stone, E. M., Bolster-Foucault, C., Daw, J. R., Hatfield, L. A., Fry, C. E., Boyer, C. B., Ben-Michael, E., Joyce, C. M., Linas, B. S., Schmid, I., Au, E. H., Wieten, S. E., Jarrett, B., . . . Stuart, E. A. (2022). Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. BMJ Open. https://doi.org/10.1136/bmjopen-2021-053820
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.