Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.

Quantitative measures of health policy implementation determinants and outcomes: a systematic review

Open access

Abstract

Background: Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures. Methods: Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures. Results: Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported. Conclusions: Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health. Registration: Not registered

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13012-020-01007-w

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 PRISMA flow diagram
Fig. 1 PRISMA flow diagramCC BY · PubMed Central
Fig. 2 Pragmatic rating scale results across identified measures. Footnote: pragmatic criteria scores from Psychometric and Pragmatic Evidence Rating Scale (PAPERS) (Lewis et al. [11], Stanick et al. [42]). Total possible score = 20, total median score across 38 measures = 11. Scores ranged from 0 to 18. Rating scales for each domain are provided in Supplemental Table 4
Fig. 2 Pragmatic rating scale results across identified measures. Footnote: pragmatic criteria scores from Psychometric and Pragmatic Evidence Rating Scale (PAPERS) (Lewis et al. [11], Stanick et al. [42]). Total possible score = 20, total median score across 38 measures = 11. Scores ranged from 0 to 18. Rating scales for each domain are provided in Supplemental Table 4CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 107 references

Research works printed but not linked: 4

  1. Eyler AA, Chriqui JF, Moreland-Russell S, Brownson RC, editors. Prevention, policy, and public health, first edition. New York, NY: Oxford University Press; 2016.
  2. Kingdon J. Agendas, alternatives, and public policies, second edition. Second ed. New York: Longman; 1995.
  3. Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice, second edition. New York: Oxford University Press; 2018.
  4. Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation. https://www.covidence.org. Accessed Mar 2019.

Cited by

See all 164

Metadata

Authors
Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., Baumann, A. A., & Brownson, R. C.
Affiliations
  • Washington University in St. Louis (US)
  • Brigham Young University (US)
  • Kaiser Permanente Washington Health Research Institute (US)
  • Drexel University (US)
Corresponding author
Allen, Peg
Venue
Implementation Science
Published
2020-06-19
Volume
15
Issue
1
Pages
47-47
ISSN
1748-5908 (ISSN-L)
OpenAlex
W3036111784
Cited by
164
References
107
Citations by year
2026: 21 · 2025: 14 · 2024: 32 · 2023: 30 · 2022: 25 · 2021: 36 · 2020: 6
FWCI
4.61
Type
journal article
Language
en
Open access
Licence
Locations
  • Implementation Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Foundation for Barnes-Jewish Hospital (4798, ICTS Research 615735 from Washington University in St. Louis Institute of Clinical and Translational Science Pilot Program, NIH/National Center for Advancing Translational Sciences (NCATS) grant UL1 TR002345)
  • National Cancer Institute (P50CA244431)
  • Centers for Disease Control and Prevention (U48DP006395-01-00)
  • National Institute of Mental Health (R01MH106510)
  • National Institute of Diabetes and Digestive and Kidney Diseases (P30DK020579)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Public Health Policies and Education
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
Keywords
health policy implementation, implementation outcomes, implementation factors, quantitative indicators, measurement quality, psychometric properties, acceptability, feasibility, appropriateness, compliance, organizational culture, implementation climate, Consolidated Framework for Implementation Research, measurement development
MeSH terms
Implementation Science, Attitude of Health Personnel, Health Policy, Humans, Psychometrics, Organizational Culture, Practice Guidelines as Topic, Guideline Adherence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Quantitative measures of health policy implementation determinants and outcomes: a systematic review. https://app.cochat.ai/papers/10.1186/s13012-020-01007-w
The paper
Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., Baumann, A. A., & Brownson, R. C. (2020). Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implementation Science. https://doi.org/10.1186/s13012-020-01007-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.