The DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Quantitative measures of health policy implementation determinants and outcomes: a systematic review
Open accessAbstract
Background: Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures. Methods: Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures. Results: Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported. Conclusions: Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health. Registration: Not registered
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2 Pragmatic rating scale results across identified measures. Footnote: pragmatic criteria scores from Psychometric and Pragmatic Evidence Rating Scale (PAPERS) (Lewis et al. [11], Stanick et al. [42]). Total possible score = 20, total median score across 38 measures = 11. Scores ranged from 0 to 18. Rating scales for each domain are provided in Supplemental Table 4](/paper-figures/4d5e6eb336944e58a78e8b78df5236e0/quantitative-measures-of-health-policy-implementation-determinants-and-outcomes-figure-2.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We now have systematic reviews of implementation outcomes [15] and their measurement properties in behavioral health [16], public policy [17], stroke care [7], and physical healthcare [18].”
From Ten years of implementation outcomes research: a scoping review (2023) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“In a systematic review of quantitative measures assessing health policy implementation determinants and outcomes, Allen and colleagues found that acceptability, feasibility, appropriateness, and compliance were most frequently measured [17].”
From Ten years of implementation outcomes research: a scoping review (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2008) · Cited by 5,345
Theories of the Policy Process
Sabatier, P. (2019) · Routledge eBooks · doi:10.4324/9780367274689 · Cited by 3,490
Internet, mail, and mixed-mode surveys: The tailored design method *
Wetzel, A. P. (2010) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.20083 · Cited by 2,556
Perceived organizational support, discretionary treatment, and job satisfaction.
Eisenberger, R., Cummings, J., Armeli, S., & Lynch, P. (1997) · Journal of Applied Psychology · doi:10.1037//0021-9010.82.5.812 · Cited by 1,819
Measuring consumer innovativeness
Goldsmith, R. E., & Hofacker, C. F. (1991) · Journal of the Academy of Marketing Science · doi:10.1007/bf02726497 · Cited by 1,457
Bridging Research and Practice
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.05.024 · Cited by 1,386
Self-citationPredictors and outcomes of openness to changes in a reorganizing workplace.
Wanberg, C. R., & Banas, J. T. (2000) · Journal of Applied Psychology · doi:10.1037//0021-9010.85.1.132 · Cited by 1,376
Vaccination greatly reduces disease, disability, death and inequity worldwide
Andre, F., Booy, R., Bock, H., Clemens, J., Datta, S., John, T., Lee, B., Lolekha, S., Peltola, H., Ruff, T., et al. (2008) · Bulletin of the World Health Organization · doi:10.2471/blt.07.040089 · Cited by 1,234
Research works printed but not linked: 4
- Eyler AA, Chriqui JF, Moreland-Russell S, Brownson RC, editors. Prevention, policy, and public health, first edition. New York, NY: Oxford University Press; 2016.
- Kingdon J. Agendas, alternatives, and public policies, second edition. Second ed. New York: Longman; 1995.
- Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice, second edition. New York: Oxford University Press; 2018.
- Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation. https://www.covidence.org. Accessed Mar 2019.
Cited by
SCOPE 2021: a new scorecard for osteoporosis in Europe
Kanis, J. A., Norton, N., Harvey, N. C., Jacobson, T., Johansson, H., Lorentzon, M., McCloskey, E. V., Willers, C., & Borgström, F. (2021) · Archives of Osteoporosis · doi:10.1007/s11657-020-00871-9 · Cited by 714
Ten years of implementation outcomes research: a scoping review
Proctor, E. K., Bunger, A. C., Lengnick-Hall, R., Gerke, D. R., Martin, J. K., Phillips, R. J., & Swanson, J. C. (2023) · Implementation Science · doi:10.1186/s13012-023-01286-z · Cited by 424
“We now have systematic reviews of implementation outcomes [15] and their measurement properties in behavioral health [16], public policy [17], stroke care [7], and physical healthcare [18].”
BackgroundOsteoporosis in Europe: a compendium of country-specific reports
Willers, C., Norton, N., Harvey, N. C., Jacobson, T., Johansson, H., Lorentzon, M., McCloskey, E. V., Borgström, F., Kanis, J. A., & IOF, T. S. R. P. O. T. (2022) · Archives of Osteoporosis · doi:10.1007/s11657-021-00969-8 · Cited by 272
Woodward, E. N., Singh, R. S., Ndebele-Ngwenya, P., Melgar Castillo, A., Dickson, K. S., & Kirchner, J. E. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00146-5 · Cited by 270
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81
Self-citation
Metadata
- Authors
- Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., Baumann, A. A., & Brownson, R. C.
- Affiliations
- Washington University in St. Louis (US)
- Brigham Young University (US)
- Kaiser Permanente Washington Health Research Institute (US)
- Drexel University (US)
- Corresponding author
- Allen, Peg
- ORCID
- Venue
- Implementation Science
- Published
- 2020-06-19
- Volume
- 15
- Issue
- 1
- Pages
- 47-47
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC7304175
- OpenAlex
- W3036111784
- Semantic Scholar
- 28e372e9ba44bdc9c4169ccefc095ff3a46a9820
- Cited by
- 164
- References
- 107
- Citations by year
- 2026: 21 · 2025: 14 · 2024: 32 · 2023: 30 · 2022: 25 · 2021: 36 · 2020: 6
- FWCI
- 4.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Foundation for Barnes-Jewish Hospital (4798, ICTS Research 615735 from Washington University in St. Louis Institute of Clinical and Translational Science Pilot Program, NIH/National Center for Advancing Translational Sciences (NCATS) grant UL1 TR002345)
- National Cancer Institute (P50CA244431)
- Centers for Disease Control and Prevention (U48DP006395-01-00)
- National Institute of Mental Health (R01MH106510)
- National Institute of Diabetes and Digestive and Kidney Diseases (P30DK020579)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Public Health Policies and Education
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- health policy implementation, implementation outcomes, implementation factors, quantitative indicators, measurement quality, psychometric properties, acceptability, feasibility, appropriateness, compliance, organizational culture, implementation climate, Consolidated Framework for Implementation Research, measurement development
- MeSH terms
- Implementation Science, Attitude of Health Personnel, Health Policy, Humans, Psychometrics, Organizational Culture, Practice Guidelines as Topic, Guideline Adherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Quantitative measures of health policy implementation determinants and outcomes: a systematic review. https://app.cochat.ai/papers/10.1186/s13012-020-01007-w
- The paper
- Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., Baumann, A. A., & Brownson, R. C. (2020). Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implementation Science. https://doi.org/10.1186/s13012-020-01007-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.