Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
The aim of this systematic scoping review was to identify and analyze indicators that address implementation quality or success in health care services and to deduce recommendations for further indicator development. This review was conducted according to the Joanna Briggs Manual and the PRISMA Statement. CINAHL, EMBASE, MEDLINE, and PsycINFO were searched. Studies or reviews published between August 2008 and 2018 that reported monitoring of the quality or the implementation success in health care services by using indicators based on continuous variables and proportion-based, ratio-based, standardized ratio–based, or rate-based variables or indices were included. The records were screened by title and abstract, and the full-text articles were also independently double-screened by 3 reviewers for eligibility. In total, 4376 records were identified that resulted in 10 eligible studies, including 67 implementation indicators. There was heterogeneity regarding the theoretical backgrounds, designs, objectives, settings, and implementation indicators among the publications. None of the indicators addressed the implementation outcomes of appropriateness or sustainability. Service implementation efficiency was identified as an additional outcome. Achieving consensus in framing implementation outcomes and indicators will be a new challenge in health services research. Considering the new debates regarding health care complexity, the further development of indicators based on complementary qualitative and quantitative approaches is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 20×
“Several authors14,36,37 have already applied or discussed the Conceptional Framework for Implementation Research of Proctor et al.5 Throughout the last few years, they have all concluded that there is still a need for the development of an implementation framework including its related measures.14,36,37 The Context an…”
BackgroundMethodInfluentialPRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement
McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.021 · Cited by 5,700
The Challenge of Innovation Implementation
Klein, K. J., & Sorra, J. S. (1996) · The Academy of Management Review · doi:10.2307/259164 · Cited by 2,113 · cited 3×
“Therefore, it is important to differentiate between innovation and implementation effectiveness to measure implementation success.(4) Innovation and implementation effectiveness can be measured in research settings or in a daily routine setting.”
BackgroundDefining and classifying clinical indicators for quality improvement
Mainz, J. (2003) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzg081 · Cited by 1,161
The Quality Implementation Framework: A Synthesis of Critical Steps in the Implementation Process
Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012) · American Journal of Community Psychology · doi:10.1007/s10464-012-9522-x · Cited by 1,102
Pfadenhauer, L. M., Gerhardus, A., Mozygemba, K., Lysdahl, K. B., Booth, A., Hofmann, B., Wahlster, P., Polus, S., Burns, J., Brereton, L., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0552-5 · Cited by 1,016
Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015) · Implementation Science · doi:10.1186/s13012-015-0342-x · Cited by 430
Predicting program start-up using the stages of implementation measure
Saldana, L., Chamberlain, P., Wang, W., & Hendricks Brown, C. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0363-y · Cited by 148 · cited 4×
“However, the outcome adoption is conceivable.28 Saldana et al. aimed to investigate behavior in the early implementation stages regarding the prediction of a successful start of the Multidimensional Treatment Foster Care program and to measure the predictive validity of the first 3 “Stages of Implementation Completion…”
BackgroundMethodInfluentialAdvancing the application, quality and harmonization of implementation science measures
Rabin, B. A., Purcell, P., Naveed, S., Moser, R. P., Henton, M. D., Proctor, E. K., Brownson, R. C., & Glasgow, R. E. (2012) · Implementation Science · doi:10.1186/1748-5908-7-119 · Cited by 106
Cited by
A pragmatic method for costing implementation strategies using time-driven activity-based costing
Cidav, Z., Mandell, D., Pyne, J., Beidas, R., Curran, G., & Marcus, S. (2020) · Implementation Science · doi:10.1186/s13012-020-00993-1 · Cited by 174
“In particular, despite growing evidence that EBPs can be cost-effective over time, start-up costs are frequently cited as a major barrier to implementation [2, 3].”
BackgroundLewis, C. C., Mettert, K., & Lyon, A. R. (2021) · Implementation Research and Practice · doi:10.1177/2633489521994197 · Cited by 38
“Of the 15 systemic reviews of implementation measures (J. D. Allen et al., 2017; Miake-Lye et al., 2020; Rabin et al., 2016; Willmeroth et al., 2019), only two explored measures of intervention/innovation characteristics (Chaudoir et al., identified measures of intervention characteristics, but…”
BackgroundChen, J., Cutrona, S. L., Dharod, A., Bunch, S. C., Foley, K. L., Ostasiewski, B., Hale, E. R., Bridges, A., Moses, A., Donny, E. C., et al. (2023) · JMIR Medical Informatics · doi:10.2196/43097 · Cited by 4
Chen, J., Cutrona, S. L., Dharod, A., Moses, A., Bridges, A., Ostasiewski, B., Foley, K. L., Houston, T. K., iDAPT (Implementation & Informatics Developing Adaptable Processes and Technologies for Cancer Control) Implementation Science Center for Cancer Control, Chen, J., et al. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf050 · Cited by 5
Balayah, Z., Stavropoulou, C., Scarbrough, H., Nigam, A., & Ziemann, A. (2025) · Frontiers in Health Services · doi:10.3389/frhs.2025.1373429 · Cited by 2
Metadata
- Authors
- Willmeroth, T., Wesselborg, B., & Kuske, S.
- Venue
- INQUIRY: The Journal of Health Care Organization, Provision, and Financing
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 56
- Pages
- 46958019861257-46958019861257
- ISSN
- 0046-9580 (ISSN-L)
- 1945-7243
- OpenAlex
- W2966065423
- Cited by
- 53
- References
- 11
- FWCI
- 4.48
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review. https://app.cochat.ai/papers/10.1177/0046958019861257
- The paper
- Willmeroth, T., Wesselborg, B., & Kuske, S. (2019). Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review. INQUIRY: The Journal of Health Care Organization, Provision, and Financing. https://doi.org/10.1177/0046958019861257
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.