Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda
Open accessThe DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
An unresolved issue in the field of implementation research is how to conceptualize and evaluate successful implementation. This paper advances the concept of "implementation outcomes" distinct from service system and clinical treatment outcomes. This paper proposes a heuristic, working "taxonomy" of eight conceptually distinct implementation outcomes-acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability-along with their nominal definitions. We propose a two-pronged agenda for research on implementation outcomes. Conceptualizing and measuring implementation outcomes will advance understanding of implementation processes, enhance efficiency in implementation research, and pave the way for studies of the comparative effectiveness of implementation strategies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the failure of most implementation processes to incorporate ongoing, iterative intervention development increases the risk that local users will find new technologies to be low on key implementation outcomes such as acceptability, feasibility, and appropriateness (Proctor et al., 2011) and may limit effectiveness (Chambers et al., 2013)”
From User‐centered design for psychosocial intervention development and implementation. (2016) · cites this paper as Proctor et al., 2011
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Increasing use of implementation frameworks may offer a more systematic evaluation of newborn care packages while clarifying equitable service delivery in vulnerable populations often present in LMIC settings (35, 36).”
From Born too soon in a resource-limited setting: A 10-year mixed methods review of a special care baby unit for refugees and migrants on the Myanmar-Thailand border (2023) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While reporting systems like those by Proctor [1] or Pinnock [75] would optimize learning across studies, few manuscripts specify strategies as recommended [76, 77].”
From A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022 (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“implementation outcomes are influenced by a complex interaction of contextual, organizational, workforce, policy, and system-level factors [12,13,14].”
From A Scoping Review of Implemented Innovations in Cancer Care: Implications for Pan-Canadian Scaling (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These factors align closely with established implementation science frameworks that emphasize the importance of organizational readiness, implementation climate, leadership engagement, and continuous learning systems [11,13,14].”
From A Scoping Review of Implemented Innovations in Cancer Care: Implications for Pan-Canadian Scaling (2026) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
Crossing the Quality Chasm: A New Health System for the 21st Century
Baker, A. (2001) · BMJ · doi:10.1136/bmj.323.7322.1192 · Cited by 10,736
Crossing the Quality Chasm: A New Health System for the 21st Century
Committee on Quality of Health Care in America (2002) · Journal For Healthcare Quality · doi:10.1111/j.1945-1474.2002.tb00463.x · Cited by 8,483
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217 · cited 2×
“The literature reflects quite varied uses of the term ‘‘sustainability,’’ but our proposed definition incorporates aspects of those offered by Johnson et al. (2004), Turner and Sanders (2006), Glasgow et al. (1999) , Goodman et al. (1993), and Rabin et al. (2008).”
Background(2005) · Cited by 3,748
Crossing the Quality Chasm: A New Health System for the 21st Century
Bloom, B. S. (2002) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.287.5.646-a · Cited by 3,726
Making psychological theory useful for implementing evidence based practice: a consensus approach
Michie, S. (2005) · Quality and Safety in Health Care · doi:10.1136/qshc.2004.011155 · Cited by 3,440
Gluesing, J. C. (2012) · Leadership in Science and Technology: A Reference Handbook · doi:10.4135/9781412994231.n14 · Cited by 3,210
Davis, F. D. (1993) · International Journal of Man-Machine Studies · doi:10.1006/imms.1993.1022 · Cited by 2,964 · cited 1×
“Lack of acceptability has long been noted as a challenge in implementation (Davis 1993).”
BackgroundAldenderfer, M. S., & Blashfield, R. K. (1984) · Cited by 2,358
Research works printed but not linked: 6
- 5. Bates, D. M., & Sarkar, D. (2007). lme4: Linear mixed-effects models using S4 classes. R Package Version, 0, 99875-99876.
- 18. Glasgow, R. E. (2007b). The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. Paper presented at the NIH Conference on Building the Science of Dissemination and Implementation in the Service of Public Health: Bethesda, MD.
- 24. Hayes, S. C. (1998). President's message: Dissemination research now. The Behavior Therapist, 21(9), 166-169.
- Institute of Medicine Committee on Crossing the Quality Chasm. (2006). Adaption to mental health and addictive disorder: Improving the quality of health care for mental and substance-use conditions. Washington, DC: Institute of Medicine, National Academies Press.
- 33. Jones, K. (1997). Mission drift in qualitative research, or moving toward a systematic review of qualitative studies. The Qualita- tive Report, 9(1), 95-112.
- 45. Mihalic, S. (2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4, 83-86.
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
“The CFIR Outcomes Addendum provides high-level guidance for identifying implementation outcomes by drawing on other frameworks [81, 82].”
BackgroundMaking sense of implementation theories, models and frameworks
Nilsen, P. (2015) · Implementation Science · doi:10.1186/s13012-015-0242-0 · Cited by 4,447
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240
Self-citationPsychometric assessment of three newly developed implementation outcome measures
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017) · Implementation Science · doi:10.1186/s13012-017-0635-3 · Cited by 2,819
“This study contributes to the literature by developing valid and reliable measures of important implementation outcomes [1].”
BackgroundMethodInfluentialImplementation strategies: recommendations for specifying and reporting
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-139 · Cited by 2,585
“[47,49,50], and we direct readers to the dissemination and implementation section of the Grid Enabled Measurement Initiative and the Seattle Implementation Research Collaborative’s measures project for repositories of implementation outcomes [82,83].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M.
- Affiliations
- Washington University in St. Louis (US)
- Roberts Wesleyan College (US)
- Corresponding author
- Proctor, Enola
- Venue
- Administration and Policy in Mental Health and Mental Health Services Research
- Published
- 2010-10-19
- Volume
- 38
- Issue
- 2
- Pages
- 65-76
- ISSN
- 0894-587X (ISSN-L)
- 1573-3289
- PMC
- PMC3068522
- OpenAlex
- W2140092196
- Semantic Scholar
- 3c7cf782e4094a7d66d8ad5da80a8083a8e72307
- Cited by
- 8,721
- References
- 71
- Citations by year
- 2026: 1487 · 2025: 1144 · 2024: 1303 · 2023: 1041 · 2022: 950 · 2021: 893 · 2020: 669 · 2019: 391 · 2018: 253 · 2017: 162 · 2016: 122 · 2015: 116 · 2014: 83 · 2013: 57 · 2012: 28
- FWCI
- 56.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Administration and Policy in Mental Health and Mental Health Services Research (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (5p30mh068579-05)
- National Institute of Mental Health (P30-MH-068579, T32MH019960)
- National Center for Research Resources (UL1RR024992)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Health Professions › General Health Professions › Mental Health and Patient Involvement
- Keywords
- implementation outcomes, implementation research, acceptability, adoption, appropriateness, feasibility, fidelity, implementation costs, penetration, sustainability, implementation strategies, mental health services
- MeSH terms
- Diffusion of Innovation, Humans, Mental Health Services, Outcome and Process Assessment, Health Care, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. https://app.cochat.ai/papers/10.1007/s10488-010-0319-7
- The paper
- Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-010-0319-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
