The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria
Open accessAbstract
BACKGROUND: High-quality measurement is critical to advancing knowledge in any field. New fields, such as implementation science, are often beset with measurement gaps and poor quality instruments, a weakness that can be more easily addressed in light of systematic review findings. Although several reviews of quantitative instruments used in implementation science have been published, no studies have focused on instruments that measure implementation outcomes. Proctor and colleagues established a core set of implementation outcomes including: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, sustainability (Adm Policy Ment Health Ment Health Serv Res 36:24-34, 2009). The Society for Implementation Research Collaboration (SIRC) Instrument Review Project employed an enhanced systematic review methodology (Implement Sci 2: 2015) to identify quantitative instruments of implementation outcomes relevant to mental or behavioral health settings. METHODS: Full details of the enhanced systematic review methodology are available (Implement Sci 2: 2015). To increase the feasibility of the review, and consistent with the scope of SIRC, only instruments that were applicable to mental or behavioral health were included. The review, synthesis, and evaluation included the following: (1) a search protocol for the literature review of constructs; (2) the literature review of instruments using Web of Science and PsycINFO; and (3) data extraction and instrument quality ratings to inform knowledge synthesis. Our evidence-based assessment rating criteria quantified fundamental psychometric properties as well as a crude measure of usability. Two independent raters applied the evidence-based assessment rating criteria to each instrument to generate a quality profile. RESULTS: We identified 104 instruments across eight constructs, with nearly half (n = 50) assessing acceptability and 19 identified for adoption, with all other implementation outcomes revealing fewer than 10 instruments. Only one instrument demonstrated at least minimal evidence for psychometric strength on all six of the evidence-based assessment criteria. The majority of instruments had no information regarding responsiveness or predictive validity. CONCLUSIONS: Implementation outcomes instrumentation is underdeveloped with respect to both the sheer number of available instruments and the psychometric quality of existing instruments. Until psychometric strength is established, the field will struggle to identify which implementation strategies work best, for which organizations, and under what conditions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, the lack of valid and pragmatic assessments for implementation-related factors and outcomes has been noted in the health services and implementation science literature [110, 111] and has been echoed by researchers across health-related interventions [112].”
From Barriers and facilitators to implementation of evidence-based task-sharing mental health interventions in low- and middle-income countries: a systematic review using implementation science frameworks (2022) · cites this paper as [111]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“few if any have been assessed for other measurement properties that are of key importance to implementation researchers and other implementation stakeholders (e.g., intermediaries, policymakers, practice leaders), such as stability, responsiveness, and predictive validity [2–4].”
From Psychometric assessment of three newly developed implementation outcome measures (2017) · cites this paper as [2]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We now have systematic reviews of implementation outcomes [15] and their measurement properties in behavioral health [16], public policy [17], stroke care [7], and physical healthcare [18].”
From Ten years of implementation outcomes research: a scoping review (2023) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“This assessment responds to the need for reliable measures within implementation science, specifically in the area of sustainability research (1, 32, 33).”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [33]
The cited source says what the claim attributes to it.
In the citing paper
“Despite their importance, existing measures of implementation outcomes have largely unknown psychometric and pragmatic qualities [2].”
From Psychometric assessment of three newly developed implementation outcome measures (2017) · cites this paper as [2]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quality criteria were proposed for measurement properties of health status questionnaires
Terwee, C. B., Bot, S. D. M., de Boer, M. R., van der Windt, D. A. W. M., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. W. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.012 · Cited by 10,764 · cited 1×
“The EBA rating criteria were also informed by the work of Terwee and colleagues [10].”
Moore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150 · cited 2×
“The Adoption of Information Technology Innovation [16] scale had the highest overall rating (score = 14), with “ 4 — excellent ” ratings for structural validity and norms and “ 3 — good ” ratings for reliability and usability; however, no information about predictive validity is available.”
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 10×
“Appropriateness (N = 7) is the perceived fit, relevance, or compatibility of the innovation or evidence based practice for a given practice setting, provider, or consumer; and/or perceived fit of the innovation to address a particular issue or problem [6].”
InfluentialTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978 · cited 1×
“Raters used the conservative “worst score counts” methodology [12].”
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · cited 1×
“Proctor and colleagues [5] articulated the following core set of seven implementation outcomes: acceptability, feasibility, uptake, penetration, cost, fidelity, and sustainability.”
Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800 · cited 1×
“4 % [3] lacked criterion validity and 49.”
Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · cited 1×
“science have been published [2–4], none have focused on implementation outcomes instruments.”
BackgroundSelf-citationGrimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., & Vale, L. (2006) · Journal of General Internal Medicine · doi:10.1007/s11606-006-0269-7 · Cited by 631 · cited 1×
“Establishing the psychometric properties of implementation outcome instruments is a necessary step to ensuring that predictors, moderators, and mediators of implementation are identified and that comparative effectiveness of implementation strategies is measureable [8].”
Hunsley, J., & Mash, E. J. (2007) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.3.022806.091419 · Cited by 528
A national survey of practicing psychologists' attitudes toward psychotherapy treatment manuals
Addis, M. E., & Krasnow, A. D. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.68.2.331 · Cited by 509 · cited 2×
“The Practitioner’s Attitudes toward Treatment Manuals [15] had the highest overall rating among implementation process acceptability instruments (total score = 17), with “4—excellent” ratings for reliability, structural validity, and norms; “3—good” rating for usability; and “2—adequate” rating for predictive validity.”
Research works printed but not linked: 1
- Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, et al. Toward evidence-based quality improvement. J Gen Intern Med. 2006;21:S14–20.
Cited by
Psychometric assessment of three newly developed implementation outcome measures
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017) · Implementation Science · doi:10.1186/s13012-017-0635-3 · Cited by 2,819
“Despite their importance, existing measures of implementation outcomes have largely unknown psychometric and pragmatic qualities [2].”
BackgroundSelf-citationLewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., Walsh-Bailey, C., & Weiner, B. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00136 · Cited by 588
“Many of these outcomes are appropriately construed as latent variables, but others are manifest/observable in nature (30); a recent systematic review offers measures of these outcomes and measure meta-data (31).”
BackgroundSelf-citationChambers, D. A., & Norton, W. E. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.011 · Cited by 581
“Indeed, one of the key outcomes of implementation is that of fidelity, the degree to which an intervention is delivered as designed.(14,15) Fidelity measures are developed for the purpose of monitoring and maintaining integrity to the intervention, often focusing predominantly on the “core components” or “active ingre…”
BackgroundWoodward, E. N., Matthieu, M. M., Uchendu, U. S., Rogal, S., & Kirchner, J. E. (2019) · Implementation Science · doi:10.1186/s13012-019-0861-y · Cited by 470
The Handbook of Behavior Change
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318 · Cited by 407
Metadata
- Authors
- Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G.
- Affiliations
- University of Washington (US)
- Indiana University Bloomington (US)
- University of North Carolina at Chapel Hill (US)
- University of Montana (US)
- Virginia Commonwealth University (US)
- Corresponding author
- Lewis, Cara C.
- Venue
- Implementation Science
- Published
- 2015-11-04
- Volume
- 10
- Issue
- 1
- Pages
- 155-155
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC4634818
- OpenAlex
- W2120941905
- Semantic Scholar
- edba1483b061fe1b9a35dd81f979980308ce3525
- Cited by
- 430
- References
- 26
- Citations by year
- 2026: 24 · 2025: 24 · 2024: 33 · 2023: 58 · 2022: 58 · 2021: 51 · 2020: 100 · 2019: 36 · 2018: 25 · 2017: 15 · 2016: 5 · 2015: 1
- FWCI
- 2.76
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- VCU Scholars Compass (Virginia Commonwealth University) (submittedVersion, cc-by)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- Georgia Clinical and Translational Science Alliance
- National Institutes of Health (2010-2015, UL1TR000083)
- National Institute of Mental Health (R01MH106510)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- implementation outcomes, psychometric properties, mental health services, behavioral health services, acceptability, adoption, fidelity, feasibility, cost, penetration, sustainability, measurement gaps
- MeSH terms
- Diffusion of Innovation, Humans, Mental Health Services, Psychometrics, Reproducibility of Results, Program Evaluation, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. https://app.cochat.ai/papers/10.1186/s13012-015-0342-x
- The paper
- Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015). Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implementation Science. https://doi.org/10.1186/s13012-015-0342-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.