The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index.
An updated protocol for a systematic review of implementation-related measures
Open accessAbstract
BACKGROUND: Implementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript. METHODS: Our primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the "cited by" function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting "poor" to "excellent" evidence. DISCUSSION: We will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Determinant frameworks provide a base set of concepts, terms, and definitions by which to articulate dynamic complex contexts and develop much needed measures of context [10].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Lewis et al.’s measurement reviews on the original CFIR constructs [10] focused specifically on assessing questionnaires administered to healthcare professionals or leaders within behavioral health settings [97] using multi-dimensional criteria for validity and pragmatism [98].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [10]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“With in-kind support, SIRC developed a methodology and published preliminary results of an enhanced systematic review of measures that assessed constructs delineated in the (1) Consolidated Framework for Implementation Research (CFIR [16]) [3] and (2) Implementation Outcomes Framework (IOF; [2, 17]); (see Lewis et al.”
MethodProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 3×
“In this protocol, we will identify whether the measure demonstrates predictive validity for each of the eight IOF outcomes [17] (acceptability, feasibility, appropriateness, adoption, penetration, cost, fidelity, sustainability).”
BackgroundMethodWelcome to Implementation Science
Eccles, M. P., & Mittman, B. S. (2006) · Implementation Science · doi:10.1186/1748-5908-1-1 · Cited by 2,254 · cited 1×
“integrate evidence-based practices into real-world settings [1]—is a rapidly growing, transdisciplinary field with enormous potential to change the way that mental health and health services are delivered in community”
BackgroundTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978 · cited 4×
“A “worst score counts” methodology will be applied for the rating of the packets [30].”
BackgroundMethodInfluentialProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · cited 1×
“With in-kind support, SIRC developed a methodology and published preliminary results of an enhanced systematic review of measures that assessed constructs delineated in the (1) Consolidated Framework for Implementation Research (CFIR [16]) [3] and (2) Implementation Outcomes Framework (IOF; [2, 17]); (see Lewis et al.”
MethodBridging Research and Practice
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.05.024 · Cited by 1,386 · cited 2×
“A second measurement issue stems from the similarities, redundancies, and differences in theories, models, and frameworks [6].”
BackgroundAarons, G. A. (2004) · Mental Health Services Research · doi:10.1023/b:mhsr.0000024351.12294.65 · Cited by 1,272 · cited 4×
“Evidence-Based Practice Scale, EBPAS [25]) will also be sought by entering the full name of the measure in quotation marks into the database search field.”
MethodInfluentialTerwee, C. B., Jansma, E. P., Riphagen, I. I., & de Vet, H. C. W. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9528-5 · Cited by 849 · cited 1×
“, imperfect database indexing, variable terminology, lack of reporting in titles, and abstracts [23]).”
BackgroundChaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800 · cited 3×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundMethodWeiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · cited 2×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundSelf-citation
Research works printed but not linked: 1
- Stanick C, Halko H, Dorsey C, Weiner BJ, Powell B, Palinkas L, Lewis CC. A stakeholder-driven operationalization of the “pragmatic” measures construct. 2017 in review.
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
“Determinant frameworks provide a base set of concepts, terms, and definitions by which to articulate dynamic complex contexts and develop much needed measures of context [10].”
BackgroundGuidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475
Action, actor, context, target, time (AACTT): a framework for specifying behaviour
Presseau, J., McCleary, N., Lorencatto, F., Patey, A. M., Grimshaw, J. M., & Francis, J. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0951-x · Cited by 411
“Given on-going efforts to systematise the measurement of theoretical constructs [54], and description of intervention components [41–43], there is a need for ensuring that the object of change, i.”
BackgroundImplementation science made too simple: a teaching tool
Curran, G. M. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00001-z · Cited by 285
Woodward, E. N., Singh, R. S., Ndebele-Ngwenya, P., Melgar Castillo, A., Dickson, K. S., & Kirchner, J. E. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00146-5 · Cited by 270
Metadata
- Authors
- Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- Virginia Commonwealth University (US)
- University of Washington (US)
- University of Montana (US)
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Lewis, Cara C.
- ORCID
- Venue
- Systematic Reviews
- Published
- 2018-04-25
- Volume
- 7
- Issue
- 1
- Pages
- 66-66
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5918558
- OpenAlex
- W2806234161
- Cited by
- 118
- References
- 36
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 9 · 2023: 26 · 2022: 14 · 2021: 25 · 2020: 18 · 2019: 7 · 2018: 2
- FWCI
- 2.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- implementation science, psychometric properties, reliability and validity, Consolidated Framework for Implementation Research, evidence-based practice, mental health interventions, implementation process
- MeSH terms
- Systematic Reviews as Topic, Health Plan Implementation, Humans, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An updated protocol for a systematic review of implementation-related measures. https://app.cochat.ai/papers/10.1186/s13643-018-0728-3
- The paper
- Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018). An updated protocol for a systematic review of implementation-related measures. Systematic Reviews. https://doi.org/10.1186/s13643-018-0728-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.