An updated protocol for a systematic review of implementation-related measures
Open accessThe DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Implementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript. METHODS: Our primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the "cited by" function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting "poor" to "excellent" evidence. DISCUSSION: We will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Determinant frameworks provide a base set of concepts, terms, and definitions by which to articulate dynamic complex contexts and develop much needed measures of context [10].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Lewis et al.’s measurement reviews on the original CFIR constructs [10] focused specifically on assessing questionnaires administered to healthcare professionals or leaders within behavioral health settings [97] using multi-dimensional criteria for validity and pragmatism [98].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [10]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“With in-kind support, SIRC developed a methodology and published preliminary results of an enhanced systematic review of measures that assessed constructs delineated in the (1) Consolidated Framework for Implementation Research (CFIR [16]) [3] and (2) Implementation Outcomes Framework (IOF; [2, 17]); (see Lewis et al.”
MethodProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 3×
“In this protocol, we will identify whether the measure demonstrates predictive validity for each of the eight IOF outcomes [17] (acceptability, feasibility, appropriateness, adoption, penetration, cost, fidelity, sustainability).”
BackgroundMethodWelcome to Implementation Science
Eccles, M. P., & Mittman, B. S. (2006) · Implementation Science · doi:10.1186/1748-5908-1-1 · Cited by 2,254 · cited 1×
“integrate evidence-based practices into real-world settings [1]—is a rapidly growing, transdisciplinary field with enormous potential to change the way that mental health and health services are delivered in community”
BackgroundTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978 · cited 4×
“A “worst score counts” methodology will be applied for the rating of the packets [30].”
BackgroundMethodInfluentialProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · cited 1×
“With in-kind support, SIRC developed a methodology and published preliminary results of an enhanced systematic review of measures that assessed constructs delineated in the (1) Consolidated Framework for Implementation Research (CFIR [16]) [3] and (2) Implementation Outcomes Framework (IOF; [2, 17]); (see Lewis et al.”
MethodBridging Research and Practice
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.05.024 · Cited by 1,386 · cited 2×
“A second measurement issue stems from the similarities, redundancies, and differences in theories, models, and frameworks [6].”
BackgroundAarons, G. A. (2004) · Mental Health Services Research · doi:10.1023/b:mhsr.0000024351.12294.65 · Cited by 1,272 · cited 4×
“Evidence-Based Practice Scale, EBPAS [25]) will also be sought by entering the full name of the measure in quotation marks into the database search field.”
MethodInfluentialTerwee, C. B., Jansma, E. P., Riphagen, I. I., & de Vet, H. C. W. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9528-5 · Cited by 849 · cited 1×
“, imperfect database indexing, variable terminology, lack of reporting in titles, and abstracts [23]).”
BackgroundChaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800 · cited 3×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundMethodWeiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · cited 2×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundSelf-citationHandbook of research methods in clinical psychology
(2003) · Choice Reviews Online · doi:10.5860/choice.41-2486 · Cited by 690
A Glossary for Dissemination and Implementation Research in Health
Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008) · Journal of Public Health Management and Practice · doi:10.1097/01.phh.0000311888.06252.bb · Cited by 676 · cited 1×
“Second, to fit our scope, measures (and associated articles) will need to demonstrate relevance to implementation, which is defined as the process of integrating evidence-based practices into a community setting [24].”
BackgroundSocial Science Methodology: A Criterial Framework
Gerring, J. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815492 · Cited by 384
Social science methodology: a criterial framework
(2002) · Choice Reviews Online · doi:10.5860/choice.39-6491 · Cited by 336
Instrumentation issues in implementation science
Martinez, R. G., Lewis, C. C., & Weiner, B. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0118-8 · Cited by 243 · cited 3×
“, latent versus objective or manifest) for the target level of analysis [7], which may lead to an overreliance on and inappropriate use of self-report measures [4].”
BackgroundSelf-citationWalsh, W. B., & Betz, N. E. (1985) · Medical Entomology and Zoology · Cited by 214
Are there valid proxy measures of clinical behaviour? a systematic review
Hrisos, S., Eccles, M. P., Francis, J. J., Dickinson, H. O., Kaner, E. F., Beyer, F., & Johnston, M. (2009) · Implementation Science · doi:10.1186/1748-5908-4-37 · Cited by 162 · cited 1×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundLewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2016) · Implementation Science · doi:10.1186/s13012-014-0193-x · Cited by 155 · cited 5×
“Although development of the pragmatic construct and its associated rating criteria is currently underway [20, 21], we have made substantial revisions to the SIRC-established systematic review methodology and the approach to psychometric evaluation [3].”
BackgroundMethodInfluentialSelf-citationDavidson, M. (2014) · Springer eBooks · doi:10.1007/978-94-007-0753-5_1581 · Cited by 154
Powell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C. (2017) · Implementation Science · doi:10.1186/s13012-017-0649-x · Cited by 144 · cited 1×
“Although development of the pragmatic construct and its associated rating criteria is currently underway [20, 21], we have made substantial revisions to the SIRC-established systematic review methodology and the approach to psychometric evaluation [3].”
MethodSelf-citationAchenbach, T. M. (2005) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3403_9 · Cited by 116 · cited 1×
“Of greatest concern is that without reliable and valid measurement scientific findings are limited in their interpretability, comparability, and generalizability [4, 8].”
BackgroundAdvancing implementation science through measure development and evaluation: a study protocol
Lewis, C. C., Weiner, B. J., Stanick, C., & Fischer, S. M. (2015) · Implementation Science · doi:10.1186/s13012-015-0287-0 · Cited by 115 · cited 1×
“This preliminary research informed a series of measures- and methods-focused studies that were funded by the National Institute of Mental Health with the long-term objective of making available a comprehensive battery of reliable, valid, and pragmatic measures that researchers and stakeholders could use to advance imp…”
MethodSelf-citationLefebvre, C., Eisinga, A., McDonald, S., & Paul, N. (2008) · Emerging Themes in Epidemiology · doi:10.1186/1742-7622-5-13 · Cited by 93 · cited 1×
“To identify CFIR- and IOF-related measures, we will access two widely used bibliographic databases: (1) PubMed will allow for automatic inclusion of synonyms and Medical Subject Headings, and (2) Embase will maximize our ability to capture international content, spanning biomedical journals from 90 countries [22].”
MethodMeasurement resources for dissemination and implementation research in health
Rabin, B. A., Lewis, C. C., Norton, W. E., Neta, G., Chambers, D., Tobin, J. N., Brownson, R. C., & Glasgow, R. E. (2015) · Implementation Science · doi:10.1186/s13012-016-0401-y · Cited by 92 · cited 1×
“[9] for a summary of these studies and related resources.”
BackgroundSelf-citationMeasurement Issues in Dissemination and Implementation Research
Proctor, E. K., & Brownson, R. C. (2012) · Dissemination and Implementation Research in HealthTranslating Science to Practice · doi:10.1093/acprof:oso/9780199751877.003.0013 · Cited by 87
Measures for Predictors of Innovation Adoption
Chor, K. H. B., Wisdom, J. P., Olin, S.-C. S., Hoagwood, K. E., & Horwitz, S. M. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0551-7 · Cited by 85 · cited 2×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundSystems Antecedents for Dissemination and Implementation: A Review and Analysis of Measures
Emmons, K. M., Weiner, B., Fernandez, M. E., & Tu, S.-P. (2012) · Health Education & Behavior · doi:10.1177/1090198111409748 · Cited by 72 · cited 1×
“, fidelity and clinician behavior [10, 11]; organizational readiness to change [12]), one on five key organization-level constructs (leadership, vision, managerial relations, climate, and absorptive capacity [13]), and two on numerous constructs depicted in a model that was rather limited in scope (Framework for Effec…”
BackgroundSelf-citationPsychometric issues in assessment research.
Haynes, S. N., Nelson, K., & Blaine, D. D. (1999) · Cited by 55
Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Powell, B. J., Palinkas, L. A., & Lewis, C. C. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3709-2 · Cited by 54
Self-citationLin, W., & Yao, G. (2014) · Springer eBooks · doi:10.1007/978-94-007-0753-5_516 · Cited by 49
Measurement Issues in Dissemination and Implementation Research
Lewis, C. C., Proctor, E. K., & Brownson, R. C. (2017) · Oxford University Press eBooks · doi:10.1093/oso/9780190683214.003.0014 · Cited by 43
Self-citationGinty, A. T. (2013) · Springer eBooks · doi:10.1007/978-1-4419-1005-9_480 · Cited by 43
Fidelity of Intervention Implementation: A Review of Instruments
Ibrahim, S., & Sidani, S. (2015) · Health · doi:10.4236/health.2015.712183 · Cited by 36
Khadjesari, Z., Vitoratou, S., Sevdalis, N., & Hull, L. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017972 · Cited by 32
Walsh, W. B., & Betz, N. E. (1995) · Cited by 25
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
“Determinant frameworks provide a base set of concepts, terms, and definitions by which to articulate dynamic complex contexts and develop much needed measures of context [10].”
BackgroundGuidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475
Action, actor, context, target, time (AACTT): a framework for specifying behaviour
Presseau, J., McCleary, N., Lorencatto, F., Patey, A. M., Grimshaw, J. M., & Francis, J. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0951-x · Cited by 411
“Given on-going efforts to systematise the measurement of theoretical constructs [54], and description of intervention components [41–43], there is a need for ensuring that the object of change, i.”
BackgroundImplementation science made too simple: a teaching tool
Curran, G. M. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00001-z · Cited by 285
Woodward, E. N., Singh, R. S., Ndebele-Ngwenya, P., Melgar Castillo, A., Dickson, K. S., & Kirchner, J. E. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00146-5 · Cited by 270
Metadata
- Authors
- Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- Virginia Commonwealth University (US)
- University of Washington (US)
- University of Montana (US)
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Lewis, Cara C.
- ORCID
- Venue
- Systematic Reviews
- Published
- 2018-04-25
- Volume
- 7
- Issue
- 1
- Pages
- 66-66
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5918558
- OpenAlex
- W2806234161
- Cited by
- 118
- References
- 36
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 9 · 2023: 26 · 2022: 14 · 2021: 25 · 2020: 18 · 2019: 7 · 2018: 2
- FWCI
- 2.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- implementation science, psychometric properties, reliability and validity, Consolidated Framework for Implementation Research, evidence-based practice, mental health interventions, implementation process
- MeSH terms
- Systematic Reviews as Topic, Health Plan Implementation, Humans, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.