Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Background: Despite their inclusion in Rogers’ seminal diffusion of innovations theory, few implementation studies empirically evaluate the role of intervention characteristics. Now, with growing evidence on the role of adaptation in implementation, high-quality measures of characteristics such as adaptability, trialability, and complexity are needed. Only two systematic reviews of implementation measures captured those related to the intervention or innovation and their assessment of psychometric properties was limited. This manuscript reports on the results of eight systematic reviews of measures of intervention characteristics with nuanced data regarding a broad range of psychometric properties. Methods: The systematic review proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and citation searches. Phase II, data extraction, involved coding psychometric information. Phase III, data analysis, involved two trained specialists independently rating each measure using PAPERS (Psychometric And Pragmatic Evidence Rating Scales). Results: Searches identified 16 measures or scales: zero for intervention source, one for evidence strength and quality, nine for relative advantage, five for adaptability, six for trialability, nine for complexity, and two for design quality and packaging. Information about internal consistency and norms was available for most measures, whereas information about other psychometric properties was most often not available. Ratings for psychometric properties fell in the range of “poor” to “good.” Conclusion: The results of this review confirm that few implementation scholars are examining the role of intervention characteristics in behavioral health studies. Significant work is needed to both develop new measures (e.g., for intervention source) and build psychometric evidence for existing measures in this forgotten domain. Plain Language Summary: Intervention characteristics have long been perceived as critical factors that directly influence the rate of adopting an innovation. It remains unclear the extent to which intervention characteristics including relative advantage, complexity, trialability, intervention source, design quality and packaging, evidence strength and quality, adaptability, and cost impact implementation of evidence-based practices in behavioral health settings. To unpack the differential influence of these factors, high quality measures are needed. Systematic reviews can identify measures and synthesize the data regarding their quality to identify gaps in the field and inform measure development and testing efforts. Two previous reviews identified measures of intervention characteristics, but they did not provide information about the extent of the existing evidence nor did they evaluate the host of evidence available for identified measures. This manuscript summarizes the results of nine systematic reviews (i.e., one for each of the factors listed above) for which 16 unique measures or scales were identified. The nuanced findings will help direct measure development work in this forgotten domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Systematic reviews of implementation context measures have been conducted, all of which have found significant gaps and signal the continued need for measure validation and development [89–95].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
Moore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150 · cited 3×
“The rating for internal consistency was “3—good” (Adoption of Information Technology Innovation Result Demonstrability scale; Moore & Benbasat, 1991).”
BackgroundProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 2×
“Characteristics included: (1) country of origin, (2) concept defined by authors, (3) number of articles contained in each measure packet, (4) number of scales, (5) number of items, (6) setting in which measure had been used, (7) level of analysis, (8) target problem, and (9) stage of implementation as defined by the E…”
BackgroundMethodTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978
Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0898-y · Cited by 1,329
Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223 · cited 1×
“Intervention characteristics may have a differential influence across the stages or phases of implementation (Moullin et al., 2019).”
BackgroundAssessing organizational readiness for change
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00233-7 · Cited by 863 · cited 1×
“Five measures of adaptability were identified, four of which were scales within broader measures (e.g., Texas Christian University Organizational Readiness for Change— Adaptability Scale; Lehman et al., 2002; McLean, 2013).”
Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800
Chambers, D. A., & Norton, W. E. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.011 · Cited by 581 · cited 1×
“Beyond instruments that measure adaptability, the growing focus on intervention adaptation in implementation more generally (e.g., Chambers & Norton, 2016; Stirman et al., 2019) indicates that instruments assessing all manner of intervention characteristics may be useful to evaluate the outcomes of…”
Background
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
Reimagining Rural: Shifting Paradigms About Health and Well-Being in the Rural United States
Afifi, R., Parker, E. A., Dino, G., Hall, D. M., & Ulin, B. F. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-052020-123413 · Cited by 92
Lyon, A. R., Connors, E. H., Lawson, G. M., Nadeem, E., & Owens, J. S. (2024) · School Mental Health · doi:10.1007/s12310-024-09731-0 · Cited by 58
Self-citationHelland, S. S., Mellblom, A. V., Kjøbli, J., Wentzel-Larsen, T., Espenes, K., Engell, T., & Kirkøen, B. (2022) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01213-2 · Cited by 41
Optimizing Implementation in Cancer Control (OPTICC): protocol for an implementation science center
Lewis, C. C., Hannon, P. A., Klasnja, P., Baldwin, L., Hawkes, R. J., Blackmer, J., & Johnson, A. N. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00117-w · Cited by 27
Self-citation
Metadata
- Authors
- Lewis, C. C., Mettert, K., & Lyon, A. R.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- University of Washington (US)
- Corresponding author
- Lewis, Cara C
- Venue
- Implementation Research and Practice
- Publisher
- SAGE Publications
- Published
- 2021-01
- Volume
- 2
- Pages
- 2633489521994197-2633489521994197
- ISSN
- 2633-4895 (ISSN-L)
- PMC
- PMC9978609
- OpenAlex
- W3136813232
- Cited by
- 38
- References
- 38
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 13 · 2023: 5 · 2022: 4 · 2021: 2
- FWCI
- 1.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Research and Practice (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (1R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- intervention characteristics, implementation success, diffusion of innovations, psychometric properties, reliability and validity, adaptability, trialability, complexity, relative advantage, behavioral health interventions, evidence-based practice adoption, scale development, implementation measures, cost impact, innovation adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review. https://app.cochat.ai/papers/10.1177/2633489521994197
- The paper
- Lewis, C. C., Mettert, K., & Lyon, A. R. (2021). Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review. Implementation Research and Practice. https://doi.org/10.1177/2633489521994197
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.