Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Background: Despite their inclusion in Rogers’ seminal diffusion of innovations theory, few implementation studies empirically evaluate the role of intervention characteristics. Now, with growing evidence on the role of adaptation in implementation, high-quality measures of characteristics such as adaptability, trialability, and complexity are needed. Only two systematic reviews of implementation measures captured those related to the intervention or innovation and their assessment of psychometric properties was limited. This manuscript reports on the results of eight systematic reviews of measures of intervention characteristics with nuanced data regarding a broad range of psychometric properties. Methods: The systematic review proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and citation searches. Phase II, data extraction, involved coding psychometric information. Phase III, data analysis, involved two trained specialists independently rating each measure using PAPERS (Psychometric And Pragmatic Evidence Rating Scales). Results: Searches identified 16 measures or scales: zero for intervention source, one for evidence strength and quality, nine for relative advantage, five for adaptability, six for trialability, nine for complexity, and two for design quality and packaging. Information about internal consistency and norms was available for most measures, whereas information about other psychometric properties was most often not available. Ratings for psychometric properties fell in the range of “poor” to “good.” Conclusion: The results of this review confirm that few implementation scholars are examining the role of intervention characteristics in behavioral health studies. Significant work is needed to both develop new measures (e.g., for intervention source) and build psychometric evidence for existing measures in this forgotten domain. Plain Language Summary: Intervention characteristics have long been perceived as critical factors that directly influence the rate of adopting an innovation. It remains unclear the extent to which intervention characteristics including relative advantage, complexity, trialability, intervention source, design quality and packaging, evidence strength and quality, adaptability, and cost impact implementation of evidence-based practices in behavioral health settings. To unpack the differential influence of these factors, high quality measures are needed. Systematic reviews can identify measures and synthesize the data regarding their quality to identify gaps in the field and inform measure development and testing efforts. Two previous reviews identified measures of intervention characteristics, but they did not provide information about the extent of the existing evidence nor did they evaluate the host of evidence available for identified measures. This manuscript summarizes the results of nine systematic reviews (i.e., one for each of the factors listed above) for which 16 unique measures or scales were identified. The nuanced findings will help direct measure development work in this forgotten domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Systematic reviews of implementation context measures have been conducted, all of which have found significant gaps and signal the continued need for measure validation and development [89–95].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 8×
“Email: cara.c.lewis@kp.org In 2009, the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) synthesized the implementation theories, frameworks, and models to offer a meta-framework that could guide measurement of key implementation constructs housed within five…”
MethodInfluentialMoore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150 · cited 3×
“The rating for internal consistency was “3—good” (Adoption of Information Technology Innovation Result Demonstrability scale; Moore & Benbasat, 1991).”
BackgroundProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 1×
“Trained research specialists mapped measures and/or their scales to one or more of the eight aforementioned intervention characteristics (Proctor et al., 2011).”
BackgroundAdvancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 2×
“Characteristics included: (1) country of origin, (2) concept defined by authors, (3) number of articles contained in each measure packet, (4) number of scales, (5) number of items, (6) setting in which measure had been used, (7) level of analysis, (8) target problem, and (9) stage of implementation as defined by the E…”
BackgroundMethodTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978 · cited 1×
“Moreover, this approach offers potential users of a measure a single rating capturing the evidence across all previous uses that is less negative than would be produced with the “worst score counts” approach that accounts for only a single study (Terwee et al., 2012).”
BackgroundWiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0898-y · Cited by 1,329
Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223 · cited 1×
“Intervention characteristics may have a differential influence across the stages or phases of implementation (Moullin et al., 2019).”
BackgroundAssessing organizational readiness for change
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00233-7 · Cited by 863 · cited 1×
“Five measures of adaptability were identified, four of which were scales within broader measures (e.g., Texas Christian University Organizational Readiness for Change— Adaptability Scale; Lehman et al., 2002; McLean, 2013).”
Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800 · cited 3×
“Both studies concluded, across the domains they reviewed, that the state of measurement was only emerging, with only 38% of measures across both studies demonstrating any evidence of reliability and validity (45% of those in Chor et al., 2015; 25% of those in Chaudoir et al., 2013).”
MethodResultChambers, D. A., & Norton, W. E. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.011 · Cited by 581 · cited 1×
“Beyond instruments that measure adaptability, the growing focus on intervention adaptation in implementation more generally (e.g., Chambers & Norton, 2016; Stirman et al., 2019) indicates that instruments assessing all manner of intervention characteristics may be useful to evaluate the outcomes of…”
BackgroundThe Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364
Implementation science made too simple: a teaching tool
Curran, G. M. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00001-z · Cited by 285 · cited 2×
“The current study confirms that intervention characteristics are understudied in behavioral health implementation research, at least with regard to the development and testing of measures intended to assess “the thing” being implemented (Curran, 2020).”
BackgroundLyon, A. R., Munson, S. A., Renn, B. N., Atkins, D. C., Pullmann, M. D., Friedman, E., & Areán, P. A. (2019) · JMIR Research Protocols · doi:10.2196/14990 · Cited by 187 · cited 1×
“For instance, intervention adaptation may be undertaken with the explicit objective of improving design quality (Lyon et al., 2019).”
BackgroundSelf-citationBarriers to implementing treatment integrity procedures: Survey of treatment outcome researchers.
Perepletchikova, F., Hilt, L. M., Chereji, E., & Kazdin, A. E. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0015232 · Cited by 185 · cited 1×
“The rating for internal consistency was “3—good” and “−1—poor” for norms with a total combined score of “two—adequate” (maximum possible score = 36; Barriers to Treatment Integrity Implementation Survey; Perepletchikova et al., 2009).”
BackgroundLyon, A. R., & Bruns, E. J. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.3060 · Cited by 178 · cited 2×
“Despite the long-standing interest in these constructs theoretically, Lyon and Bruns (2019) have recently referred to intervention characteristics as the forgotten domain of implementation science.”
BackgroundSelf-citationMiake-Lye, I. M., Delevan, D. M., Ganz, D. A., Mittman, B. S., & Finley, E. P. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-4926-z · Cited by 170 · cited 2×
“Of the 15 systemic reviews of implementation measures (J. D. Allen et al., 2017; Miake-Lye et al., 2020; Rabin et al., 2016; Willmeroth et al., 2019), only two explored measures of intervention/innovation characteristics (Chaudoir et al., identified measures of intervention characteristics, but…”
BackgroundQuantitative measures of health policy implementation determinants and outcomes: a systematic review
Allen, P., Pilar, M., Walsh-Bailey, C., Hooley, C., Mazzucca, S., Lewis, C. C., Mettert, K. D., Dorsey, C. N., Purtle, J., Kepper, M. M., et al. (2020) · Implementation Science · doi:10.1186/s13012-020-01007-w · Cited by 164 · cited 1×
“Any point of controversy during the rating for this study was resolved by the project leads and utilized as a teaching opportunity that resulted in a manual to accompany PAPERS that has supported two new research teams in using our protocols (P. Allen et al., 2020; Khadjesari et al., 2020).”
MethodSelf-citationLewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2016) · Implementation Science · doi:10.1186/s13012-014-0193-x · Cited by 155 · cited 1×
“Consistent with findings from our team’s other systematic reviews across the CFIR domains (Lewis et al., 2016), the most common psychometric properties reported across all identified measures were internal consistency and norms.”
ResultSelf-citationAn updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118
Self-citationStanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117 · cited 2×
“The full rating system and criteria for the PAPERS are published elsewhere (Lewis et al., 2018; Stanick et al., 2021).”
BackgroundMethodSelf-citationCarper, M. M., McHugh, R. K., & Barlow, D. H. (2013) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0377-5 · Cited by 92
Measurement resources for dissemination and implementation research in health
Rabin, B. A., Lewis, C. C., Norton, W. E., Neta, G., Chambers, D., Tobin, J. N., Brownson, R. C., & Glasgow, R. E. (2015) · Implementation Science · doi:10.1186/s13012-016-0401-y · Cited by 92 · cited 2×
“Of the 15 systemic reviews of implementation measures (J. D. Allen et al., 2017; Miake-Lye et al., 2020; Rabin et al., 2016; Willmeroth et al., 2019), only two explored measures of intervention/innovation characteristics (Chaudoir et al., identified measures of intervention characteristics, but…”
BackgroundSelf-citationImplementation outcome instruments for use in physical healthcare settings: a systematic review
Khadjesari, Z., Boufkhed, S., Vitoratou, S., Schatte, L., Ziemann, A., Daskalopoulou, C., Uglik-Marucha, E., Sevdalis, N., & Hull, L. (2020) · Implementation Science · doi:10.1186/s13012-020-01027-6 · Cited by 92
Allen, J. D., Towne, S. D., Maxwell, A. E., DiMartino, L., Leyva, B., Bowen, D. J., Linnan, L., & Weiner, B. J. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2459-x · Cited by 91 · cited 1×
“Of the 15 systemic reviews of implementation measures (J. D. Allen et al., 2017; Miake-Lye et al., 2020; Rabin et al., 2016; Willmeroth et al., 2019), only two explored measures of intervention/innovation characteristics (Chaudoir et al., identified measures of intervention characteristics, but…”
BackgroundMeasures for Predictors of Innovation Adoption
Chor, K. H. B., Wisdom, J. P., Olin, S.-C. S., Hoagwood, K. E., & Horwitz, S. M. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0551-7 · Cited by 85 · cited 3×
“Both studies concluded, across the domains they reviewed, that the state of measurement was only emerging, with only 38% of measures across both studies demonstrating any evidence of reliability and validity (45% of those in Chor et al., 2015; 25% of those in Chaudoir et al., 2013).”
ResultPerceived Characteristics of Intervention Scale: Development and Psychometric Properties
Cook, J. M., Thompson, R., & Schnurr, P. P. (2015) · Assessment · doi:10.1177/1073191114561254 · Cited by 79 · cited 4×
“Nine measures of relative advantage were identified, of which eight were scales within a broader measure (e.g., The Perceived Characteristics of Intervention Scale—Relative Advantage Scale; Cook et al., 2015; Moore & Benbasat, 1991).”
InfluentialMeasurement of a model of implementation for health care: toward a testable theory
Cook, J. M., O’Donnell, C., Dinnen, S., Coyne, J. C., Ruzek, J. I., & Schnurr, P. P. (2012) · Implementation Science · doi:10.1186/1748-5908-7-59 · Cited by 59
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2015) · Implementation Science · doi:10.1186/s13012-015-0364-4 · Cited by 58 · cited 2×
“…and behavioral health care (psychometric total maximum score = [4]; maximum possible score = 36), with ratings of “3—good” and “2—adequate” respectfully for internal consistency and “1—minimal/emerging” and “2— adequate” respectfully for norms (Cook et al., 2015; Hunter et al., 2015; Yetter, 2010).”
BackgroundWeiner, B. J., Mettert, K. D., Dorsey, C. N., Nolen, E. A., Stanick, C., Powell, B. J., & Lewis, C. C. (2020) · Implementation Research and Practice · doi:10.1177/2633489520933896 · Cited by 58 · cited 1×
“Concentrated work on this domain is warranted, particularly when compared with the sheer number of measures identified for single constructs at other system levels like “leadership engagement” (inner setting level) for which 24 unique measures emerged (Weiner et al., 2020).”
ResultSelf-citationWillmeroth, T., Wesselborg, B., & Kuske, S. (2019) · INQUIRY: The Journal of Health Care Organization, Provision, and Financing · doi:10.1177/0046958019861257 · Cited by 53 · cited 2×
“Of the 15 systemic reviews of implementation measures (J. D. Allen et al., 2017; Miake-Lye et al., 2020; Rabin et al., 2016; Willmeroth et al., 2019), only two explored measures of intervention/innovation characteristics (Chaudoir et al., identified measures of intervention characteristics, but…”
BackgroundPeltzer, K., Matseke, G., Azwihangwisi, M., & Babor, T. (2008) · Croatian medical journal · doi:10.3325/cmj.2008.3.392 · Cited by 51 · cited 1×
“Six measures of trialability were identified, all of which were scales of broader measures (e.g., Pelzer SBI Implementation Attitude Questionnaire—Trialability Scale) (McLean, 2013; Peltzer et al., 2008).”
BackgroundScott, V. C., Kenworthy, T., Godly-Reynolds, E., Bastien, G., Scaccia, J., McMickens, C., Rachel, S., Cooper, S., Wrenn, G., & Wandersman, A. (2017) · American Journal of Orthopsychiatry · doi:10.1037/ort0000270 · Cited by 49 · cited 1×
“Scale; McLean, 2013; Scott et al., 2017).”
Borah, E. V., Wright, E. C., Donahue, D. A., Cedillos, E. M., Riggs, D. S., Isler, W. C., & Peterson, A. L. (2013) · Military Medicine · doi:10.7205/milmed-d-13-00072 · Cited by 39
Turzhitsky, V. M., Rogers, J. D., Mutyal, N. N., Roy, H. K., & Backman, V. (2009) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2009.2032666 · Cited by 31 · cited 1×
“This is ironic, given its prominence in Rogers’ seminal diffusion of innovations theory (Turzhitsky et al., 2010) and other theoretical frameworks or teaching tools that explicitly call out the importance of aspects of the innovation—or “the thing” (Curran, 2020)—being implemented.”
BackgroundMalte, C. A., McFall, M., Chow, B., Beckham, J. C., Carmody, T. P., & Saxon, A. J. (2013) · Psychology of Addictive Behaviors · doi:10.1037/a0028484 · Cited by 19 · cited 1×
“Finally, two measures of design quality & packaging were identified, both of which were scales within broader measures (e.g., Malte PostTreatment Smoking Cessation Beliefs Measure—Design Quality; Malte et al., 2013; McLean, 2013).”
BackgroundNeumiller, S., Bennett-Clark, F., Young, M. S., Dates, B. G., Broner, N., Leddy, J., Kendall, D., Richards, S., & Jong, F. D. (2009) · Journal of Dual Diagnosis · doi:10.1080/15504260903175973 · Cited by 19
Yetter, G. (2010) · Journal of Educational and Psychological Consultation · doi:10.1080/10474411003785370 · Cited by 17 · cited 8×
“Nine measures of relative advantage were identified, of which eight were scales within a broader measure (e.g., The Perceived Characteristics of Intervention Scale—Relative Advantage Scale; Cook et al., 2015; Moore & Benbasat, 1991).”
BackgroundMethodInfluentialMoise, I. K., Green, D., Toth, J., & Mulhall, P. F. (2014) · Substance Use & Misuse · doi:10.3109/10826084.2014.880484 · Cited by 6
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
Reimagining Rural: Shifting Paradigms About Health and Well-Being in the Rural United States
Afifi, R., Parker, E. A., Dino, G., Hall, D. M., & Ulin, B. F. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-052020-123413 · Cited by 92
“Key features of successful diffusion in agriculture, which remain critical to IS, include the relative advantage of the innovative practice over the current one and the ability to try out the innovative practice prior to rollout, among others (73, 109).”
BackgroundLyon, A. R., Connors, E. H., Lawson, G. M., Nadeem, E., & Owens, J. S. (2024) · School Mental Health · doi:10.1007/s12310-024-09731-0 · Cited by 58
“Specification of mechanisms can allow for investigation of null effects, replication of positive findings, identification of which strategy components are inert or unnecessary, and adaptation of strategies to fit new contexts (Lewis et al., 2021b).”
Self-citationHelland, S. S., Mellblom, A. V., Kjøbli, J., Wentzel-Larsen, T., Espenes, K., Engell, T., & Kirkøen, B. (2022) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01213-2 · Cited by 41
“In recent years, what we implement has also gained attention as important implementation determinants (Lewis et al., 2021), including the EBIs design, content, and fit with diverse needs across mental Adolescents’ mental health problems are increasing worldwide, with significant consequences for…”
BackgroundOptimizing Implementation in Cancer Control (OPTICC): protocol for an implementation science center
Lewis, C. C., Hannon, P. A., Klasnja, P., Baldwin, L., Hawkes, R. J., Blackmer, J., & Johnson, A. N. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00117-w · Cited by 27
Self-citation
Metadata
- Authors
- Lewis, C. C., Mettert, K., & Lyon, A. R.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- University of Washington (US)
- Corresponding author
- Lewis, Cara C
- Venue
- Implementation Research and Practice
- Publisher
- SAGE Publications
- Published
- 2021-01
- Volume
- 2
- Pages
- 2633489521994197-2633489521994197
- ISSN
- 2633-4895 (ISSN-L)
- PMC
- PMC9978609
- OpenAlex
- W3136813232
- Cited by
- 38
- References
- 38
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 13 · 2023: 5 · 2022: 4 · 2021: 2
- FWCI
- 1.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Research and Practice (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (1R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- intervention characteristics, implementation success, diffusion of innovations, psychometric properties, reliability and validity, adaptability, trialability, complexity, relative advantage, behavioral health interventions, evidence-based practice adoption, scale development, implementation measures, cost impact, innovation adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.