Measuring readiness for implementation: A systematic review of measures’ psychometric and pragmatic properties
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Background: Systematic measure reviews can facilitate advances in implementation research and practice by locating reliable, valid, pragmatic measures; identifying promising measures needing refinement and testing; and highlighting measurement gaps. This review identifies and evaluates the psychometric and pragmatic properties of measures of readiness for implementation and its sub-constructs as delineated in the Consolidated Framework for Implementation Research: leadership engagement, available resources, and access to knowledge and information. Methods: The systematic review methodology is described fully elsewhere. The review, which focused on measures used in mental or behavioral health, proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and cited citation searches. Phase II, data extraction, involved coding relevant psychometric and pragmatic information. Phase III, data analysis, involved two trained specialists independently rating each measure using Psychometric and Pragmatic Evidence Rating Scales (PAPERS). Frequencies and central tendencies summarized information availability and PAPERS ratings. Results: Searches identified 9 measures of readiness for implementation, 24 measures of leadership engagement, 17 measures of available resources, and 6 measures of access to knowledge and information. Information about internal consistency was available for most measures. Information about other psychometric properties was often not available. Ratings for internal consistency were “adequate” or “good.” Ratings for other psychometric properties were less than “adequate.” Information on pragmatic properties was most often available regarding cost, language readability, and brevity. Information was less often available regarding training burden and interpretation burden. Cost and language readability generally exhibited “good” or “excellent” ratings, interpretation burden generally exhibiting “minimal” ratings, and training burden and brevity exhibiting mixed ratings across measures. Conclusion: Measures of readiness for implementation and its sub-constructs used in mental health and behavioral health care are unevenly distributed, exhibit unknown or low psychometric quality, and demonstrate mixed pragmatic properties. This review identified a few promising measures, but targeted efforts are needed to systematically develop and test measures that are useful for both research and practice. Plain language abstract: Successful implementation of effective mental health or behavioral health treatments in service delivery settings depends in part on the readiness of the service providers and administrators to implement the treatment; the engagement of organizational leaders in the implementation effort; the resources available to support implementation, such as time, money, space, and training; and the accessibility of knowledge and information among service providers about the treatment and how it works. It is important that the methods for measuring these factors are dependable, accurate, and practical; otherwise, we cannot assess their presence or strength with confidence or know whether efforts to increase their presence or strength have worked. This systematic review of published studies sought to identify and evaluate the quality of questionnaires (referred to as measures) that assess readiness for implementation, leadership engagement, available resources, and access to knowledge and information. We identified 56 measures of these factors and rated their quality in terms of how dependable, accurate, and practical they are. Our findings indicate there is much work to be done to improve the quality of available measures; we offer several recommendations for doing so.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 7×
“Readiness for implementation, defined as “tangible and immediate indicators of organizational commitment to its decision to implement an intervention,” is considered a critical precursor to successful implementation (Damschroder et al., 2009; Gleeson, 2017; Kotter, 1996; Weiner, 2009).”
BackgroundInfluentialProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 2×
“These variables included construct definition, number of scales in the measure, number of items in the measure, number of uses of the measure in mental or behavioral health care, country of origin, target setting, target respondent, target problem, and implementation outcome assessed (Proctor et al., 2011).”
BackgroundA theory of organizational readiness for change
Weiner, B. J. (2009) · Implementation Science · doi:10.1186/1748-5908-4-67 · Cited by 2,268
Self-citationTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978 · cited 1×
“negative assessment of a PAPERS property than the “worst counts” approach wherein a PAPERS property is assigned the lowest rating for that property when there are multiple ratings (Lewis et al., 2015; Terwee et al., 2012).”
BackgroundChaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800
Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · cited 6×
“…previously published reviews identified no psychometrically strong, pragmatic measures in health care, education, or business (Gagnon et al., 2014; Weiner et al., 2008); although, the first author’s Organizational Readiness for Implementation Change (ORIC) measure exhibits several desirable…”
BackgroundResultInfluentialSelf-citationAarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559 · cited 5×
“” Note the median rating of “2— adequate” for discriminant validity was based on the rating of just one measure: the Implementation Leadership Scale (Aarons et al., 2014).”
BackgroundMethodInfluentialA conceptual framework for transferring research to practice
Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00231-3 · Cited by 538 · cited 6×
“, 2014) for leadership engagement, the Texas Christian University Program Training Needs Survey (Simpson, 2002) for available resources, and the Structured Interview of Evidence Use (Palinkas et al.”
MethodInfluentialEhrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/s13012-014-0157-1 · Cited by 502 · cited 2×
“Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS).”
MethodLewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015) · Implementation Science · doi:10.1186/s13012-015-0342-x · Cited by 430 · cited 2×
“negative assessment of a PAPERS property than the “worst counts” approach wherein a PAPERS property is assigned the lowest rating for that property when there are multiple ratings (Lewis et al., 2015; Terwee et al., 2012).”
BackgroundSelf-citationGlasgow, R. E., & Riley, W. T. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.010 · Cited by 318 · cited 1×
“Finally, neither review appraised the pragmatic quality of measures, that is, the extent to which they are relevant to practice or policy stakeholders and feasible to use in real-world settings (Glasgow, 2013; Glasgow & Riley, 2013; Stanick et al., 2019).”
BackgroundInstrumentation issues in implementation science
Martinez, R. G., Lewis, C. C., & Weiner, B. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0118-8 · Cited by 243 · cited 3×
“…science faces a host of measurement issues, including reliance on single-use, adapted measures with unknown reliability and validity; skewed distribution of measures across theoretically important constructs; and inattention to practical measurement considerations (Martinez et al., 2014).”
BackgroundInfluentialSelf-citationMiake-Lye, I. M., Delevan, D. M., Ganz, D. A., Mittman, B. S., & Finley, E. P. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-4926-z · Cited by 170
An updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118 · cited 9×
“Systematic measure reviews help new fields like implementation science address instrumentation issues by locating reliable, valid, pragmatic measures; identifying promising measures needing further refinement and evaluation, and highlighting measurement gaps for key constructs (Lewis et al., 2018).”
BackgroundMethodInfluentialSelf-citationStanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117 · cited 5×
“Finally, neither review appraised the pragmatic quality of measures, that is, the extent to which they are relevant to practice or policy stakeholders and feasible to use in real-world settings (Glasgow, 2013; Glasgow & Riley, 2013; Stanick et al., 2019).”
BackgroundMethodInfluentialSelf-citationHerbeck, D. M., Hser, Y.-I., & Teruya, C. (2008) · Addictive Behaviors · doi:10.1016/j.addbeh.2007.12.003 · Cited by 93
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2017) · Implementation Science · doi:10.1186/s13012-017-0606-8 · Cited by 56 · cited 6×
“However, its predictive validity rating was “–1—poor” based on a single study in which implementation leadership was not significantly associated with the extent of sustainment of an evidence-based treatment to address adolescent substance use following the end of federally funded implementation support grants to 78 c…”
BackgroundInfluentialStanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Powell, B. J., Palinkas, L. A., & Lewis, C. C. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3709-2 · Cited by 54
Self-citation
Cited by
Damschroder, L. J., Reardon, C. M., Opra Widerquist, M. A., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-021-01181-5 · Cited by 784
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81
“The resulting standardised measures include survey items tailored to specific characteristics, such as settings, populations, and/or EBIs, which have been calibrated to create standardised scores that are comparable across the tailored items [23].”
BackgroundMethodResultSelf-citationPowell, B. J., Mettert, K. D., Dorsey, C. N., Weiner, B. J., Stanick, C. F., Lengnick-Hall, R., Ehrhart, M. G., Aarons, G. A., Barwick, M. A., Damschroder, L. J., et al. (2021) · Implementation Research and Practice · doi:10.1177/26334895211018862 · Cited by 79
Self-citationLewis, C. C., Mettert, K., & Lyon, A. R. (2021) · Implementation Research and Practice · doi:10.1177/2633489521994197 · Cited by 38
“Concentrated work on this domain is warranted, particularly when compared with the sheer number of measures identified for single constructs at other system levels like “leadership engagement” (inner setting level) for which 24 unique measures emerged (Weiner et al., 2020).”
ResultSelf-citationReflections on the measurement of implementation constructs
Wensing, M. (2021) · Implementation Research and Practice · doi:10.1177/26334895211020125 · Cited by 14
Metadata
- Authors
- Weiner, B. J., Mettert, K. D., Dorsey, C. N., Nolen, E. A., Stanick, C., Powell, B. J., & Lewis, C. C.
- Venue
- Implementation Research and Practice
- Publisher
- SAGE Publications
- Published
- 2020-01
- Volume
- 1
- Pages
- 2633489520933896-2633489520933896
- ISSN
- 2633-4895 (ISSN-L)
- OpenAlex
- W3080795028
- Cited by
- 58
- References
- 18
- FWCI
- 2.07
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.