Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS)
Open accessAbstract
BACKGROUND: Although the importance of the organizational environment for implementing evidence-based practices (EBP) has been widely recognized, there are limited options for measuring implementation climate in public sector health settings. The goal of this research was to develop and test a measure of EBP implementation climate that would both capture a broad range of issues important for effective EBP implementation and be of practical use to researchers and managers seeking to understand and improve the implementation of EBPs. METHODS: Participants were 630 clinicians working in 128 work groups in 32 US-based mental health agencies. Items to measure climate for EBP implementation were developed based on past literature on implementation climate and other strategic climates and in consultation with experts on the implementation of EBPs in mental health settings. The sample was randomly split at the work group level of analysis; half of the sample was used for exploratory factor analysis (EFA), and the other half was used for confirmatory factor analysis (CFA). The entire sample was utilized for additional analyses assessing the reliability, support for level of aggregation, and construct-based evidence of validity. RESULTS: The EFA resulted in a final factor structure of six dimensions for the Implementation Climate Scale (ICS): 1) focus on EBP, 2) educational support for EBP, 3) recognition for EBP, 4) rewards for EBP, 5) selection for EBP, and 6) selection for openness. This structure was supported in the other half of the sample using CFA. Additional analyses supported the reliability and construct-based evidence of validity for the ICS, as well as the aggregation of the measure to the work group level. CONCLUSIONS: The ICS is a very brief (18 item) and pragmatic measure of a strategic climate for EBP implementation. It captures six dimensions of the organizational context that indicate to employees the extent to which their organization prioritizes and values the successful implementation of EBPs. The ICS can be used by researchers to better understand the role of the organizational context on implementation outcomes and by organizations to evaluate their current climate as they consider how to improve the likelihood of implementation success.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is an increasing number of measures of implementation determinants, such as implementation leadership, implementation climate, or implementation intentions.
“Furthermore, there is an increasing number of measures of implementation determinants [74, 75] (e.g., implementation leadership [76], implementation climate [77, 78], or implementation intentions [79]).”
From Ten recommendations for using implementation frameworks in research and practice (2020) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788 · see citing works
Evaluating the use of exploratory factor analysis in psychological research.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.3.272 · Cited by 9,587 · see citing works
A Rationale and Test for the Number of Factors in Factor Analysis
Horn, J. L. (1965) · Psychometrika · doi:10.1007/bf02289447 · Cited by 8,844 · see citing works
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · see citing works
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 1× · see citing works
“The development and availability of a measure of strategic climate for EBP implementation is important, particularly due to recent theoretical and empirical emphasis on the importance of the organizational context for the success of EBP implementation and sustainment [20,39-41].”
BackgroundSelf-citationComparison of five rules for determining the number of components to retain.
Zwick, W. R., & Velicer, W. F. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.3.432 · Cited by 3,139 · see citing works
Safety climate in industrial organizations: Theoretical and applied implications.
Zohar, D. M. (1980) · Journal of Applied Psychology · doi:10.1037/0021-9010.65.1.96 · Cited by 2,527 · see citing works
A theory of organizational readiness for change
Weiner, B. J. (2009) · Implementation Science · doi:10.1186/1748-5908-4-67 · Cited by 2,268 · see citing works
The Challenge of Innovation Implementation
Klein, K. J., & Sorra, J. S. (1996) · The Academy of Management Review · doi:10.2307/259164 · Cited by 2,113 · see citing works
Organizational Climate and Culture
Ehrhart, M. G. (2016) · The SAGE Encyclopedia of Theory in Psychology · doi:10.4135/9781483346274.n219 · Cited by 2,055 · see citing works
Self-citationProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · see citing works
Self-citationGriffin, M. A., & Neal, A. F. (2000) · Journal of Occupational Health Psychology · doi:10.1037/1076-8998.5.3.347 · Cited by 1,881 · see citing works
Workplace safety: a meta-analysis of the roles of person and situation factors.
Christian, M., Bradley, J. C., Wallace, J., & Burke, M. J. (2009) · Journal of Applied Psychology · doi:10.1037/a0016172 · Cited by 1,717 · see citing works
Linking service climate and customer perceptions of service quality: Tests of a causal model.
Schneider, B. N., White, S. S., & Paul, M. C. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.2.150 · Cited by 1,716 · see citing works
Managerial Behavior, Performance, and Effectiveness.
Miljus, R. C., Campbell, J. P., Dunnette, M. D., Lawler, E. E., & Weick, K. E. (1971) · Industrial and Labor Relations Review · doi:10.2307/2521540 · Cited by 1,507 · see citing works
Aarons, G. A. (2004) · Mental Health Services Research · doi:10.1023/b:mhsr.0000024351.12294.65 · Cited by 1,272 · see citing works
Self-citationPatterson, M. G., West, M. A., Shackleton, V. J., Dawson, J., Lawthom, R., Maitlis, S., Robinson, D. L., & Wallace, A. M. R. (2005) · Journal of Organizational Behavior · doi:10.1002/job.312 · Cited by 1,262 · see citing works
The Handbook of Organizational Culture and Climate
Ashkanasy, N. M., Wilderom, C. P. M., & Peterson, M. F. (2010) · Sage eBooks · doi:10.4135/9781483307961 · Cited by 979 · see citing works
Assessing organizational readiness for change
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00233-7 · Cited by 863 · see citing works
Organizational readiness for implementing change: a psychometric assessment of a new measure
Shea, C. M., Jacobs, S., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014) · Implementation Science · doi:10.1186/1748-5908-9-7 · Cited by 863 · see citing works
The cross‐level effects of culture and climate in human service teams
Glisson, C., & James, L. R. (2002) · Journal of Organizational Behavior · doi:10.1002/job.162 · Cited by 758 · see citing works
Perceptions of organizational change: A stress and coping perspective.
Rafferty, A. E., & Griffin, M. A. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.5.1154 · Cited by 719 · see citing works
Employee and Customer Perceptions of Service in Banks
Schneider, B., Parkington, J. J., & Buxton, V. M. (1980) · Administrative Science Quarterly · doi:10.2307/2392454 · Cited by 640 · see citing works
Implementing computerized technology: An organizational analysis.
Klein, K. J., Conn, A. B., & Sorra, J. S. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.5.811 · Cited by 615 · see citing works
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., Green, P., & The Research Network on Youth Mental Health (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0148-5 · Cited by 608 · see citing works
Helfrich, C. D., Li, Y.-F., Sharp, N. D., & Sales, A. E. (2009) · Implementation Science · doi:10.1186/1748-5908-4-38 · Cited by 564 · see citing works
Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559 · see citing works
Self-citationKlein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.1.3 · Cited by 511 · see citing works
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Sklar, M. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182447 · Cited by 474 · see citing works
Self-citationSafety climate and injuries: An examination of theoretical and empirical relationships.
Beus, J. M., Payne, S. C., Bergman, M. E., & Arthur, W. (2010) · Journal of Applied Psychology · doi:10.1037/a0019164 · Cited by 459 · see citing works
Organizational Climate: Measures, Research and Contingencies.
Hellriegel, D., & Slocum, J. W. (1974) · Academy of Management Journal · doi:10.2307/254979 · Cited by 447 · see citing works
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015) · Implementation Science · doi:10.1186/s13012-014-0192-y · Cited by 435 · see citing works
Self-citationOrganizational Climate and Culture
Ehrhart, M. G., Schneider, B., & Macey, W. H. (2013) · Routledge eBooks · doi:10.4324/9781315857664 · Cited by 415 · see citing works
Self-citationHong, Y., Liao, H., Hu, J., & Jiang, K. (2013) · Journal of Applied Psychology · doi:10.1037/a0031666 · Cited by 386 · see citing works
The Organizational Context of Children's Mental Health Services
Glisson, C. A. (2002) · Clinical Child and Family Psychology Review · doi:10.1023/a:1020972906177 · Cited by 366 · see citing works
Patil, V. H., Singh, S. N., Mishra, S. K., & Donavan, D. T. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2007.05.008 · Cited by 326 · see citing works
Interrater Agreement Reconsidered: An Alternative to the rwg Indices
Brown, R. D., & Hauenstein, N. M. A. (2005) · Organizational Research Methods · doi:10.1177/1094428105275376 · Cited by 299 · see citing works
Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009) · American Journal of Public Health · doi:10.2105/ajph.2009.161711 · Cited by 299 · see citing works
Self-citationDusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005) · Health Education Research · doi:10.1093/her/cyg134 · Cited by 278 · see citing works
Aarons, G. A., Sommerfeld, D. H., & Walrath-Greene, C. M. (2009) · Implementation Science · doi:10.1186/1748-5908-4-83 · Cited by 276 · see citing works
Self-citationWhitman, D. S., Caleo, S., Carpenter, N. C., Horner, M. T., & Bernerth, J. B. (2012) · Journal of Applied Psychology · doi:10.1037/a0028021 · Cited by 265 · see citing works
The meaning and measurement of implementation climate
Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011) · Implementation Science · doi:10.1186/1748-5908-6-78 · Cited by 260 · see citing works
Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy.
Glisson, C., Schoenwald, S. K., Hemmelgarn, A., Green, P., Dukes, D., Armstrong, K. S., & Chapman, J. E. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0019160 · Cited by 243 · see citing works
Context matters: measuring implementation climate among individuals and groups
Jacobs, S., Weiner, B. J., & Bunger, A. C. (2014) · Implementation Science · doi:10.1186/1748-5908-9-46 · Cited by 189 · see citing works
Organizational Climate and Culture: An Introduction to Theory, Research, and Practice
Hummer, D. A. (2016) · Human Resource Development Quarterly · doi:10.1002/hrdq.21246 · Cited by 179 · see citing works
Organizational Climate Research: Achievements and the Road Ahead
Schneider, B. N., Ehrhart, M. G., & Macey, W. H. (2010) · Sage eBooks · doi:10.4135/9781483307961.n3 · Cited by 109 · see citing works
Self-citationParallel Analysis Engine to Aid Determining Number of Factors to Retain (Computer software)
Patil, V., Singh, S., Mishra, S. K., & Donovan, T. (2007) · Cited by 98 · see citing works
Aarons, G. A., Sommerfeld, D. H., & Willging, C. E. (2011) · Psychological Services · doi:10.1037/a0026196 · Cited by 98 · see citing works
Self-citationSystems Antecedents for Dissemination and Implementation: A Review and Analysis of Measures
Emmons, K. M., Weiner, B., Fernandez, M. E., & Tu, S.-P. (2012) · Health Education & Behavior · doi:10.1177/1090198111409748 · Cited by 72 · see citing works
Performance of Fit Indices in Different Conditions and the Selection of Cut-off Values
Qing-Ke, G. U. O. (2008) · Acta Psychologica Sinica · doi:10.3724/sp.j.1041.2008.00109 · Cited by 20 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [43]Aarons GA Farahnak LR Ehrhart MG Beidas RS Kendall PC Leadership and strategic organizational climate to support evidence-based practice implementation Child and adolescent therapy: Dissemination and implementation of empirically supported treatments 2014 New York, NY Guilford Press
Cited by
Sorted by citations
Psychometric assessment of three newly developed implementation outcome measures
Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017) · Implementation Science · doi:10.1186/s13012-017-0635-3 · Cited by 2,819 · see citing works
Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223 · see citing works
Self-citationTen recommendations for using implementation frameworks in research and practice
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Albers, B., Nilsen, P., Broder-Fingert, S., Mukasa, B., & Aarons, G. A. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00023-7 · Cited by 488 · see citing works
Self-citationAarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015) · Implementation Science · doi:10.1186/s13012-014-0192-y · Cited by 435 · see citing works
Self-citationAarons, G. A., Green, A. E., Trott, E., Willging, C. E., Torres, E. M., Ehrhart, M. G., & Roesch, S. C. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0751-4 · Cited by 251 · see citing works
Self-citationWilliams, N. J., & Beidas, R. S. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12960 · Cited by 233 · see citing works
Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., & Aarons, G. A. (2018) · Implementation Science · doi:10.1186/s13012-017-0705-6 · Cited by 199 · see citing works
Self-citationAarons, G. A., Ehrhart, M. G., Moullin, J. C., Torres, E. M., & Green, A. E. (2017) · Implementation Science · doi:10.1186/s13012-017-0562-3 · Cited by 159 · see citing works
Self-citationHagermoser Sanetti, L. M., & Collier-Meek, M. A. (2019) · Journal of School Psychology · doi:10.1016/j.jsp.2019.07.008 · Cited by 129 · see citing works
Aarons, G. A., Green, A. E., Willging, C. E., Ehrhart, M. G., Roesch, S. C., Hecht, D. B., & Chaffin, M. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0183-z · Cited by 107 · see citing works
Self-citationMoullin, J. C., Sklar, M., Ehrhart, M. G., Green, A., & Aarons, G. A. (2021) · Implementation Science · doi:10.1186/s13012-021-01152-w · Cited by 82 · see citing works
Self-citationPowell, B. J., Mettert, K. D., Dorsey, C. N., Weiner, B. J., Stanick, C. F., Lengnick-Hall, R., Ehrhart, M. G., Aarons, G. A., Barwick, M. A., Damschroder, L. J., et al. (2021) · Implementation Research and Practice · doi:10.1177/26334895211018862 · Cited by 79 · see citing works
Self-citationRichter, A., von Thiele Schwarz, U., Lornudd, C., Lundmark, R., Mosson, R., & Hasson, H. (2015) · Implementation Science · doi:10.1186/s13012-016-0475-6 · Cited by 75 · see citing works
Shuman, C. J., Liu, X., Aebersold, M. L., Tschannen, D., Banaszak-Holl, J., & Titler, M. G. (2018) · Implementation Science · doi:10.1186/s13012-018-0753-6 · Cited by 67 · see citing works
Glasgow, R. E., Trinkley, K. E., Ford, B., & Rabin, B. A. (2024) · Global Implementation Research and Applications · doi:10.1007/s43477-024-00134-6 · Cited by 70 · see citing works
Aarons, G. A., Ehrhart, M. G., Torres, E. M., Finn, N. K., & Roesch, S. C. (2016) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2016.05.004 · Cited by 57 · see citing works
Self-citationCross-Validation of the Implementation Leadership Scale (ILS) in Child Welfare Service Organizations
Finn, N. K., Torres, E. M., Ehrhart, M. G., Roesch, S. C., & Aarons, G. A. (2016) · Child Maltreatment · doi:10.1177/1077559516638768 · Cited by 45 · see citing works
Self-citationHodge, L. M., Turner, K. M. T., Sanders, M. R., & Filus, A. (2017) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-016-9505-z · Cited by 38 · see citing works
Rosner, R., Sachser, C., Hornfeck, F., Kilian, R., Kindler, H., Muche, R., Müller, L. R. F., Thielemann, J., Waldmann, T., Ziegenhain, U., et al. (2020) · Trials · doi:10.1186/s13063-020-04922-x · Cited by 40 · see citing works
Wiltsey Stirman, S., Finley, E. P., Shields, N., Cook, J., Haine-Schlagel, R., Burgess, J. F., Dimeff, L., Koerner, K., Suvak, M., Gutner, C. A., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0544-5 · Cited by 35 · see citing works
Lyon, A. R., Corbin, C. M., Brown, E. C., Ehrhart, M. G., Locke, J., Davis, C., Picozzi, E., Aarons, G. A., & Cook, C. R. (2022) · Implementation Science · doi:10.1186/s13012-022-01222-7 · Cited by 34 · see citing works
Self-citationBeidas, R. S., Volpp, K. G., Buttenheim, A. N., Marcus, S. C., Olfson, M., Pellecchia, M., Stewart, R. E., Williams, N. J., Becker-Haimes, E. M., Candon, M., et al. (2019) · JMIR Research Protocols · doi:10.2196/12121 · Cited by 28 · see citing works
Identifying the organizational innovation-specific capacity needed for exposure therapy
Becker‐Haimes, E. M., Byeon, Y. V., Frank, H. E., Williams, N. J., Kratz, H. E., & Beidas, R. S. (2020) · Depression and Anxiety · doi:10.1002/da.23035 · Cited by 27 · see citing works
Nguyen, M. X. B., Chu, A. V., Powell, B. J., Tran, H. V., Nguyen, L. H., Dao, A. T. M., Pham, M. D., Vo, S. H., Bui, N. H., Dowdy, D. W., et al. (2020) · Implementation Science · doi:10.1186/s13012-020-01020-z · Cited by 19 · see citing works
- Authors
- Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R.
- Affiliations
- San Diego State University (US)
- University of California San Diego (US)
- Corresponding author
- Ehrhart, Mark G
- Venue
- Implementation Science
- Published
- 2014-10-23
- Volume
- 9
- Issue
- 1
- Pages
- 157-157
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC4210525
- OpenAlex
- W2128459358
- Cited by
- 502
- References
- 50
- Citations by year
- 2026: 33 · 2025: 46 · 2024: 80 · 2023: 61 · 2022: 69 · 2021: 56 · 2020: 40 · 2019: 35 · 2018: 32 · 2017: 23 · 2016: 17 · 2015: 9 · 2014: 1
- FWCI
- 30.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Open Scholarship Institutional Repository (Washington University in St. Louis) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- National Institute of Mental Health (R21MH098124, P30MH074678, R21MH082731, R01MH072961, R25MH080916)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- evidence-based practice implementation, implementation climate, organizational context, mental health agencies, scale development, psychometric validation, clinician attitudes
- MeSH terms
- California, Factor Analysis, Statistical, Female, Health Plan Implementation, Humans, Male, Mental Health Services, Middle Aged, Organizational Innovation, Patient Care Team, Pennsylvania, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
