A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence
Open accessThe DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.
Abstract
Background: Measurement is a critical component for any field. Systematic reviews are a way to locate measures and uncover gaps in current measurement practices. The present study identified measures used in behavioral health settings that assessed all constructs within the Process domain and two constructs from the Inner setting domain as defined by the Consolidated Framework for Implementation Research (CFIR). While previous conceptual work has established the importance social networks and key stakeholders play throughout the implementation process, measurement studies have not focused on investigating the quality of how these activities are being carried out. Methods: The review occurred in three phases: Phase I, data collection included (1) search string generation, (2) title and abstract screening, (3) full text review, (4) mapping to CFIR-constructs, and (5) “cited-by” searches. Phase II, data extraction, consisted of coding information relevant to the nine psychometric properties included in the Psychometric And Pragmatic Rating Scale (PAPERS). In Phase III, data analysis was completed. Results: Measures were identified in only seven constructs: Structural characteristics ( n = 13), Networks and communication ( n = 29), Engaging ( n = 1), Opinion leaders ( n = 5), Champions ( n = 5), Planning ( n = 5), and Reflecting and evaluating ( n = 5). No quantitative assessment measures of Formally appointed implementation leaders, External change agents, or Executing were identified. Internal consistency and norms were reported on most often, whereas no studies reported on discriminant validity or responsiveness. Not one measure in the sample reported all nine psychometric properties evaluated by the PAPERS. Scores in the identified sample of measures ranged from “-2” to “10” out of a total of “36.” Conclusions: Overall measures demonstrated minimal to adequate evidence and available psychometric information was limited. The majority were study specific, limiting their generalizability. Future work should focus on more rigorous measure development and testing of currently existing measures, while moving away from creating new, single use measures. Plain Language Summary: How we measure the processes and players involved for implementing evidence-based interventions is crucial to understanding what factors are helping or hurting the intervention’s use in practice and how to take the intervention to scale. Unfortunately, measures of these factors—stakeholders, their networks and communication, and their implementation activities—have received little attention. This study sought to identify and evaluate the quality of these types of measures. Our review focused on collecting measures used for identifying influential staff members, known as opinion leaders and champions, and investigating how they plan, execute, engage, and evaluate the hard work of implementation. Upon identifying these measures, we collected all published information about their uses to evaluate the quality of their evidence with respect to their ability to produce consistent results across items within each use (i.e., reliable) and if they assess what they are intending to measure (i.e., valid). Our searches located over 40 measures deployed in behavioral health settings for evaluation. We observed a dearth of evidence for reliability and validity and when evidence existed the quality was low. These findings tell us that more measurement work is needed to better understand how to optimize players and processes for the purposes of successful implementation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Systematic reviews of implementation context measures have been conducted, all of which have found significant gaps and signal the continued need for measure validation and development [89–95].”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [95]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 6×
“The Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) comprises 37 constructs believed to predict, moderate, or mediate implementation factors.”
BackgroundMethodInfluentialSelf-citationValente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 13,723
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 1×
“…defined by the Exploration, Adoption/Preparation, Implementation, Sustainment (EPIS) model (Aarons et al., 2011); and (9) if any implementation outcomes were assessed (i.e., acceptability, appropriateness, adoption, cost, feasibility, fidelity, penetration, sustainability; (Proctor et al., 2011).”
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · cited 1×
“The CFIR was developed based on a review of 500 published sources across 13 research areas (Greenhalgh et al., 2004) and 18 additional sources to centralize and unify this myriad of frameworks, create consistent language, and minimize overlap and redundancies.”
MethodMaking sense of implementation theories, models and frameworks
Nilsen, P. (2015) · Implementation Science · doi:10.1186/s13012-015-0242-0 · Cited by 4,447
Scale Development: Theory and Applications.
Xie, Y., & DeVellis, R. F. (1992) · Contemporary Sociology · doi:10.2307/2075704 · Cited by 3,635
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 2×
“Descriptive information was collected on: (1) whether the construct was defined; (2) number of uses; (3) number of items; (4) country of origin; (5) setting administered; (6) level of analysis; (7) population assessed; (8) stage of implementation defined by the Exploration, Adoption/Preparation, Implementation, Sustai…”
BackgroundTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978 · cited 2×
“…approach was adopted to offer a less negative assessment of each property by considering all empirical uses instead of the commonly used “worst score counts” approach (Lewis et al., 2015; Terwee et al., 2012) where only the one lowest performing article was considered in cases of multiple ratings.”
MethodCurrent Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application
Cook, D. A., & Beckman, T. J. (2006) · The American Journal of Medicine · doi:10.1016/j.amjmed.2005.10.036 · Cited by 1,745 · cited 2×
“Afterall, “science is measurement” (Siegel, 1964) and our confidence in knowledge gained depends on how reliable (i.e., consistency across administrations) and valid (i.e., quality of inferences or claims) measures are (D. A. Cook & Beckman, 2006).”
BackgroundAarons, G. A. (2004) · Mental Health Services Research · doi:10.1023/b:mhsr.0000024351.12294.65 · Cited by 1,272 · cited 2×
“One such measure, the Evidence-based Practice Attitudes Scale (EBPAS; (Aarons, 2004), has established national norms (Aarons et al.”
BackgroundLocal opinion leaders: effects on professional practice and health care outcomes
Flodgren, G., Parmelli, E., Doumit, G., Gattellari, M., O'Brien, M. A., Grimshaw, J., & Eccles, M. P. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000125.pub4 · Cited by 1,070
Social Networks and Health: Models, Methods, and Applications
Valente, T. W. (2010) · doi:10.1093/acprof:oso/9780195301014.001.0001 · Cited by 933
Assessing organizational readiness for change
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00233-7 · Cited by 863 · cited 2×
“The Organizational Readiness for Change (ORC) Communication scale (Lehman et al., 2002) reported the most psychometric properties: internal consistency (“3—good”), convergent validity (“2—adequate”), known-groups validity (“2—adequate”), predictive validity (“1—minimal/emerging”), and norms…”
BackgroundChaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-22 · Cited by 800
Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008) · Medical Care Research and Review · doi:10.1177/1077558708317802 · Cited by 786 · cited 2×
“…et al., 2012; Hrisos et al., 2009; Ibrahim & Sidani, 2015; King & Byers, 2007; Lewis et al., 2015; T. Scott et al., 2003; Squires et al., 2011; Weiner et al., 2008) recent implementation science and quality improvement (QI) measurement-focused reviews reveal the same general conclusion:…”
BackgroundDamschroder, L. J., & Lowery, J. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-51 · Cited by 734
Self-citationScott, T., Mannion, R., Davies, H., & Marshall, M. (2003) · Health Services Research · doi:10.1111/1475-6773.00154 · Cited by 630
Quality collaboratives: lessons from research
Ovretveit, J. (2002) · Quality and Safety in Health Care · doi:10.1136/qhc.11.4.345 · Cited by 568
Helfrich, C. D., Li, Y.-F., Sharp, N. D., & Sales, A. E. (2009) · Implementation Science · doi:10.1186/1748-5908-4-38 · Cited by 564
Inside help: An integrative review of champions in healthcare-related implementation
Miech, E. J., Rattray, N. A., Flanagan, M. E., Damschroder, L., Schmid, A. A., & Damush, T. M. (2018) · SAGE Open Medicine · doi:10.1177/2050312118773261 · Cited by 508 · cited 3×
“Furthermore, no study (n = 199) analyzed in the review developed or validated a standardized measure that could identify champions, determine their effectiveness, or distinguish between the multitude of types (Miech et al., 2018).”
BackgroundInfluentialSelf-citationLewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015) · Implementation Science · doi:10.1186/s13012-015-0342-x · Cited by 430 · cited 4×
“At least 10 (Chaudoir et al., 2013; Chor et al., 2015; Emmons et al., 2012; Hrisos et al., 2009; Ibrahim & Sidani, 2015; King & Byers, 2007; Lewis et al., 2015; T. Scott et al., 2003; Squires et al., 2011; Weiner et al., 2008) recent implementation science and quality improvement (QI)…”
BackgroundMethodInfluentialSelf-citationThe Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364
Aarons, G. A., Research Network on Youth Mental Health, Glisson, C., Hoagwood, K., Kelleher, K., Landsverk, J., & Cafri, G. (2010) · Psychological Assessment · doi:10.1037/a0019188 · Cited by 316
Rycroft-Malone, J., Seers, K., Chandler, J., Hawkes, C. A., Crichton, N., Allen, C., Bullock, I., & Strunin, L. (2013) · Implementation Science · doi:10.1186/1748-5908-8-28 · Cited by 303
Implementing and Validating a Comprehensive Unit-Based Safety Program
Pronovost, P., Weast, B., Rosenstein, B., Sexton, J. B., Holzmueller, C. G., Paine, L., Davis, R., & Rubin, H. R. (2005) · Journal of Patient Safety · doi:10.1097/01209203-200503000-00008 · Cited by 288
Local opinion leaders: effects on professional practice and healthcare outcomes
Flodgren, G., O'Brien, M. A., Parmelli, E., & Grimshaw, J. M. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000125.pub5 · Cited by 259
Howell, J. M., & Higgins, C. A. (1990) · Organizational Dynamics · doi:10.1016/0090-2616(90)90047-s · Cited by 249
Instrumentation issues in implementation science
Martinez, R. G., Lewis, C. C., & Weiner, B. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0118-8 · Cited by 243 · cited 2×
“Measurement–a foundational step in developing a strong empirical infrastructure for any scientific field (D. A. Cook & Beckman, 2006; Martinez et al., 2014; Stichter & Conroy, 2004) is often slower to progress and at times considered secondary.”
BackgroundSelf-citationSustaining Community‐Based Programs for Families: Conceptualization and Measurement*
Mancini, J. A., & Marek, L. I. (2004) · Family Relations · doi:10.1111/j.0197-6664.2004.00040.x · Cited by 199
Valente, T. W., Chou, C. P., & Pentz, M. A. (2007) · American Journal of Public Health · doi:10.2105/ajph.2005.063644 · Cited by 191
Predictors of Community Therapists’ Use of Therapy Techniques in a Large Public Mental Health System
Beidas, R. S., Marcus, S., Aarons, G. A., Hoagwood, K. E., Schoenwald, S., Evans, A. C., Hurford, M. O., Hadley, T., Barg, F. K., Walsh, L. M., et al. (2015) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2014.3736 · Cited by 177
Are there valid proxy measures of clinical behaviour? a systematic review
Hrisos, S., Eccles, M. P., Francis, J. J., Dickinson, H. O., Kaner, E. F., Beyer, F., & Johnston, M. (2009) · Implementation Science · doi:10.1186/1748-5908-4-37 · Cited by 162
Role of Champions in the Implementation of Patient Safety Practice Change
Soo, S., Berta, W., & Baker, G. (2009) · Healthcare Quarterly · doi:10.12927/hcq.2009.20979 · Cited by 162
Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., Glisson, C., Rolls-Reutz, J., & The Research Network on Youth Mental Health (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0147-6 · Cited by 149 · cited 1×
“%) measures could not be evaluated using the PAPERS (Beidas et al., 2015; Lundgren et al., 2011; Ramsay et al., 2016; Schoenwald et al., 2008).”
MethodEffective Implementation of collaborative care for depression: what is needed?
Whitebird, R. R., Solberg, L. I., Jaeckels, N., Pietruszewski, P. B., Hadzic, S., Unützer, J., Ohnsorg, K. A., Rossom, R. C., Beck, A. L., Joslyn, K. E., et al. (2014) · PubMed · Cited by 145
Organizational Readiness for Change and opinions toward treatment innovations
Fuller, B. E., Rieckmann, T. R., Nunes, E. V., Miller, M. J., Arfken, C. L., Edmundson, E., & McCarty, D. W. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.026 · Cited by 137
Balasubramanian, B. A., Cohen, D. J., Davis, M. M., Gunn, R., Dickinson, L. M., Miller, W. L., Crabtree, B. F., & Stange, K. C. (2015) · Implementation Science · doi:10.1186/s13012-015-0219-z · Cited by 136
The TMACT: A New Tool for Measuring Fidelity to Assertive Community Treatment
Monroe-DeVita, M., Teague, G. B., & Moser, L. L. (2011) · Journal of the American Psychiatric Nurses Association · doi:10.1177/1078390310394658 · Cited by 136
Rye, M., Torres, E. M., Friborg, O., Skre, I., & Aarons, G. A. (2017) · Implementation Science · doi:10.1186/s13012-017-0573-0 · Cited by 133 · cited 1×
“One such measure, the Evidence-based Practice Attitudes Scale (EBPAS; (Aarons, 2004), has established national norms (Aarons et al., 2010) and strong psychometric evidence across a variety of settings and samples (C. R. Cook et al., 2018; Rye et al., 2017).”
BackgroundAn updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118 · cited 3×
“Details on the full PAPERS rating system and criteria anchor descriptions is published elsewhere (Lewis et al., 2018).”
BackgroundMethodInfluentialSelf-citationDamschroder, L. J., Reardon, C. M., Sperber, N., Robinson, C. H., Fickel, J. J., & Oddone, E. Z. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0424-6 · Cited by 117 · cited 2×
“(Damschroder & Lowery, 2013) and Telephone Lifestyle Coaching (TLC; Damschroder et al., 2017) programs revealed the Networks and communications construct to have strong, positive impacts on implementation outcomes.”
BackgroundMethodSelf-citationStanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117
Self-citationClinton-McHarg, T., Yoong, S. L., Tzelepis, F., Regan, T., Fielding, A., Skelton, E., Kingsland, M., Ooi, J. Y., & Wolfenden, L. (2016) · Implementation Science · doi:10.1186/s13012-016-0512-5 · Cited by 116 · cited 2×
“When comparing to another CFIR-guided measure review evaluating six properties (Clinton-McHarg et al., 2016), only three (i.”
MethodPalinkas, L. A., Holloway, I. W., Rice, E., Fuentes, D., Wu, Q., & Chamberlain, P. (2011) · Implementation Science · doi:10.1186/1748-5908-6-113 · Cited by 113 · cited 2×
“%) measures were not suitable for rating (Feinberg et al., 2005; Hanbury, 2013; Johnson et al., 2015; Li et al., 2012; Palinkas et al., 2011).”
BackgroundWood, K., Giannopoulos, V., Louie, E., Baillie, A., Guajardo, M. G. U., Lee, K. S., Haber, P., & Morley, K. C. (2020) · Implementation Research and Practice · doi:10.1177/2633489520959072 · Cited by 112
Social Networks and Community Prevention Coalitions
Feinberg, M. E., Riggs, N. R., & Greenberg, M. T. (2005) · The Journal of Primary Prevention · doi:10.1007/s10935-005-5390-4 · Cited by 111
Program structure, staff perceptions, and client engagement in treatment
Broome, K. M., Flynn, P. M., Knight, D. K., & Simpson, D. D. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.030 · Cited by 109
Teams communicating through STEPPS
Stead, K., Kumar, S., Schultz, T. J., Tiver, S., Pirone, C. J., Adams, R. J., & Wareham, C. A. (2009) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2009.tb02619.x · Cited by 104
Squires, J. E., Estabrooks, C. A., O'Rourke, H. M., Gustavsson, P., Newburn-Cook, C. V., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-83 · Cited by 101 · cited 2×
“At least 10 (Chaudoir et al., 2013; Chor et al., 2015; Emmons et al., 2012; Hrisos et al., 2009; Ibrahim & Sidani, 2015; King & Byers, 2007; Lewis et al., 2015; T. Scott et al., 2003; Squires et al., 2011; Weiner et al., 2008) recent implementation science and quality improvement (QI) measurement-focused reviews revea…”
BackgroundMeasures for Predictors of Innovation Adoption
Chor, K. H. B., Wisdom, J. P., Olin, S.-C. S., Hoagwood, K. E., & Horwitz, S. M. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0551-7 · Cited by 85
Microsystems in Health Care: Part 9. Developing Small Clinical Units to Attain Peak Performance
Batalden, P. B., Nelson, E. C., Edwards, W. H., Godfrey, M. M., & Mohr, J. J. (2003) · The Joint Commission Journal on Quality and Safety · doi:10.1016/s1549-3741(03)29068-7 · Cited by 80
A Statewide Common Elements Initiative for Children’s Mental Health
Dorsey, S., Berliner, L., Lyon, A. R., Pullmann, M. D., & Murray, L. K. (2016) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-014-9430-y · Cited by 79
Implementing Quality Improvement in Hospitals: The Role of Leadership and Culture
Parker, V. A., Wubbenhorst, W. H., Young, G. J., Desai, K. R., & Charns, M. P. (1999) · American Journal of Medical Quality · doi:10.1177/106286069901400109 · Cited by 77
Systems Antecedents for Dissemination and Implementation: A Review and Analysis of Measures
Emmons, K. M., Weiner, B., Fernandez, M. E., & Tu, S.-P. (2012) · Health Education & Behavior · doi:10.1177/1090198111409748 · Cited by 72 · cited 2×
“At least 10 (Chaudoir et al., 2013; Chor et al., 2015; Emmons et al., 2012; Hrisos et al., 2009; Ibrahim & Sidani, 2015; King & Byers, 2007; Lewis et al., 2015; T. Scott et al., 2003; Squires et al., 2011; Weiner et al., 2008) recent implementation science and quality improvement (QI)…”
BackgroundSmolders, M., Laurant, M., Verhaak, P., Prins, M., van Marwijk, H., Penninx, B., Wensing, M., & Grol, R. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181ca27f6 · Cited by 68
Chang, E. T., Rose, D. E., Yano, E. M., Wells, K. B., Metzger, M. E., Post, E. P., Lee, M. L., & Rubenstein, L. V. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2217-z · Cited by 62
Staff Perception of Barriers to Behavior Therapy at a Psychiatric Hospital
Corrigan, P. W., Kwartarini, W. Y., & Pramana, W. (1992) · Behavior Modification · doi:10.1177/01454455920161007 · Cited by 61
Measurement of a model of implementation for health care: toward a testable theory
Cook, J. M., O’Donnell, C., Dinnen, S., Coyne, J. C., Ruzek, J. I., & Schnurr, P. P. (2012) · Implementation Science · doi:10.1186/1748-5908-7-59 · Cited by 59
Bradshaw, C. P., Debnam, K., Koth, C. W., & Leaf, P. (2009) · Journal of Positive Behavior Interventions · doi:10.1177/1098300708319126 · Cited by 55
Scott, V. C., Kenworthy, T., Godly-Reynolds, E., Bastien, G., Scaccia, J., McMickens, C., Rachel, S., Cooper, S., Wrenn, G., & Wandersman, A. (2017) · American Journal of Orthopsychiatry · doi:10.1037/ort0000270 · Cited by 49
Characteristics of Opinion Leaders in Substance Abuse Treatment Agencies
Moore, K. A., Peters, R. H., Hills, H. A., LeVasseur, J. B., Rich, A. R., Hunt, W. M., Young, M. S., & Valente, T. W. (2004) · The American Journal of Drug and Alcohol Abuse · doi:10.1081/ada-120029873 · Cited by 46
Dückers, M. L., Wagner, C., & Groenewegen, P. P. (2008) · BMC Health Services Research · doi:10.1186/1472-6963-8-172 · Cited by 46
Li, J., Weeks, M. R., Borgatti, S. P., Clair, S., & Dickson‐Gómez, J. (2012) · Substance Use & Misuse · doi:10.3109/10826084.2012.644097 · Cited by 46
Innovative culture in long-term care settings: The influence of organizational characteristics
Nieboer, A. P., & Strating, M. M. H. (2012) · Health Care Management Review · doi:10.1097/hmr.0b013e318222416b · Cited by 41
Cook, C. R., Davis, C. J., Brown, E. C., Locke, J. J., Ehrhart, M. G., Aarons, G. A., Larson, M. F., & Lyon, A. R. (2018) · Implementation Science · doi:10.1186/s13012-018-0804-z · Cited by 41
Supporting a Psychiatric Hospital Culture of Safety
Mahoney, J. S., Ellis, T. E., Garland, G., Palyo, N., & Greene, P. K. (2012) · Journal of the American Psychiatric Nurses Association · doi:10.1177/1078390312460577 · Cited by 36
Fidelity of Intervention Implementation: A Review of Instruments
Ibrahim, S., & Sidani, S. (2015) · Health · doi:10.4236/health.2015.712183 · Cited by 36
Decision and Choice: Contributions of Sidney Siegel
Edwards, W., Siegel, S., Messick, S., & Brayfield, A. H. (1967) · Journal of the American Statistical Association · doi:10.2307/2282933 · Cited by 34
Bond, G. R., Drake, R. E., Rapp, C. A., McHugo, G. J., & Xie, H. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-009-0226-y · Cited by 34
Hanna, J. E., Kubiak, S. P., Pasman, E., Gaba, A., Andre, M. A., Smelson, D. A., & Pinals, D. A. (2019) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2019.06.012 · Cited by 33
Advancing implementation frameworks with a mixed methods case study in child behavioral health
Barwick, M., Barac, R., Kimber, M., Akrong, L., Johnson, S. N., Cunningham, C. E., Bennett, K., Ashbourne, G., & Godden, T. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz005 · Cited by 33 · cited 1×
“…research reveals how changes in group differentiation negatively impact implementation, specifically how merging organizations, addition and/or elimination of roles, and shift changes create isolation from support structures and increase pressures from new responsibilities (Barwick et al., 2020).”
BackgroundAdvancing the Science of Implementation across the Cancer Cancer Continuum
(2018) · Oxford University Press eBooks · doi:10.1093/med/9780190647421.001.0001 · Cited by 32
McGovern, M. P., Urada, D. I., Lambert-Harris, C. A., Sullivan, S. T., & Mazade, N. A. (2012) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.08.013 · Cited by 29
Is organizational progress in the EFQM model related to employee satisfaction?
Baraibar, C. M., Arcelay, A., Cantero-González, D., Colina-Alonso, A., García-Urbaneja, M., Llinares, R. M. G., Letona, J., Martínez-Carazo, C., del Pino, M. M., Nuño-Solínis, R., et al. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-468 · Cited by 28
Implementing cognitive therapies into routine psychosis care: organisational foundations
Dark, F. L., Whiteford, H., Ashkanasy, N. M., Harvey, C. A., Crompton, D., & Newman, E. (2015) · BMC Health Services Research · doi:10.1186/s12913-015-0953-6 · Cited by 27
Ravitz, P., Cooke, R. G., Mitchell, S., Reeves, S., Teshima, J., Lokuge, B., Lawson, A., McNaughton, N., Skinner, W., Cooper, C., et al. (2013) · The Canadian Journal of Psychiatry · doi:10.1177/070674371305800605 · Cited by 27
Advancing Implementation Science Measurement
Lewis, C. C., & Dorsey, C. N. (2020) · Springer eBooks · doi:10.1007/978-3-030-03874-8_9 · Cited by 27
Self-citationLundgren, L. M., Krull, I., Zerden, L. D. S., & McCarty, D. W. (2011) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2011.02.002 · Cited by 26
Bergmark, M., Bejerholm, U., Svensson, B., & Markström, U. (2018) · International Journal of Integrated Care · doi:10.5334/ijic.3547 · Cited by 23
A Review of Organizational Culture Instruments for Nurse Executives
King, T., & Byers, J. F. (2006) · JONA The Journal of Nursing Administration · doi:10.1097/00005110-200701000-00005 · Cited by 22
Implementation of Integrated Dual Disorders Treatment in Eight California Programs
Chandler, D. (2009) · American Journal of Psychiatric Rehabilitation · doi:10.1080/15487760903248473 · Cited by 21
Boothroyd, R. A., Greenbaum, P. E., Wang, W., Kutash, K., & Friedman, R. M. (2011) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-011-9239-x · Cited by 16
Guerrero, E. G., Heslin, K. C., Chang, E. T., Fenwick, K. M., & Yano, E. M. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0582-0 · Cited by 13
The National Criminal Justice Treatment Practices survey: An overview of the special issue
Taxman, F. S., Young, D. W., & Fletcher, B. W. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.017 · Cited by 13
A Longitudinal Study of Medical Practices’ Treatment of Patients Who Use Tobacco
Ramsay, P. P., Shortell, S. M., Casalino, L. P., Rodríguez, H. P., & Rittenhouse, D. R. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2015.07.005 · Cited by 12
Sheehan, J. J., Griffiths, K. M., Rickwood, D., & Carron‐Arthur, B. (2014) · Crisis · doi:10.1027/0227-5910/a000295 · Cited by 12
Farley, K., Hanbury, A., & Thompson, C. A. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-38 · Cited by 10
Stichter, J. P., & Conroy, M. A. (2004) · Behavioral Disorders · doi:10.1177/019874290403000106 · Cited by 9
Hanbury, A. (2013) · Psychology Health & Medicine · doi:10.1080/13548506.2013.764459 · Cited by 4
Johnson, K., Quanbeck, A., Maus, A., Gustafson, D. H., & Dearing, J. W. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0327-y · Cited by 3
(1998) · doi:10.1002/(sici)1099-1379(199805)19:3<235::aid-job837>3.0.co;2-c
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
Self-citationRobinson, C., & Damschroder, L. J. (2023) · Implementation Science Communications · doi:10.1186/s43058-022-00380-5 · Cited by 60
Self-citationLeveraging implementation science to advance antibiotic stewardship practice and research
Livorsi, D. J., Drainoni, M., Reisinger, H. S., Nanda, N., McGregor, J. C., Barlam, T. F., Morris, A. M., & Szymczak, J. E. (2021) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2021.480 · Cited by 50
Madrigal, L., Manders, O. C., Kegler, M. C., Haardörfer, R., Piper, S., Blais, L. M., Weber, M. B., & Escoffery, C. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00350-x · Cited by 34
Pilar, M. R., Jost, E., Walsh‐Bailey, C., Powell, B. J., Mazzucca, S., Eyler, A. A., Purtle, J., Allen, P. M., & Brownson, R. C. (2022) · Implementation Research and Practice · doi:10.1177/26334895221141116 · Cited by 30
Metadata
- Authors
- Dorsey, C. N., Mettert, K. D., Puspitasari, A. J., Damschroder, L. J., & Lewis, C. C.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- Mayo Clinic (US)
- VA Ann Arbor Healthcare System (US)
- Corresponding author
- Dorsey, Caitlin N
- Venue
- Implementation Research and Practice
- Publisher
- SAGE Publications
- Published
- 2021-01
- Volume
- 2
- Pages
- 26334895211002474-26334895211002474
- ISSN
- 2633-4895 (ISSN-L)
- PMC
- PMC9978628
- OpenAlex
- W3144193208
- Cited by
- 15
- References
- 91
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 1 · 2023: 1 · 2022: 5 · 2021: 1
- FWCI
- 0.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Research and Practice (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- implementation research, Consolidated Framework for Implementation Research, psychometric properties, opinion leaders, implementation champions, stakeholder networks, social networks, implementation leadership, implementation planning, implementation evaluation, behavioral health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.