Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties
Open accessThe DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index.
Abstract
Background: Systematic reviews of measures can facilitate advances in implementation research and practice by locating reliable and valid measures and highlighting measurement gaps. Our team completed a systematic review of implementation outcome measures published in 2015 that indicated a severe measurement gap in the field. Now, we offer an update with this enhanced systematic review to identify and evaluate the psychometric properties of measures of eight implementation outcomes used in behavioral health care. Methods: The systematic review methodology is described in detail in a previously published protocol paper and summarized here. The review proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and measure forward searches. Phase II, data extraction, involved coding psychometric information. Phase III, data analysis, involved two trained specialists independently rating each measure using PAPERS (Psychometric And Pragmatic Evidence Rating Scales). Results: Searches identified 150 outcomes measures of which 48 were deemed unsuitable for rating and thus excluded, leaving 102 measures for review. We identified measures of acceptability ( N = 32), adoption ( N = 26), appropriateness ( N = 6), cost ( N = 31), feasibility ( N = 18), fidelity ( N = 18), penetration ( N = 23), and sustainability ( N = 14). Information about internal consistency and norms were available for most measures (59%). Information about other psychometric properties was often not available. Ratings for internal consistency and norms ranged from “adequate” to “excellent.” Ratings for other psychometric properties ranged mostly from “poor” to “good.” Conclusion: While measures of implementation outcomes used in behavioral health care (including mental health, substance use, and other addictive behaviors) are unevenly distributed and exhibit mostly unknown psychometric quality, the data reported in this article show an overall improvement in availability of psychometric information. This review identified a few promising measures, but targeted efforts are needed to systematically develop and test measures that are useful for both research and practice. Plain language abstract: When implementing an evidence-based treatment into practice, it is important to assess several outcomes to gauge how effectively it is being implemented. Outcomes such as acceptability, feasibility, and appropriateness may offer insight into why providers do not adopt a new treatment. Similarly, outcomes such as fidelity and penetration may provide important context for why a new treatment did not achieve desired effects. It is important that methods to measure these outcomes are accurate and consistent. Without accurate and consistent measurement, high-quality evaluations cannot be conducted. This systematic review of published studies sought to identify questionnaires (referred to as measures) that ask staff at various levels (e.g., providers, supervisors) questions related to implementation outcomes, and to evaluate the quality of these measures. We identified 150 measures and rated the quality of their evidence with the goal of recommending the best measures for future use. Our findings suggest that a great deal of work is needed to generate evidence for existing measures or build new measures to achieve confidence in our implementation evaluations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We now have systematic reviews of implementation outcomes [15] and their measurement properties in behavioral health [16], public policy [17], stroke care [7], and physical healthcare [18].”
From Ten years of implementation outcomes research: a scoping review (2023) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, Mettert and colleagues reported that acceptability had the greatest number of measurement options [15].”
From Ten years of implementation outcomes research: a scoping review (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“This systematic review is part of a larger project to identify measures of all implementation constructs associated with the Consolidated Framework for Implementation Research Constructs (CFIR) (Damschroder et al., 2009), which are included in this special section.”
BackgroundMoore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150 · cited 1×
“Six measures of appropriateness were identified, of which one was a scale within a broader measure (i.e., Moore & Benbasat Adoption of IT Innovation Measure— Compatibility scale) (Moore & Benbasat, 1991).”
BackgroundProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 3×
“Trained research specialists (C.D., K.M.) mapped measures and/or their scales to one or more of the eight aforementioned implementation outcomes (Proctor et al., 2011).”
BackgroundMethodMotivational interviewing: preparing people for change.
Miller, W. R., & Rollnick, S. (2002) · Guilford Press eBooks · Cited by 7,768
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217 · cited 1×
“While many other implementation frameworks exist (Glasgow et al., 1999), the outcomes framework developed by Proctor and colleagues provides clear differentiation of implementation outcomes from clinical and services outcomes and offers a harmonized focus for the field.”
BackgroundAdvancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 2×
“(1) country of origin, (2) concept defined by authors, (3) number of articles contained in each measure packet, (4) number of scales, (5) number of items, (6) setting in which measure had been used, (7) level of analysis, (8) target problem, and (9) stage of implementation as defined by the Exploration, Adoption/Prepa…”
BackgroundMethodProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · cited 1×
“Proctor and colleagues’ seminal work (Proctor et al., 2009, 2011) articulated at least eight implementation outcomes for the field: (1) acceptability, (2) adoption, (3) appropriateness, (4) cost, (5) feasibility, (6) fidelity, (7) penetration, and (8) sustainability (Table 1) (Lewis et al., 2015).”
Development of an instrument measuring user satisfaction of the human-computer interface
Chin, J. P., Diehl, V. A., & Norman, L. K. (1988) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '88 · doi:10.1145/57167.57203 · Cited by 1,609
Journal of General Internal Medicine
(2013) · Journal of General Internal Medicine · doi:10.1007/11606.1525-1497 · Cited by 1,283
Aarons, G. A. (2004) · Mental Health Services Research · doi:10.1023/b:mhsr.0000024351.12294.65 · Cited by 1,272
Further evaluation of the Dutch supervision system for suicides of mental health care users
(2013) · EUR Research Repository (Erasmus University Rotterdam) · doi:10.1176/appi · Cited by 1,030
Assessing organizational readiness for change
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00233-7 · Cited by 863 · cited 1×
“For example, the Texas Christian University Organizational Readiness for Change Scale contains 19 scales measuring constructs such as motivation for change, available resources, staff attributes, and organizational climate (Lehman et al., 2002).”
MethodA conceptual framework for transferring research to practice
Simpson, D. D. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00231-3 · Cited by 538
Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.75.6.829 · Cited by 533
A national survey of practicing psychologists' attitudes toward psychotherapy treatment manuals
Addis, M. E., & Krasnow, A. D. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.68.2.331 · Cited by 509
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015) · Implementation Science · doi:10.1186/s13012-014-0192-y · Cited by 435
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015) · Implementation Science · doi:10.1186/s13012-015-0342-x · Cited by 430 · cited 4×
“Our systematic review and psychometric assessment revealed that evidence of measures’ reliability and validity is largely unknown or poor (Lewis et al., 2015).”
BackgroundMethodInfluentialSelf-citationCarroll, K. (2000) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(99)00049-6 · Cited by 386 · cited 2×
“…health care (psychometric total maximum score = 9; maximum possible score = 36), with ratings of “4—excellent” for convergent validity, “1—minimal/emerging” for discriminant validity, “1—minimal/emerging” for concurrent validity, and “3— good” for structural validity (Carroll et al., 2000).”
BackgroundMethodBridging the Research-to-Practice Gap
Carnine, D. (1997) · Exceptional Children · doi:10.1177/001440299706300406 · Cited by 384
The Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364
Elliott, S. N., & Treuting, M. V. B. (1991) · Journal of School Psychology · doi:10.1016/0022-4405(91)90014-i · Cited by 328
Reimers, T. M., & Wacker, D. P. (1988) · Behavioral Disorders · doi:10.1177/019874298801400104 · Cited by 295
Kazdin, A. E., Bass, D., Siegel, T., & Thomas, C. (1989) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.57.4.522 · Cited by 282
Outcomes of a Partnered Facilitation Strategy to Implement Primary Care–Mental Health
Kirchner, J. E., Ritchie, M. J., Pitcock, J. A., Parker, L. E., Curran, G. M., & Fortney, J. C. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3027-2 · Cited by 255 · cited 3×
“Finally, 14 measures of sustainability were identified, one of which was deemed unsuitable for rating (i.e., Kirchner Sustainability Measure) (Kirchner et al., 2014), and another was a scale within a larger measure (i.e., Eisen Provider Knowledge & Attitudes Survey— Sustainability scale) (Eisen et…”
BackgroundInfluentialThe Role of Staff Turnover in the Implementation of Evidence-Based Practices in Mental Health Care
Woltmann, E. M., Whitley, R., McHugo, G. J., Brunette, M., Torrey, W. C., Coots, L., Lynde, D., & Drake, R. E. (2008) · Psychiatric Services · doi:10.1176/ps.2008.59.7.732 · Cited by 246
Instrumentation issues in implementation science
Martinez, R. G., Lewis, C. C., & Weiner, B. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0118-8 · Cited by 243 · cited 1×
“The development of “in-house” measures used only once for a specific study contributes to the proliferation of measures that have limited evidence of reliability and validity (Martinez et al., 2014).”
BackgroundSelf-citationSustaining Community‐Based Programs for Families: Conceptualization and Measurement*
Mancini, J. A., & Marek, L. I. (2004) · Family Relations · doi:10.1111/j.0197-6664.2004.00040.x · Cited by 199
The Sustainability of Evidence-Based Practices in Routine Mental Health Agencies
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2010) · Community Mental Health Journal · doi:10.1007/s10597-009-9202-y · Cited by 193
Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002) · Journal of Clinical Child and Adolescent Psychology · doi:10.1207/153744202753604458 · Cited by 190
Development of Level of Institutionalization Scales for Health Promotion Programs
Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. (1993) · Health Education Quarterly · doi:10.1177/109019819302000208 · Cited by 181
Assessing treatment acceptance: The abbreviated acceptability rating profile
Tarnowski, K. J., & Simonian, S. J. (1992) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(92)90007-6 · Cited by 167
Measuring perceptions of innovation adoption: the diffusion of a federal drug prevention policy
Pankratz, M. (2002) · Health Education Research · doi:10.1093/her/17.3.315 · Cited by 139
Balasubramanian, B. A., Cohen, D. J., Davis, M. M., Gunn, R., Dickinson, L. M., Miller, W. L., Crabtree, B. F., & Stange, K. C. (2015) · Implementation Science · doi:10.1186/s13012-015-0219-z · Cited by 136
An updated protocol for a systematic review of implementation-related measures
Lewis, C. C., Mettert, K. D., Dorsey, C. N., Martinez, R. G., Weiner, B. J., Nolen, E., Stanick, C., Halko, H., & Powell, B. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0728-3 · Cited by 118 · cited 6×
“Psychometric ratings using the Psychometric and Pragmatic Evaluation Rating Scale (PAPERS; Lewis et al., 2018) ranged from −1 to 14 with a possible minimum of score of −9 and a possible maximum score of 36, illustrating a profound need for measurement studies in this field.”
BackgroundMethodInfluentialSelf-citationStanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117 · cited 1×
“The full rating system and criteria for the PAPERS are published elsewhere (Lewis et al., 2018; Stanick et al., 2019).”
MethodSelf-citationMilne, D. L. (2009) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.657 · Cited by 112
Cook, J. M., Schnurr, P. P., Biyanova, T., & Coyne, J. C. (2009) · Psychiatric Services · doi:10.1176/appi.ps.60.5.671 · Cited by 110
Glisson, C. A., Hemmelgarn, A. L., Green, P., & Williams, N. J. (2013) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2013.02.005 · Cited by 109
Fortney, J., Enderle, M., McDougall, S., Clothier, J., Otero, J., Altman, L., & Curran, G. (2012) · Implementation Science · doi:10.1186/1748-5908-7-30 · Cited by 106 · cited 1×
“As mentioned previously, measures were considered “unsuitable for rating” if the format of construct assessment did not produce psychometric information or format of the measure did not conform to the rating scale (e.g., Fortney Measure of Adoption Rate) (Fortney et al., 2012).”
MethodPOWER, T. J., HESS, L. E., & BENNETT, D. S. (1995) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/00004703-199508000-00005 · Cited by 96
Chagnon, F., Pouliot, L., Malo, C., Gervais, M.-J., & Pigeon, M.-È. (2010) · Implementation Science · doi:10.1186/1748-5908-5-41 · Cited by 95
Carper, M. M., McHugh, R. K., & Barlow, D. H. (2013) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0377-5 · Cited by 92 · cited 2×
“Twenty-six measures of adoption were identified, three of which were scales part of broader measures (e.g., Perceptions of Computerized Therapy Questionnaire— Future Use Intentions scale) (Carper et al., 2013) and two of which were deemed “unsuitable for rating.””
BackgroundWhittingham, K., Sofronoff, K., & Sheffield, J. K. (2005) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2005.05.003 · Cited by 92
Tanol, G., Johnson, L., McComas, J. J., & Cote, E. (2010) · Journal of School Psychology · doi:10.1016/j.jsp.2010.06.001 · Cited by 91
Solberg, L. I., Asche, S. E., Margolis, K. L., & Whitebird, R. R. (2008) · American Journal of Medical Quality · doi:10.1177/1062860608314942 · Cited by 90
Bird, V. J., Boutillier, C. L., Leamy, M., Williams, J., Bradstreet, S., & Slade, M. (2014) · British Journal of Psychiatry · doi:10.1192/bjp.bp.113.128314 · Cited by 90
Himelhoch, S., Riddle, J., & Goldman, H. H. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201200247 · Cited by 89
Williams, N. J., Glisson, C., Hemmelgarn, A., & Green, P. (2017) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0742-5 · Cited by 86 · cited 2×
“…possible score = 36), with ratings of “3—good” and 2—adequate” for internal consistency, “3—good” and “0—no evidence” for convergent validity, “0—no evidence” and “3—good” for predictive validity,, and “3—good” and “4—”excellent” for norms, respectively (Ruzek et al., 2016; Williams et al., 2017).”
Cohen, D. J., Balasubramanian, B. A., Davis, M., Hall, J., Gunn, R., Stange, K. C., Green, L. A., Miller, W. L., Crabtree, B. F., England, M. J., et al. (2015) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2015.s1.150050 · Cited by 75
Guerrero, E. G., Padwa, H., Fenwick, K. M., Harris, L. M., & Aarons, G. A. (2015) · Implementation Science · doi:10.1186/s13012-016-0438-y · Cited by 75
Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2015) · Implementation Science · doi:10.1186/s13012-015-0255-8 · Cited by 69
Johnston, C., & Fine, S. (1993) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/18.6.717 · Cited by 67
Adolescent Suicide Prevention: School Psychologists' Acceptability of School-Based Programs
Eckert, T. L., Miller, D. N., DuPaul, G. J., & Riley-Tillman, T. C. (2003) · School Psychology Review · doi:10.1080/02796015.2003.12086183 · Cited by 67
Azocar, F., Cuffel, B., Goldman, W., & McCarter, L. (2003) · The Journal of Behavioral Health Services & Research · doi:10.1007/bf02287816 · Cited by 66
Integrating technology into cognitive behavior therapy for adolescent depression: a pilot study
Kobak, K. A., Mundt, J. C., & Kennard, B. (2015) · Annals of General Psychiatry · doi:10.1186/s12991-015-0077-8 · Cited by 63
Elliott, S. N., Witt, J. C., Galvin, G. A., & Moe, G. L. (1986) · Professional Psychology Research and Practice · doi:10.1037/0735-7028.17.3.235 · Cited by 61
Kolko, D. J., Baumann, B. L., Herschell, A. D., Hart, J. A., Holden, E. A., & Wisniewski, S. R. (2012) · Child Maltreatment · doi:10.1177/1077559511427346 · Cited by 60
Costs to Community Mental Health Agencies to Sustain an Evidence-Based Practice
Roundfield, K. D., & Lang, J. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600193 · Cited by 60
Adoption of Evidence-Based Practices among Substance Abuse Treatment Providers
Haug, N. A., Shopshire, M., Tajima, B., Gruber, V., & Guydish, J. (2008) · Journal of Drug Education · doi:10.2190/de.38.2.f · Cited by 59 · cited 2×
“Thirty-two measures of acceptability were identified, one of which was a specific scale within a broader measure (i.e., SFTRC Course Evaluation—Attitude scale) (Haug et al., 2008).”
BackgroundBriesch, A. M., & Chafouleas, S. M. (2009) · Journal of Educational and Psychological Consultation · doi:10.1080/10474410903408885 · Cited by 57 · cited 1×
“…had the highest psychometric rating score among measures of feasibility used in mental and behavioral health care (psychometric total maximum score = 6; maximum possible score = 36), with ratings of “4—excellent” for internal consistency and “2—adequate” for norms (Briesch & Chafouleas, 2009).”
A UK validation of the Stages of Recovery Instrument
Weeks, G., Slade, M., & Hayward, M. (2010) · International Journal of Social Psychiatry · doi:10.1177/0020764010365414 · Cited by 57
McIntosh, K., MacKay, L. D., Hume, A. E., Doolittle, J., Vincent, C. G., Horner, R. H., & Ervin, R. A. (2011) · Journal of Positive Behavior Interventions · doi:10.1177/1098300710385348 · Cited by 54 · cited 2×
“The School-wide Universal Behavior Sustainability Index-school Teams had the highest psychometric rating score among measures of penetration used in mental and behavioral health care (psychometric total maximum score = 12; maximum possible score = 36), with ratings of “4—excellent,” for internal consistency and “3—goo…”
BackgroundAcceptability of behavioral staff management techniques
Davis, J. R., Rawana, E. P., & Capponi, D. R. (1989) · Behavioral Interventions · doi:10.1002/bin.2360040104 · Cited by 52
McLeod, B. D., Smith, M. M., Southam-Gerow, M. A., Weisz, J. R., & Kendall, P. C. (2015) · Psychological Assessment · doi:10.1037/pas0000037 · Cited by 52
Research utilization among children's mental health providers
Barwick, M. A., Boydell, K. M., Stasiulis, E., Ferguson, H. B., Blase, K., & Fixsen, D. (2008) · Implementation Science · doi:10.1186/1748-5908-3-19 · Cited by 50
Dimeff, L. A., Woodcock, E. A., Harned, M. S., & Beadnell, B. A. (2011) · Behavior Therapy · doi:10.1016/j.beth.2010.06.004 · Cited by 49
Developing a Tool to Measure Knowledge Exchange Outcomes
Skinner, K. A. (2007) · Canadian Journal of Program Evaluation · doi:10.3138/cjpe.022.003 · Cited by 48
A Fidelity Measure for Integrated Management of Depression in Primary Care
Oxman, T. E., Schulberg, H. C., Greenberg, R. L., Dietrich, A. J., Williams, J. W., Nutting, P. A., & Bruce, M. L. (2006) · Medical Care · doi:10.1097/01.mlr.0000233683.82254.63 · Cited by 43
Ruzek, J. I., Eftekhari, A., Rosen, C. S., Crowley, J. J., Kuhn, E., Foa, E. B., Hembree, E. A., & Karlin, B. E. (2016) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/tra0000004 · Cited by 42 · cited 2×
“…possible score = 36), with ratings of “3—good” and 2—adequate” for internal consistency, “3—good” and “0—no evidence” for convergent validity, “0—no evidence” and “3—good” for predictive validity,, and “3—good” and “4—”excellent” for norms, respectively (Ruzek et al., 2016; Williams et al., 2017).”
Stamatakis, K. A., McQueen, A., Filler, C., Boland, E., Dreisinger, M., Brownson, R. C., & Luke, D. A. (2012) · Implementation Science · doi:10.1186/1748-5908-7-65 · Cited by 42
Web-Based Training in Early Autism Screening: Results from a Pilot Study
Kobak, K. A., Stone, W. L., Ousley, O., & Swanson, A. R. (2011) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2011.0029 · Cited by 41
Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L. (2013) · Implementation Science · doi:10.1186/1748-5908-8-19 · Cited by 41
How do components of evidence-based psychological treatment cluster in practice?
Gifford, E. V., Tavakoli, S., Weingardt, K. R., Finney, J. W., Pierson, H. M., Rosen, C. S., Hagedorn, H., Cook, J. M., & Curran, G. M. (2011) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2011.07.008 · Cited by 39
Newton, J. T., & Sturmey, P. (2004) · Psychological Reports · doi:10.2466/pr0.94.2.475-481 · Cited by 39
Buckley, L., Goering, P. N., Parikh, S. V., Butterill, D., & Foo, E. K. H. (2003) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2003.00407.x · Cited by 38
Hides, L. M., Lubman, D. I., Elkins, K., Catania, L., & Rogers, N. (2007) · Drug and Alcohol Review · doi:10.1080/09595230701499126 · Cited by 37
Implementation of evidence-based practices: Applying a goal commitment framework
Chou, A. F., Vaughn, T. E., McCoy, K. D., & Doebbeling, B. N. (2011) · Health Care Management Review · doi:10.1097/hmr.0b013e3181dc8233 · Cited by 36
Bond, G. R., Drake, R. E., Rapp, C. A., McHugo, G. J., & Xie, H. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-009-0226-y · Cited by 34
Zwerver, F., Schellart, A. J., Anema, J. R., & van der Beek, A. J. (2012) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-012-9378-9 · Cited by 33
Kelly, P. J., Deane, F. P., & Lovett, M. J. (2012) · Psychology of Addictive Behaviors · doi:10.1037/a0027887 · Cited by 33
Kolla, N. J., Links, P. S., McMain, S., Streiner, D. L., Cardish, R. J., & Cook, M. (2009) · The Canadian Journal of Psychiatry · doi:10.1177/070674370905400306 · Cited by 32
Khadjesari, Z., Vitoratou, S., Sevdalis, N., & Hull, L. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017972 · Cited by 32 · cited 1×
“A coinciding measurement review (Khadjesari et al., 2017) is underway to identify measures of implementation outcomes in physical health care settings.”
BackgroundEisen, J. C., Marko-Holguin, M., Fogel, J., Cardenas, A., Bahn, M., Bradford, N., Fagan, B., Wiedmann, P., & Van Voorhees, B. W. (2013) · The Primary Care Companion For CNS Disorders · doi:10.4088/pcc.10m01065 · Cited by 26
Brothers, B. M., Carpenter, K. M., Shelby, R. A., Thornton, L. M., Frierson, G. M., Patterson, K. L., & Andersen, B. L. (2014) · Translational Behavioral Medicine · doi:10.1007/s13142-014-0273-0 · Cited by 26
Slade, M., Cahill, S., Kelsey, W., Powell, R., Strathdee, G., & Valiakalayil, A. (2001) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s001270170017 · Cited by 24
Beehler, G. P., Funderburk, J. S., King, P. R., Wade, M. J., & Possemato, K. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0325-0 · Cited by 21
Implementing Residential Integrated Treatment for Co-occurring Disorders
McKee, S. A., Harris, G. T., & Cormier, C. A. (2013) · Journal of Dual Diagnosis · doi:10.1080/15504263.2013.807073 · Cited by 20
David, P., & Schiff, M. (2015) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2015.07.012 · Cited by 19
Malte, C. A., McFall, M., Chow, B., Beckham, J. C., Carmody, T. P., & Saxon, A. J. (2013) · Psychology of Addictive Behaviors · doi:10.1037/a0028484 · Cited by 19
Neumiller, S., Bennett-Clark, F., Young, M. S., Dates, B. G., Broner, N., Leddy, J., Kendall, D., Richards, S., & Jong, F. D. (2009) · Journal of Dual Diagnosis · doi:10.1080/15504260903175973 · Cited by 19
Walitzer, K. S., Dermen, K. H., Barrick, C., & Shyhalla, K. (2015) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2015.04.003 · Cited by 18
Yetter, G. (2010) · Journal of Educational and Psychological Consultation · doi:10.1080/10474411003785370 · Cited by 17 · cited 4×
“The Pre-referral Intervention Team Inventory had the highest psychometric rating score among measures of appropriateness used in mental and behavioral health care (psychometric total maximum score = 14; maximum possible score = 36), with ratings of “4—excellent” for internal consistency, “3—good” for convergent validi…”
MethodInfluentialHoving, C., Mudde, A. N., & de Vries, H. (2005) · European Journal of Cancer Prevention · doi:10.1097/01.cej.0000186633.81753.8b · Cited by 16
the eVITAL group, Alonso-Trujillo, F., Walsh, C. O., & Salvador‐Carulla, L. (2010) · BMC Research Notes · doi:10.1186/1756-0500-3-83 · Cited by 16
Ebert, L., Malte, C. A., Hamlett-Berry, K., Beckham, J. C., McFall, M., & Saxon, A. J. (2014) · American Journal of Public Health · doi:10.2105/ajph.2013.301776 · Cited by 15
Chodosh, J., Price, R. M., Cadogan, M. P., Damron‐Rodriguez, J., Osterweil, D., Czerwinski, A., Tan, Z. S., Merkin, S. S., Gans, D., & Frank, J. C. (2015) · Journal of the American Geriatrics Society · doi:10.1111/jgs.13775 · Cited by 14
Lloyd, D., Couineau, A., Hawkins, K., Kartal, D., Nixon, R. D. V., Perry, D., & Forbes, D. (2015) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.14m09139 · Cited by 14
George, S. M. S., Huang, S., Vidot, D. C., Smith, J. D., Brown, C. H., & Prado, G. J. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0344-x · Cited by 13
Amodeo, M., Storti, S. A., & Larson, M. J. (2010) · Substance Use & Misuse · doi:10.3109/10826080903534467 · Cited by 13
Myers, S. J. (2008) · UA Campus Repository (The University of Arizona) · Cited by 12
Henninger, K. L. (2010) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/1558879 · Cited by 11
Chung, H., Duffy, F. F., Katzelnick, D. J., Williams, M. D., Trivedi, M. H., Rae, D. S., & Regier, D. A. (2013) · Psychiatric Services · doi:10.1176/appi.ps.201200227 · Cited by 8
Tobin, M. J., Clarke, A. R., Buss, R., Einfeld, S., Beard, J., Dudley, M. J., Knowles, M., & Dietrich, U. C. (2001) · Australian Health Review · doi:10.1071/ah010143 · Cited by 7
Azocar, F., Cuffel, B., Goldman, W., & McCarter, L. M. (2003) · The Journal of Behavioral Health Services & Research · doi:10.1097/00075484-200301000-00008 · Cited by 5
Roweton, W. E. (1991) · Psychology in the Schools · doi:10.1002/1520-6807(199110)28:4<345::aid-pits2310280410>3.0.co;2-f
Research works printed but not linked: 5
- 31. Easton J. E. (2009). Teacher acceptability of treatment plan implementation monitoring and feedback methods [Unpublished master’s thesis]. North Carolina State University, Raleigh, NC.
- 38. Forchuk C. Hartford K. Blomqvist A. Martin M. Chan L. Donner A. (2002). Therapeutic relationships: From hospital to community: First year results. Ottawa Report to the Canadian Health Services Research Foundation. Lawson Health Research Institute, London, ON.
- 68. Maher L. Gustafson D. Evans A. (2007). Sustainability and model guide. NHS Institute for Innovation and Improvement.
- 78. Milne D. Noone S. (1996). Teaching and training for non-teachers. The British Psychological Society.
- 104. Trent L. R. (2010). Development of a measure of disseminability (MOD). University of Mississippi.
Cited by
Ten years of implementation outcomes research: a scoping review
Proctor, E. K., Bunger, A. C., Lengnick-Hall, R., Gerke, D. R., Martin, J. K., Phillips, R. J., & Swanson, J. C. (2023) · Implementation Science · doi:10.1186/s13012-023-01286-z · Cited by 424
Moullin, J. C., Sklar, M., Ehrhart, M. G., Green, A., & Aarons, G. A. (2021) · Implementation Science · doi:10.1186/s13012-021-01152-w · Cited by 82
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81
“Consequently, many existing measures have only been used once [18], illustrating limited standardisation in measurement.”
BackgroundMethodSelf-citationImplementation of Parenting Programs in Real-World Community Settings: A Scoping Review
Pinto, R., Canário, C., Leijten, P. H. O., López, M. J. R., & Cruz, O. (2023) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-023-00465-0 · Cited by 60
“Furthermore, the framework by Proctor et al. (2011), despite being published a decade ago, continues to be frequently used in the literature (e.g., Khadjesari et al., 2020; Mettert et al., 2020; Xu et al., 2023).”
Psychometric Testing of the German Versions of Three Implementation Outcome Measures
Kien, C., Griebler, U., Schultes, M.-T., Thaler, K. J., & Stamm, T. (2021) · Global Implementation Research and Applications · doi:10.1007/s43477-021-00019-y · Cited by 56
Metadata
- Authors
- Mettert, K., Lewis, C., Dorsey, C., Halko, H., & Weiner, B.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- Harvard University (US)
- University of Washington (US)
- Corresponding author
- Mettert, Kayne
- Venue
- Implementation Research and Practice
- Publisher
- SAGE Publications
- Published
- 2020-01
- Volume
- 1
- Pages
- 2633489520936644-2633489520936644
- ISSN
- 2633-4895 (ISSN-L)
- PMC
- PMC9924262
- OpenAlex
- W3082080819
- Semantic Scholar
- 4f80bbc490a413737ef8eeb7506b6bd739a914a7
- Cited by
- 162
- References
- 104
- Citations by year
- 2026: 24 · 2025: 37 · 2024: 35 · 2023: 36 · 2022: 17 · 2021: 12 · 2020: 1
- FWCI
- 3.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Research and Practice (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (R01MH106510)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- outcome measures, psychometric properties, measurement gaps, behavioral health services, mental health services, addiction treatment, implementation fidelity, acceptability, feasibility, appropriateness, questionnaire quality, implementation evaluation, scale development, internal consistency, norms, construct validity, content validity, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.