Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale
Open accessPublished version of a preprint
Abstract
BACKGROUND: Enhancing the sustainability of evidence-based prevention programs for mental and behavioral health requires tools for measuring both sustainability determinants and sustainment outcomes. The aim of this study was to develop the Sustainment Measurement System Scale (SMSS) and to assess its reliability and construct validity for measuring both determinants and outcomes of efforts to sustain prevention programs and initiatives. METHODS: A 42-item scale comprised of items identified from qualitative data collected from 45 representatives of 10 programs and 8 SAMHSA program officers was administered to 186 representatives of 145 programs funded by 7 SAMHSA prevention grant initiatives. Cronbach's alphas were used to determine inter-item reliability. Convergent validity was assessed by comparisons of a global measure of sustainment with current SAMHSA-funding status and continued operation in the same form. Discriminant validity was assessed by comparisons of sustainability determinants with whether or not the program had undergone adaptations. RESULTS: Confirmatory factor analysis provided support for a 35-item model fit to the data. Cronbach's alpha was .84 for the sustainment outcome construct and ranged from .70 to .93 for the sustainability determinant constructs. All of the determinant constructs were significantly associated with sustainment outcome individual and global measures for the entire sample (p < 0.01 to 0.001) and for community-based programs and programs with a substance abuse focus (p < 0.05 to 0.001). Convergent validity was supported by significant associations between the global sustainment measure and current SAMHSA funding status and continued operation in the same form (p < 0.001). Four of the sustainability determinant constructs (responsive to community needs; coalitions, partnerships, and networks; organizational staff capability; and evaluation, feedback, and program outcomes) were also significantly associated with current SAMHSA funding status (p < 0.5 to 0.01). With the exception of organizational staff capability, all sustainability determinants were unrelated to program adaptation as predicted. CONCLUSIONS: The SMSS demonstrated good reliability and convergent and discriminant validity in assessing likelihood of sustainment of SAMHSA funded prevention programs and initiatives. The measure demonstrates potential in identifying predictors of program sustainment and as a tool for enhancing the likelihood of successful sustainment through ongoing evaluation and feedback.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The PSAT is both pragmatic and generalizable to different settings and amongst those in different roles, including both practitioners and research team members.
“While there have been other measures developed for sustainability capacity in certain areas or to assess sustainment, this capacity focused measure is both pragmatic and generalizable to different settings and amongst those in different roles, including both practitioners and research team members (5).”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [5]
The Background (page unavailable) identifies the PSAT as a tool for assessing public-health programs’ capacity for sustainability. It does not say the PSAT is pragmatic or generalizable across settings and roles; the Conclusion’s discussion of broader applicability concerns the SMSS, not the PSAT. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788 · see citing works
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384 · see citing works
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · see citing works
Confirmatory factor analysis for applied research
Brown, T. A. (2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,613 · see citing works
(1991) · Cited by 15,554 · see citing works
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · see citing works
Psychometric theory (2nd ed.).
Sitgreaves, R. (1979) · Contemporary Psychology · doi:10.1037/018882 · Cited by 14,384 · see citing works
EQS : structural equations program manual
Bentler, P. M. (1989) · Medical Entomology and Zoology · Cited by 9,669 · see citing works
Evaluating the use of exploratory factor analysis in psychological research.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.3.272 · Cited by 9,587 · see citing works
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · see citing works
Audit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485 · cited 1× · see citing works
“While monitoring and feedback are recognized as important for prevention [18, 19], much of the relevant science on feedback in health has involved improvement in clinical performance [20-26].”
BackgroundAdvancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · see citing works
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
A compilation of strategies for implementing clinical innovations in health and mental health
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E., & York, J. L. (2012) · Medical Care Research and Review · doi:10.1177/1077558711430690 · Cited by 1,069 · see citing works
Russell, D. W. (2002) · Personality and Social Psychology Bulletin · doi:10.1177/014616702237645 · Cited by 954 · see citing works
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · see citing works
An Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · see citing works
Weller, S., & Romney, A. (1988) · Sage eBooks · doi:10.4135/9781412986069 · Cited by 724 · see citing works
Knowledge Translation in Health Care: Moving from Evidence to Practice
(2009) · International Journal of Health Care Quality Assurance · doi:10.1108/ijhcqa.2009.06222fae.002 · Cited by 689 · see citing works
Developing a comprehensive definition of sustainability
Moore, J. E., Mascarenhas, A., Bain, J., & Straus, S. E. (2017) · Implementation Science · doi:10.1186/s13012-017-0637-1 · Cited by 630 · see citing works
Public health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · see citing works
Asch, S. M., McGlynn, E. A., Hogan, M. M., Hayward, R. A., Shekelle, P., Rubenstein, L. V., KEESEY, J. W., Adams, J., & Kerr, E. A. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-12-200412210-00010 · Cited by 504 · see citing works
Proctor, E., Luke, D., Calhoun, A., McMillen, C., Brownson, R., McCrary, S., & Padek, M. (2015) · Implementation Science · doi:10.1186/s13012-015-0274-5 · Cited by 429 · see citing works
Ivers, N. M., Sales, A., Colquhoun, H., Michie, S., Foy, R., Francis, J. J., & Grimshaw, J. M. (2014) · Implementation Science · doi:10.1186/1748-5908-9-14 · Cited by 400 · see citing works
The Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364 · see citing works
Huijg, J. M., Gebhardt, W. A., Dusseldorp, E., Verheijden, M. W., van der Zouwe, N., Middelkoop, B. J., & Crone, M. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-33 · Cited by 355 · see citing works
Patil, V. H., Singh, S. N., Mishra, S. K., & Donavan, D. T. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2007.05.008 · Cited by 326 · see citing works
Meta-Analysis: Audit and Feedback Features Impact Effectiveness on Care Quality
Hysong, S. J. (2009) · Medical Care · doi:10.1097/mlr.0b013e3181893f6b · Cited by 306 · see citing works
Chamberlain, P., Brown, C. H., & Saldana, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-116 · Cited by 262 · see citing works
Self-citationSustainability of public health interventions: where are the gaps?
Walugembe, D. R., Sibbald, S. L., Ber, M. J. L., & Kothari, A. R. (2019) · Health Research Policy and Systems · doi:10.1186/s12961-018-0405-y · Cited by 206 · see citing works
Sustaining Community‐Based Programs for Families: Conceptualization and Measurement*
Mancini, J. A., & Marek, L. I. (2004) · Family Relations · doi:10.1111/j.0197-6664.2004.00040.x · Cited by 199 · see citing works
Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward, A. M., Ugueto, A. M., & Bernstein, A. (2013) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-013-0485-5 · Cited by 178 · see citing works
Linking Sustainability Research to Intervention Types
Scheirer, M. A. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.300976 · Cited by 161 · see citing works
Sustaining Evidence-Based Prevention Programs: Correlates in a Large-Scale Dissemination Initiative
Cooper, B. R., Bumbarger, B. K., & Moore, J. E. (2015) · Prevention Science · doi:10.1007/s11121-013-0427-1 · Cited by 117 · see citing works
Using the Program Sustainability Assessment Tool to Assess and Plan for Sustainability
Calhoun, A., Mainor, A., Moreland-Russell, S., Maier, R. C., Brossart, L., & Luke, D. A. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130185 · Cited by 100 · see citing works
Persell, S. D., Kaiser, D., Dolan, N. C., Andrews, B., Levi, S., Khandekar, J. D., Gavagan, T. F., Thompson, J. A., Friesema, E. M., & Baker, D. W. (2010) · Medical Care · doi:10.1097/mlr.0b013e318202913d · Cited by 83 · see citing works
Bonner, C., Fajardo, M. A., Doust, J. A., McCaffery, K. J., & Trevena, L. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0927-x · Cited by 70 · see citing works
Conceptualizing and measuring sustainability of prevention programs, policies, and practices
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Reynolds, C., Green, C. D., Olson, C., Adade, A., & Brown, C. H. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz170 · Cited by 66 · see citing works
Self-citationBrown, C. H., Mohr, D. C., Gallo, C. G., Mader, C., Palinkas, L., Wingood, G., Prado, G., Kellam, S. G., Pantin, H., Poduska, J., et al. (2013) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/qai.0b013e31829372bd · Cited by 55 · see citing works
Self-citationSutherland, R., Campbell, E. M., Nathan, N., Wolfenden, L., Lubans, D. R., Morgan, P. J., Gillham, K. E., Oldmeadow, C. J., Searles, A., Reeves, P., et al. (2019) · BMC Public Health · doi:10.1186/s12889-019-6965-0 · Cited by 48 · see citing works
Saldana, L., Bennett, I. M., Powers, D., Vredevoogd, M., Grover, T., Schaper, H., & Campbell, M. R. (2019) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-019-00944-z · Cited by 42 · see citing works
Hodge, L. M., Turner, K. M. T., Sanders, M. R., & Filus, A. (2017) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-016-9505-z · Cited by 38 · see citing works
Cook, C. R., Davis, C. J., Brown, E. C., Locke, J. J., Ehrhart, M. G., Aarons, G. A., Larson, M. F., & Lyon, A. R. (2018) · Implementation Science · doi:10.1186/s13012-018-0804-z · Cited by 38 · see citing works
Aligning Measurement-based Quality Improvement with Implementation of Evidence-based Practices
Hermann, R. C., Chan, J., Zazzali, J. L., & Lerner, D. (2006) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-006-0055-1 · Cited by 32 · see citing works
Measuring sustainment of prevention programs and initiatives: a study protocol
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J. A., Valente, T. W., Chou, C.-P., Landsverk, J. A., Kellam, S. G., & Brown, C. H. (2015) · Implementation Science · doi:10.1186/s13012-016-0467-6 · Cited by 32 · see citing works
Self-citationEhrhart, M. G., Torres, E. M., Hwang, J., Sklar, M., & Aarons, G. A. (2019) · Substance Abuse Treatment Prevention and Policy · doi:10.1186/s13011-019-0222-5 · Cited by 32 · see citing works
Berkel, C., Gallo, C. G., Sandler, I. N., Mauricio, A. M., Smith, J. D., & Brown, C. H. (2019) · The Journal of Primary Prevention · doi:10.1007/s10935-018-00534-z · Cited by 27 · see citing works
Self-citationGallo, C. G., Pantin, H. M., Villamar, J. A., Prado, G. J., Tapia, M. I., Ogihara, M., Cruden, G., & Brown, C. H. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0538-4 · Cited by 24 · see citing works
Self-citationPassive Monitoring Versus Active Assessment of Clinical Performance
Hysong, S. J., Khan, M. M., & Petersen, L. A. (2011) · Medical Care · doi:10.1097/mlr.0b013e318222a36c · Cited by 22 · see citing works
Smiddy, M. P., Murphy, O. M., Savage, E., Fitzgerald, T., Sullivan, B. O., Murphy, C., Bernard, M., & Browne, J. P. (2019) · Journal of Infection Prevention · doi:10.1177/1757177419833165 · Cited by 22 · see citing works
A Framework for Measurement Feedback to Improve Decision-Making in Mental Health
Seidman, E., Chorpita, B. F., Reay, W. E., Stelk, W. J., Garland, A. F., Kutash, K., Mullican, C., & Ringeisen, H. L. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-009-0260-9 · Cited by 17 · see citing works
Automatic classification of communication logs into implementation stages via text analysis
Wang, D., Ogihara, M., Gallo, C., Villamar, J. A., Smith, J. D., Vermeer, W., Cruden, G., Benbow, N., & Brown, C. H. (2015) · Implementation Science · doi:10.1186/s13012-016-0483-6 · Cited by 17 · see citing works
Self-citationAudit and feedback interventions
Foy, R., & Eccles, M. (2013) · Wiley eBooks · doi:10.1002/9781118413555.ch16 · Cited by 12 · see citing works
Audit and Feedback Features Impact Effectiveness on Care Quality
Hysong, S. J. (2009) · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [25]Institute of Medicine Committee on Quality of Health Care in America: Crossing the quality chasm: a new health care system for the 21st century 2001 Washington National Academies Press
- [26]Institute of Medicine Committee on Quality of Health Care in America: Performance measurement: accelerating improvement 2006 Washington DC National Academies Press
- [54]Nunnally JC Psychometric theory 1978 2 New York McGraw Hill
- [55]Nunnally JC Bernstein IH Psychometric theory 1994 3 New York McGraw Hill
Cited by
Sorted by citations
Malone, S., Prewitt, K., Hackett, R., Lin, J. C., McKay, V., Walsh-Bailey, C., & Luke, D. A. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00181-2 · Cited by 148 · see citing works
Moullin, J. C., Sklar, M., Ehrhart, M. G., Green, A., & Aarons, G. A. (2021) · Implementation Science · doi:10.1186/s13012-021-01152-w · Cited by 82 · see citing works
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81 · see citing works
Assessing the sustainability capacity of evidence-based programs in community and health settings
Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1004167 · Cited by 22 · see citing works
“While there have been other measures developed for sustainability capacity in certain areas or to assess sustainment, this capacity focused measure is both pragmatic and generalizable to different settings and amongst those in different roles, including both practitioners and research team members (5).”
Background
- Authors
- Palinkas, L. A., Chou, C.-P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H.
- Affiliations
- University of Southern California (US)
- California State University, Northridge (US)
- Northwestern University (US)
- Corresponding author
- Palinkas, Lawrence A.
- Venue
- Implementation Science
- Published
- 2020-09-03
- Volume
- 15
- Issue
- 1
- Pages
- 71-71
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC7470441
- OpenAlex
- W3082245953
- Cited by
- 70
- References
- 54
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 17 · 2023: 11 · 2022: 11 · 2021: 15
- FWCI
- 9.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute on Drug Abuse (R34 DA037516-01A1, P30 DA02878)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Public Health Policies and Education
- Keywords
- program sustainment, prevention programs, scale development, mental health prevention, sustainability determinants, program adaptation, assessment and feedback, community-based programs, substance use prevention
- MeSH terms
- Factor Analysis, Statistical, Humans, United States, United States Substance Abuse and Mental Health Services Administration, Reproducibility of Results, Program Evaluation, Substance-Related Disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
