The Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Open accessAbstract
INTRODUCTION: Public health programs can deliver benefits only if they are able to sustain programs, policies, and activities over time. Although numerous sustainability frameworks and models have been developed, there are almost no assessment tools that have demonstrated reliability or validity or have been widely disseminated. We present the Program Sustainability Assessment Tool (PSAT), a new and reliable instrument for assessing the capacity for program sustainability of various public health and other programs. METHODS: A measurement development study was conducted to assess the reliability of the PSAT. Program managers and staff (n = 592) representing 252 public health programs used the PSAT to rate the sustainability of their program. State and community-level programs participated, representing 4 types of chronic disease programs: tobacco control, diabetes, obesity prevention, and oral health. RESULTS: The final version of the PSAT contains 40 items, spread across 8 sustainability domains, with 5 items per domain. Confirmatory factor analysis shows good fit of the data with the 8 sustainability domains. The subscales have excellent internal consistency; the average Cronbach's α is 0.88, ranging from 0.79 to 0.92. Preliminary validation analyses suggest that PSAT scores are related to important program and organizational characteristics. CONCLUSION: The PSAT is a new and reliable assessment instrument that can be used to measure a public health program's capacity for sustainability. The tool is designed to be used by researchers, evaluators, program managers, and staff for large and small public health programs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The PSAT was developed and tested on over 250 state and local public health programs across a variety of program types.
“The assessment instrument was developed and tested on over 250 state and local public health programs across a variety of program types (tobacco control, obesity, nutrition, etc.).”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [15]
The Methods (Participating programs and respondents) report that 252 programs completed the initial PSAT, including state- and community-level programs across four chronic disease areas. Tobacco control and obesity prevention match the citing sentence’s examples, but nutrition is not among the program areas listed. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe PSAT was developed and validated to measure sustainability capacity of public health, social service, and educational programs.
“The Program Sustainability Assessment Tool has been developed and validated to measure sustainability capacity of public health, social service, and educational programs.”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [15]
The Results report preliminary validation analyses in 252 public health programs, and the Discussion says the PSAT is ready to assess program sustainability capacity. The Discussion also says it was designed for social service programs, but had been tested only with chronic disease programs; it does not establish validation for social service or educational programs. The indexed passages provide section labels but no page numbers. (LLM, conf=0.89)
Experimental — accuracy not yet measuredFactor analysis with 592 participants from 252 state and community public health programs yielded eight PSAT domains with good discriminant and construct validity.
“They conducted factor analysis with a sample of 592 participants from 252 state and community public health programs that yielded eight PSAT domains—environmental support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications, and strategic planning—with good discriminant and construct validity [28].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [28]
The Methods report 592 respondents from 252 programs, and the Results report good fit for an eight-domain confirmatory factor model. The Subscale reliability table notes that Political Support is now called Environmental Support. However, the Results describe only preliminary construct-validity findings and some discriminant-validation evidence, rather than establishing good validity on both measures. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe Program Sustainability Assessment Tool (PSAT) had been previously developed with established reliability and validity.
“Both measures had been previously developed with established reliability and validity, and have been widely used across a variety of contexts and settings.”
From Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) (2024) · cites this paper as [7]
The Discussion reports that the PSAT was reliable, and the Results report good internal consistency and preliminary evidence of validity. However, the Methods say the pilot study was not designed as an in-depth validation study, so describing validity as established overstates the evidence in this paper. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPsychometric analyses of the original PSAT data demonstrated good reliability (internal consistency) and confirmatory factor analysis supported the suitability of eight domains measured by five items per domain.
“Psychometric analyses of the PSAT using these original data demonstrated good reliability (i.e., internal consistency) and confirmatory factor analysis supported the suitability of eight domains measured by five items per domain (15).”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [15]
The PSAT consists of 40 questions, five items in each of eight domain subscales, with 7-point Likert-scale responses.
“The PSAT is comprised of 8 subscales, measuring sustainability capacity in eight distinct conceptual domains. Each subscale is made up of five items, all assessed on a 7-point Likert scale.”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [15]
In the original PSAT development study, average internal consistency of the 8 subscales was 0.88 and domain subscales ranged from 0.79 to 0.92.
“In our original PSAT development study, average internal consistency (Cronbach's alpha) of the 8 subscales was 0.88 and domain subscales ranged from 0.79 to 0.92 (15).”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [15]
The PSAT contains eight domains with five items each, for a total of 40 items.
“The PSAT contains eight different domains with five items in each domain for a total of 40 items.”
From Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) (2024) · cites this paper as [7]
Many approaches highlighted the importance of the ability of an initiative to garner and maintain resources through stable sources.
“Many approaches highlighted the importance of the ability of an initiative to garner and maintain resources [15, 27, 36–41] through stable sources [19, 36, 39, 42–45].”
From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [36]
The ability of an initiative to share resources with partners and other organisations, seek out alternative and supplemental resources and/or uncover multiple funding sources were highlighted across some approaches as important to overall sustainability.
“The ability of an initiative to share resources with partners and other organisations [41, 46], seek out alternative and supplemental resources [18, 47–49] and/or uncover multiple funding sources [8, 36, 49, 50] were also highlighted across some approaches as important to overall sustainability.”
From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [36]
Many approaches presented in this review recommend that they be used and evaluated further within other healthcare initiatives and settings to explore applicability and further development needed.
“Many approaches presented in this review recommend that they be used and evaluated further within other healthcare initiatives and settings to explore applicability and further development needed [11, 36, 56, 79].”
From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [36]
Although several studies have applied the Program Sustainability Assessment Tool measure or framework, it is not evident whether the tool captures all relevant domains.
“Although several studies have applied the PSAT measure or framework (62, 76, 138, 141), it is not evident whether the tool captures all relevant domains, and research is needed to validate the tool across diverse contexts.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [76]
There is emerging evidence for the reliability and validity of a small number of sustainability-specific assessments and tools designed to determine an organization's capacity in key factors linked to sustainment success and to develop strategic action plans.
“There is emerging evidence for the reliability and validity of a small number of sustainability-specific assessments and tools, which are designed to determine an organization's capacity in key factors (e.g., partnerships, strategic planning) that have been shown to be linked to sustainment success (61, 76) and to develop a strategic action plan for building capacity in areas of weakness (19, 35,”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [76]
Several measures exist for factors influencing sustainability, including the British National Health Service Sustainability Index, the Program Sustainability Assessment Tool, and the Sustained Implementation Support Scale.
“Other examples include the British National Health Service Sustainability Index, a 10-item measure of an organization's likelihood to sustain practices (78), the Program Sustainability Assessment Tool (76, 138), and the Sustained Implementation Support Scale, a scale that measures factors that affect the implementation process among practitioners (61).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
<b>lavaan</b> : An <i>R</i> Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955 · see citing works
Confirmatory factor analysis for applied research
Brown, T. A. (2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,613 · see citing works
Hu, L.-T., & Bentler, P. M. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.424 · Cited by 12,298 · see citing works
Book Review: Structural Equation Modeling: Concepts, Issues, and Applications
Rigdon, E. E., & Hoyle, R. H. (1997) · Journal of Marketing Research · doi:10.2307/3151904 · Cited by 4,085 · see citing works
Hu, L., & Bentler, P. M. (1995) · Cited by 4,008 · see citing works
Factor analysis in the development and refinement of clinical assessment instruments.
Floyd, F. J., & Widaman, K. F. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.286 · Cited by 3,970 · see citing works
Uses and abuses of coefficient alpha
Schmitt, N. (1996) · Psychological Assessment · doi:10.1037/1040-3590.8.4.350 · Cited by 2,588 · see citing works
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · see citing works
An introduction to concept mapping for planning and evaluation
Trochim, W. M. K. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90016-5 · Cited by 1,512 · see citing works
Dissemination and Implementation Research in HealthTranslating Science to Practice
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2012) · doi:10.1093/acprof:oso/9780199751877.001.0001 · Cited by 1,355 · see citing works
Effect of the number of variables on measures of fit in structural equation modeling
Kenny, D. A., & McCoach, D. B. (2003) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1003_1 · Cited by 1,231 · see citing works
Shediac-Rizkallah, M. C., & Bone, L. R. (1998) · Health Education Research · doi:10.1093/her/13.1.87 · Cited by 1,129 · see citing works
An Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · see citing works
Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability
Scheirer, M. A. (2005) · American Journal of Evaluation · doi:10.1177/1098214005278752 · Cited by 710 · see citing works
Large‐System Transformation in Health Care: A Realist Review
Best, A., Greenhalgh, T., Lewis, S. J., Saul, J. E., Carroll, S., & Bitz, J. (2012) · Milbank Quarterly · doi:10.1111/j.1468-0009.2012.00670.x · Cited by 575 · see citing works
Public health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · see citing works
Self-citationLinking Sustainability Research to Intervention Types
Scheirer, M. A. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.300976 · Cited by 161 · see citing works
Using the Program Sustainability Assessment Tool to Assess and Plan for Sustainability
Calhoun, A., Mainor, A., Moreland-Russell, S., Maier, R. C., Brossart, L., & Luke, D. A. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130185 · Cited by 100 · see citing works
Self-citationOrganizational Characteristics of Successful Innovative Health Care Programs Sustained Over Time
Evashwick, C., & Ory, M. G. (2003) · Family & Community Health · doi:10.1097/00003727-200307000-00003 · Cited by 76 · see citing works
Luke, D. A., Ribisl, K. M., Walton, M. A., & Davidson, W. S. (2002) · Substance Use & Misuse · doi:10.1081/ja-120001498 · Cited by 46 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [5]Hutchinson K. Literature review of program sustainability assessment tools. Vancouver (BC): The Capture Project, Simon Fraser University; 2010. http://www.thecaptureplatform.ca/intervention/379/. Accessed January 9, 2013.
Cited by
Sorted by citations
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · see citing works
Shelton, R. C., Chambers, D. A., & Glasgow, R. E. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00134 · Cited by 448 · see citing works
Lennox, L., Maher, L., & Reed, J. (2018) · Implementation Science · doi:10.1186/s13012-017-0707-4 · Cited by 421 · see citing works
Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties
Mettert, K., Lewis, C., Dorsey, C., Halko, H., & Weiner, B. (2020) · Implementation Research and Practice · doi:10.1177/2633489520936644 · Cited by 162 · see citing works
Malone, S., Prewitt, K., Hackett, R., Lin, J. C., McKay, V., Walsh-Bailey, C., & Luke, D. A. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00181-2 · Cited by 148 · see citing works
Self-citationKemp, C. G., Jarrett, B. A., Kwon, C.-S., Song, L., Jetté, N., Sapag, J. C., Bass, J., Murray, L., Rao, D., & Baral, S. (2019) · BMC Medicine · doi:10.1186/s12916-018-1237-x · Cited by 138 · see citing works
Using the Program Sustainability Assessment Tool to Assess and Plan for Sustainability
Calhoun, A., Mainor, A., Moreland-Russell, S., Maier, R. C., Brossart, L., & Luke, D. A. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130185 · Cited by 100 · see citing works
Self-citationHall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81 · see citing works
Advancing the pragmatic measurement of sustainment: a narrative review of measures
Moullin, J. C., Sklar, M., Green, A., Dickson, K. S., Stadnick, N. A., Reeder, K., & Aarons, G. A. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00068-8 · Cited by 72 · see citing works
Palinkas, L. A., Chou, C.-P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H. (2020) · Implementation Science · doi:10.1186/s13012-020-01030-x · Cited by 70 · see citing works
Understanding mis-implementation in Public Health practice
Brownson, R. C., Allen, P., Jacob, R. R., Harris, J. K., Duggan, K., Hipp, P. R., & Erwin, P. C. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.11.015 · Cited by 70 · see citing works
Lennox, L., Doyle, C., Reed, J. E., & Bell, D. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014417 · Cited by 59 · see citing works
Lewis, C. C., Mettert, K., & Lyon, A. R. (2021) · Implementation Research and Practice · doi:10.1177/2633489521994197 · Cited by 38 · see citing works
Assessing capacity for sustainability of effective programs and policies in local health departments
Tabak, R. G., Duggan, K., Smith, C., Aisaka, K., Moreland-Russell, S., & Brownson, R. C. (2016) · Journal of Public Health Management and Practice · doi:10.1097/phh.0000000000000254 · Cited by 40 · see citing works
Self-citationStoll, S., Janevic, M., Lara, M., Ramos-Valencia, G., Stephens, T. B., Persky, V., Uyeda, K., Ohadike, Y., & Malveaux, F. (2015) · Preventing Chronic Disease · doi:10.5888/pcd12.150133 · Cited by 32 · see citing works
Scheirer, M. A., Santos, S. L. Z., Tagai, E. K., Bowie, J., Slade, J., Carter, R., & Holt, C. L. (2017) · Implementation Science · doi:10.1186/s13012-017-0576-x · Cited by 29 · see citing works
Hall, A., Shoesmith, A., Shelton, R. C., Lane, C., Wolfenden, L., & Nathan, N. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182111414 · Cited by 20 · see citing works
Assessing the sustainability capacity of evidence-based programs in community and health settings
Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1004167 · Cited by 22 · see citing works
“, internal consistency) and confirmatory factor analysis supported the suitability of eight domains measured by five items per domain (15).”
BackgroundMethodSelf-citationFrancis, L., Dunt, D., & Cadilhac, D. A. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010944 · Cited by 24 · see citing works
Examination of sustainability indicators for fall prevention strategies in three states
Smith, M. L., Durrett, N. K., Schneider, E. C., Byers, I. N., Shubert, T. E., Wilson, A. D., Towne, S. D., & Ory, M. G. (2018) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2018.02.001 · Cited by 22 · see citing works
Johnson, K., Collins, D., Shamblen, S., Kenworthy, T., & Wandersman, A. (2017) · Prevention Science · doi:10.1007/s11121-017-0784-2 · Cited by 23 · see citing works
Tandon, S. D., McGown, M., Campbell, L., Smith, J. D., Yeh, C., & Brady, C. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.07.033 · Cited by 21 · see citing works
Agulnik, A., Malone, S., Puerto-Torres, M., Gonzalez-Ruiz, A., Vedaraju, Y., Wang, H., Graetz, D., Prewitt, K., Villegas, C., Cardenas-Aguierre, A., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053116 · Cited by 20 · see citing works
Self-citationSustainability of an interprofessional collaborative practice model for population health
Shirey, M. R., Selleck, C. S., White-Williams, C., Talley, M., & Harper, D. C. (2020) · Nursing Administration Quarterly · doi:10.1097/naq.0000000000000429 · Cited by 19 · see citing works
King, D. K., Gonzalez, S. J., Hartje, J. A., Hanson, B. L., Edney, C., Snell, H., Zoorob, R. J., & Roget, N. A. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/ibx020 · Cited by 17 · see citing works
Dorsey, C. N., Mettert, K. D., Puspitasari, A. J., Damschroder, L. J., & Lewis, C. C. (2021) · Implementation Research and Practice · doi:10.1177/26334895211002474 · Cited by 15 · see citing works
Malone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00656-y · Cited by 12 · see citing works
“Our team developed two theoretically informed measures of sustainability capacity, the Program Sustainability Assessment Tool (PSAT) [7, 8], designed to assess sustainability in public health settings, and the Clinical Sus-tainability Assessment Tool (CSAT) [6, 9], designed to assess sustainability…”
MethodSelf-citationCooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00303-4 · Cited by 7 · see citing works
“’s [28] initial psychometric evaluation of the strategic planning scale.”
BackgroundMethodMalone, S., Newland, J., Kudchadkar, S. R., Prewitt, K., McKay, V., Prusaczyk, B., Proctor, E., Brownson, R. C., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1005802 · Cited by 6 · see citing works
Self-citation
- Authors
- Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S.
- Affiliations
- Washington University in St. Louis (US)
- Saint Louis University (US)
- Corresponding author
- Luke, Douglas A.
- Venue
- Preventing Chronic Disease
- Published
- 2014-01-23
- Volume
- 11
- Pages
- 130184-130184
- ISSN
- 1545-1151 (ISSN-L)
- PMC
- PMC3900326
- OpenAlex
- W2068297408
- Cited by
- 364
- References
- 20
- Citations by year
- 2026: 30 · 2025: 48 · 2024: 43 · 2023: 45 · 2022: 50 · 2021: 46 · 2020: 35 · 2019: 18 · 2018: 22 · 2017: 14 · 2016: 3 · 2015: 7 · 2014: 3
- FWCI
- 16.13
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- public-domain
- Locations
- Preventing Chronic Disease (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Centers for Disease Control and Prevention
- U.S. Public Health Service (200-2011-42065)
- National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092950)
- Topics
- Health Sciences › Health Professions › General Health Professions › Public Health Policies and Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- program sustainability, public health programs, sustainability assessment tools, measurement development, reliability statistics, tobacco control, diabetes prevention, obesity prevention, oral health program
- MeSH terms
- Community Health Services, Factor Analysis, Statistical, Health Promotion, Humans, Public Health, Reproducibility of Results, Program Evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
