Assessing the sustainability capacity of evidence-based programs in community and health settings
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 16 of the 20 sources its claims cite were checked for support.
Abstract
Background: Within many public health settings, there remain large challenges to sustaining evidence-based practices. The Program Sustainability Assessment Tool has been developed and validated to measure sustainability capacity of public health, social service, and educational programs. This paper describes how this tool was utilized between January 2014 and January 2019. We describe characteristics of programs that are associated with increased capacity for sustainability and ultimately describe the utility of the PSAT in sustainability research and practice. Methods: The PSAT is comprised of 8 subscales, measuring sustainability capacity in eight distinct conceptual domains. Each subscale is made up of five items, all assessed on a 7-point Likert scale. Data were obtained from persons who used the PSAT on the online website (https://sustaintool.org/), from 2014 to 2019. In addition to the PSAT scale, participants were asked about four program-level characteristics. The resulting dataset includes 5,706 individual assessments reporting on 2,892 programs. Results: The mean overall PSAT score was 4.73, with the lowest and highest scoring subscales being funding stability and program adaptation, respectively. Internal consistency for each subscale was excellent (average Cronbach's alpha = 0.90, ranging from 0.85 to 0.94). Confirmatory factor analysis highlighted good to excellent fit of the PSAT measurement model (eight distinct conceptual domains) to the observed data, with a comparative fit index of 0.902, root mean square error of approximation equal to 0.054, and standardized root mean square residual of 0.054. Overall sustainability capacity was significantly related to program size (F = 25.6; p < 0.001). Specifically, smaller programs (with staff sizes of ten or below) consistently reported lower program sustainability capacity. Capacity was not associated with program age and did not vary significantly by program level. Discussion: The PSAT maintained its excellent reliability when tested with a large and diverse sample over time. Initial criterion validity was explored through the assessment of program characteristics, including program type and program size. The data collected reinforces the ability of the PSAT to assess sustainability capacity for a wide variety of public health and social programs.
Overview
CoChat Summary
TL;DR
This paper describes use of the Program Sustainability Assessment Tool (PSAT) with 5,706 individual assessments of 2,892 programs from 2014–2019, showing that the PSAT maintained strong reliability and an eight-domain measurement structure. Overall sustainability capacity was associated with program size, but not with program age or program level overall.
Key findings
- Across all programs, the mean overall PSAT score was 4.73; the lowest-scoring subscale was funding stability (M = 3.97) and the highest was program adaptation (M = 5.23).
- PSAT subscale internal consistency was excellent (average Cronbach’s α = 0.90, range 0.85–0.94), and confirmatory factor analysis supported the eight-factor model (CFI = 0.902, RMSEA = 0.054, SRMR = 0.054); the eight-factor model fit significantly better than a single-factor model (LR = 47,277.2, p < 0.001).
- Overall sustainability capacity was significantly related to program size (F = 25.6, p < 0.001): larger programs (>20 staff/volunteers, M = 4.93) were rated more sustainable than smaller programs (<4 staff/volunteers, M = 4.47), and programs with ≤3 staff/volunteers reported significantly less capacity on all eight subscales than programs with ≥21.
- Overall capacity did not vary significantly by program age (F = 1.49, p = 0.226) or program level (F = 0.61, p = 0.54), though subscale differences appeared: older programs (>3 years) had higher funding stability (M = 4.04, F = 7.03, p = 0.001), younger programs (<1 year) had greater program evaluation (M = 5.25, F = 8.54, p = 0.000) and program adaptation (M = 5.36, F = 7.26, p = 0.001), state-level programs had lowest communications capacity (M = 4.42, F = 6.95, p = 0.001), and beyond-state programs had higher financial stability (M = 4.11, F = 3.06, p = 0.047).
- PSAT total and subscale scores varied significantly by program type except for organizational capacity; clinical programs had the highest total PSAT scores (M = 4.92), followed by public health (M = 4.78), social services (M = 4.64), and education (M = 4.61).
Methods
- The PSAT consists of 40 items across 8 subscales (5 items each), rated on a 7-point Likert scale; data were passively collected from https://sustaintool.org/ between January 2014 and January 2019, yielding 5,706 individual assessments reporting on 2,892 programs.
- Of the 2,892 program assessments, 2,283 (79%) were individual assessments; group assessments ranged from 2 to 31 respondents with a median group size of 5, and raw data were aggregated by program.
- Analyses included descriptive statistics, one-way and two-way ANOVAs to examine PSAT score differences by program type, size, age, and level, Cronbach’s alpha for reliability, and confirmatory factor analysis to test the eight-domain measurement model against a single-factor model.
Limitations
- The paper states that the sample was comprised of those who sought out the measure for use, that specific site location was not collected through the online survey, that the assessment was conducted before a Spanish translation was available so it was limited to English-speaking respondents, and that the study does not connect sustainability capacity to sustainment outcomes due to lack of sustainment metrics.
- Evident from the text: data are based on passive self-assessments rather than a random or representative sample, and missing data were present (65% of respondents completed all 40 items; 96% completed at least half).
- Evident from the text: program-level aggregation may obscure individual respondent variation, and the analyses are observational, so the reported associations do not establish causality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
12 Valid · 2 Not valid · 2 Not checked · 56 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“While some work has adapted the PSAT for specific areas, this work suggests that the PSAT remains valid in its entirety, even when assessed in a larger and more diverse sample (31).”
§ Assessing the sustainability capacity of evidence-based programs in community and health settings > Discussion
What the sources say
- Disputedcites [31] Hall, 2021, Adaptation and validation of the program sustainability assessment tool (PSAT) for use in the elementary school setting, Int J Environ Res Public Health. sourceExperimental — accuracy not yet measured
Results §3.1 says the elementary-school PSAT required substantial adaptation: one of eight domains and several original items were removed, so this study did not validate the PSAT in its entirety. The Discussion says larger samples are still needed to establish the adapted measure’s structural validity; Results §3.5 also reports a lack of evidence for convergent validity. No page numbers are available in our copy. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The assessment instrument was developed and tested on over 250 state and local public health programs across a variety of program types (tobacco control, obesity, nutrition, etc.).”
§ Assessing the sustainability capacity of evidence-based programs in community and health settings > Introduction
What the sources say
- Partly supportedcites [15] Luke, 2014, The program sustainability assessment tool: a new instrument for public health programs, Prev Chronic Dis. sourceExperimental — accuracy not yet measured
The Methods (Participating programs and respondents) report that 252 programs completed the initial PSAT, including state- and community-level programs across four chronic disease areas. Tobacco control and obesity prevention match the citing sentence’s examples, but nutrition is not among the program areas listed. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To assess these settings, an adaptation of the PSAT was developed that focuses on clinical programs and practices.”
§ Assessing the sustainability capacity of evidence-based programs in community and health settings > Discussion
What the sources say
- Partly supportedcites [35] Agulnik, 2021, Reliability and validity of a Spanish-language measure assessing clinical capacity to sustain paediatric early warning systems (PEWS) in resource-limited hospitals, BMJ Open. sourceExperimental — accuracy not yet measured
The Background and Methods (CSAT adaptation; page not provided) describe the Clinical Sustainability Assessment Tool as a measure for sustaining clinical practices and report adapting it into Spanish. The paper cites work on the PSAT when describing the CSAT’s domains, but our copy does not explicitly say the CSAT was developed as an adaptation of the PSAT; that relationship is not fully supported. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, older programs (>3 years, M = 4.04) reported higher capacity for funding stability (M = 4.04; F = 7.03; p = 0.001), while younger programs (< 1 year) showed greater capacity in program evaluation (M = 5.25; F = 8.54; p = 0.000) and program adaptation (M = 5.36; F = 7.26; p = 0.001).”
§ Assessing the sustainability capacity of evidence-based programs in community and health settings > Results > Impact of programmatic factors on PSAT trends
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Public health programs, on the other hand, showed higher scores on partnerships and communications.”
§ Assessing the sustainability capacity of evidence-based programs in community and health settings > Results > PSAT characteristics by program type
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
32 more claims
Create a free account to see all 72 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 17
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Together, the PSAT and CSAT have been completed online over 13,000 times to measure the sustainability capacity of programs and practices across a variety of settings, including public health [10, 13], clinical care [9, 14], behavioral health [15], and education [16].”
From Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) (2024) · cites this paper as [13]
Experimental — accuracy not yet measured
Citations
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788
Structural Equation Modeling: Concepts, Issues, and Applications.
Smith, M. D., & Hoyle, R. H. (1996) · The Statistician · doi:10.2307/2988418 · Cited by 6,814
Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses
Vuong, Q. H. (1989) · Econometrica · doi:10.2307/1912557 · Cited by 5,929
(1995) · Cited by 4,008
Effect of the number of variables on measures of fit in structural equation modeling
Kenny, D. A., & McCoach, D. B. (2003) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1003_1 · Cited by 1,231
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806
An Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · cited 1×
“For example, practitioners can seek to maintain program activities, community-level partnerships, organizational practices, benefits to clients, and the salience of the program’s core issue (11).”
BackgroundGlasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012) · American Journal of Public Health · doi:10.2105/ajph.2012.300755 · Cited by 762 · cited 2×
“Many implementation science studies focus on the early stages of implementation and short-term outcomes; thus there is an important research and evaluation opportunity focusing on how effective programs are sustained over time after their initial adoption (1, 3, 4).”
BackgroundDefining and Implementing Value-Based Health Care: A Strategic Framework
Teisberg, E., Wallace, S., & O’Hara, S. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003122 · Cited by 682 · cited 1×
“Research and theory on the concept of value-based care has also focused on some of these organizational activities and describes a need to focus on overall value of care and team-based care instead of simply focusing on reducing costs of care (12).”
BackgroundPublic health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · cited 4×
“Public health program sustainability can take many forms (10).”
BackgroundMethodInfluentialSelf-citation
Cited by
Malone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00656-y · Cited by 12
“Together, the PSAT and CSAT have been completed online over 13,000 times to meas - ure the sustainability capacity of programs and practices across a variety of settings, including public health [10, 13], clinical care [9, 14], behavioral health [15], and education [16].”
BackgroundSelf-citationAligning organisational priorities and implementation science for cancer research
Best, S., Thursky, K., Buzza, M., Klaic, M., Peters, S., Guccione, L., Trainer, A. H., & Francis, J. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-10801-x · Cited by 7
“A range of tools exist e.g., the Intervention Scalability Assessment Tool (ISAT) [32] and the Program Sustainability Assessment Tool(PSAT) [33] to initiate the planning required and potentially challenging discussions into integration of services.”
MethodStaton, M. D., Bell, J. S., McGuire, A. B., Taylor, L. D., & Watson, D. P. (2024) · Journal of Substance Use and Addiction Treatment · doi:10.1016/j.josat.2024.209540 · Cited by 4
Improving the Sustainability of Public Health Interventions: The INNOVATE Model
Haregu, T., Kwaśnicka, D., Shrestha, A., Lynch, C., Absetz, P., Miranda, J. J., Thankappan, K. R., & Oldenburg, B. F. (2025) · Global Implementation Research and Applications · doi:10.1007/s43477-025-00149-7 · Cited by 6
“…et al., 2021 ), Stages of implementation completion (Chamberlain et al., 2011; Doyle et al., 2013 ), Program Sustainability Assessment Tool (PSAT) (Bacon et al., 2022 ), Program Sustainability Index (Mancini & Marek, 2004 ), and the Sustainment Measurement System Scale (Palinkas et al., 2020 ).”
Dopp, A. R., Bongard, M., Han, B., Hindmarch, G., Shiferaw, M., Mendon, S. J., Tumendemberel, B., Timmins, G. T., Reeder, K., Pantoja, P. M., et al. (2025) · Implementation Science · doi:10.1186/s13012-025-01471-2 · Cited by 3
“Setting characteristics included organizational focus (i.e., whether the primary focus of services was substance use), number of services offered (from a list of 17 categories), and eight EBP sustainment capacity domains measured by the Program Sustainability Assessment Tool (PSAT) [17, 18].”
Metadata
- Authors
- Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A.
- Affiliations
- Washington University in St. Louis (US)
- Corresponding author
- Bacon, Caren
- Venue
- Frontiers in Health Services
- Publisher
- Frontiers Media SA
- Published
- 2022-11-30
- Volume
- 2
- Pages
- 1004167-1004167
- ISSN
- 2813-0146 (ISSN-L)
- PMC
- PMC10012779
- OpenAlex
- W4310984252
- Cited by
- 22
- References
- 32
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 9
- FWCI
- 4.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Health Services (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
- Funders
- Centers for Disease Control and Prevention (75D30120C09195)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Public Health Policies and Education
- Keywords
- evidence-based programs, program sustainability, public health programs, social service programs, program size, funding stability, program adaptation, scale validation, reliability statistics, criterion validity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.