CoChat

Assessing the sustainability capacity of evidence-based programs in community and health settings

Open access
Bacon, Caren, Malone, Sara, Prewitt, Kim, Hackett, Rachel, Hastings, Molly, Dexter, Sarah, Luke, Douglas A. · 2022 · Frontiers in Health Services

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 16 of the 20 sources its claims cite were checked for support.

Abstract

Background: Within many public health settings, there remain large challenges to sustaining evidence-based practices. The Program Sustainability Assessment Tool has been developed and validated to measure sustainability capacity of public health, social service, and educational programs. This paper describes how this tool was utilized between January 2014 and January 2019. We describe characteristics of programs that are associated with increased capacity for sustainability and ultimately describe the utility of the PSAT in sustainability research and practice. Methods: The PSAT is comprised of 8 subscales, measuring sustainability capacity in eight distinct conceptual domains. Each subscale is made up of five items, all assessed on a 7-point Likert scale. Data were obtained from persons who used the PSAT on the online website (https://sustaintool.org/), from 2014 to 2019. In addition to the PSAT scale, participants were asked about four program-level characteristics. The resulting dataset includes 5,706 individual assessments reporting on 2,892 programs. Results: The mean overall PSAT score was 4.73, with the lowest and highest scoring subscales being funding stability and program adaptation, respectively. Internal consistency for each subscale was excellent (average Cronbach's alpha = 0.90, ranging from 0.85 to 0.94). Confirmatory factor analysis highlighted good to excellent fit of the PSAT measurement model (eight distinct conceptual domains) to the observed data, with a comparative fit index of 0.902, root mean square error of approximation equal to 0.054, and standardized root mean square residual of 0.054. Overall sustainability capacity was significantly related to program size (F = 25.6; p < 0.001). Specifically, smaller programs (with staff sizes of ten or below) consistently reported lower program sustainability capacity. Capacity was not associated with program age and did not vary significantly by program level. Discussion: The PSAT maintained its excellent reliability when tested with a large and diverse sample over time. Initial criterion validity was explored through the assessment of program characteristics, including program type and program size. The data collected reinforces the ability of the PSAT to assess sustainability capacity for a wide variety of public health and social programs.

Overview

CoChat Summary

TL;DR

This paper describes use of the Program Sustainability Assessment Tool (PSAT) with 5,706 individual assessments of 2,892 programs from 2014–2019, showing that the PSAT maintained strong reliability and an eight-domain measurement structure. Overall sustainability capacity was associated with program size, but not with program age or program level overall.

Key findings

  • Across all programs, the mean overall PSAT score was 4.73; the lowest-scoring subscale was funding stability (M = 3.97) and the highest was program adaptation (M = 5.23).
  • PSAT subscale internal consistency was excellent (average Cronbach’s α = 0.90, range 0.85–0.94), and confirmatory factor analysis supported the eight-factor model (CFI = 0.902, RMSEA = 0.054, SRMR = 0.054); the eight-factor model fit significantly better than a single-factor model (LR = 47,277.2, p < 0.001).
  • Overall sustainability capacity was significantly related to program size (F = 25.6, p < 0.001): larger programs (>20 staff/volunteers, M = 4.93) were rated more sustainable than smaller programs (<4 staff/volunteers, M = 4.47), and programs with ≤3 staff/volunteers reported significantly less capacity on all eight subscales than programs with ≥21.
  • Overall capacity did not vary significantly by program age (F = 1.49, p = 0.226) or program level (F = 0.61, p = 0.54), though subscale differences appeared: older programs (>3 years) had higher funding stability (M = 4.04, F = 7.03, p = 0.001), younger programs (<1 year) had greater program evaluation (M = 5.25, F = 8.54, p = 0.000) and program adaptation (M = 5.36, F = 7.26, p = 0.001), state-level programs had lowest communications capacity (M = 4.42, F = 6.95, p = 0.001), and beyond-state programs had higher financial stability (M = 4.11, F = 3.06, p = 0.047).
  • PSAT total and subscale scores varied significantly by program type except for organizational capacity; clinical programs had the highest total PSAT scores (M = 4.92), followed by public health (M = 4.78), social services (M = 4.64), and education (M = 4.61).

Methods

  • The PSAT consists of 40 items across 8 subscales (5 items each), rated on a 7-point Likert scale; data were passively collected from https://sustaintool.org/ between January 2014 and January 2019, yielding 5,706 individual assessments reporting on 2,892 programs.
  • Of the 2,892 program assessments, 2,283 (79%) were individual assessments; group assessments ranged from 2 to 31 respondents with a median group size of 5, and raw data were aggregated by program.
  • Analyses included descriptive statistics, one-way and two-way ANOVAs to examine PSAT score differences by program type, size, age, and level, Cronbach’s alpha for reliability, and confirmatory factor analysis to test the eight-domain measurement model against a single-factor model.

Limitations

  • The paper states that the sample was comprised of those who sought out the measure for use, that specific site location was not collected through the online survey, that the assessment was conducted before a Spanish translation was available so it was limited to English-speaking respondents, and that the study does not connect sustainability capacity to sustainment outcomes due to lack of sustainment metrics.
  • Evident from the text: data are based on passive self-assessments rather than a random or representative sample, and missing data were present (65% of respondents completed all 40 items; 96% completed at least half).
  • Evident from the text: program-level aggregation may obscure individual respondent variation, and the analyses are observational, so the reported associations do not establish causality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/frhs.2022.1004167

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Conceptual domains for the program sustainability framework.
Figure 1 Conceptual domains for the program sustainability framework.CC BY · PubMed Central
Figure 2 Variability of PSAT total and subscale scores (N = 2,892 programs).
Figure 2 Variability of PSAT total and subscale scores (N = 2,892 programs).CC BY · PubMed Central
Figure 3 PSAT subscale scores by type of program.
Figure 3 PSAT subscale scores by type of program.CC BY · PubMed Central
Figure 4 PSAT subscale scores by program level, staff size, and program age.
Figure 4 PSAT subscale scores by program level, staff size, and program age.CC BY · PubMed Central

Verification

Claims the paper makes

12 Valid · 2 Not valid · 2 Not checked · 56 Own results and interpretations

32 more claims

Create a free account to see all 72 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 17
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 32 references

Cited by

See all 22

Metadata

Authors
Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A.
Affiliations
Washington University in St. Louis (US)
Corresponding author
Bacon, Caren
Venue
Frontiers in Health Services
Published
2022-11-30
Volume
2
Pages
1004167-1004167
ISSN
2813-0146 (ISSN-L)
OpenAlex
W4310984252
Cited by
22
References
32
Citations by year
2026: 5 · 2025: 8 · 2024: 9
FWCI
4.87
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Health Services (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
Funders
Centers for Disease Control and Prevention (75D30120C09195)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Public Health Policies and Education
Keywords
evidence-based programs, program sustainability, public health programs, social service programs, program size, funding stability, program adaptation, scale validation, reliability statistics, criterion validity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.