A Mixed-Method Application of the Program Sustainability Assessment Tool to Evaluate the Sustainability of 4 Pediatric Asthma Care Coordination Programs
Open accessAbstract
INTRODUCTION: As part of a cross-site evaluation of the implementation of an evidence-based intervention for pediatric asthma care coordination into low-income communities, we sought to understand the factors that influenced the programs' expected sustainability of the programs after external funding ended. METHODS: We administered the Center for Public Health Systems Science's Program Sustainability Assessment Tool, a 40-item instrument assessing 8 domains of sustainability capacity, to 12 key informants across 4 program sites. We developed open-ended probes for each domain. We examined patterns in site-specific and overall domain scores, and coded qualitative data to identify challenges and strategies in each domain. RESULTS: Across sites, the domains of program evaluation (cross-site mean, 5.4 on a scale of 1-7) and program adaptation (mean, 5.2) had the highest ratings (indicating a strong finding during program evaluation) and funding stability had the lowest rating (mean, 2.7). Scores varied most across sites in the domains of strategic planning (SD, 0.9) and funding stability (SD, 0.9). Qualitative data revealed key challenges, including how implementation difficulties and externally led implementation can impede planning for sustainability. Program leaders discussed multiple strategies for enhancing capacity within each domain, including capitalizing on the interconnectedness of all domains, such as using evaluation and communication strategies to bolster internal political support throughout the implementation process. CONCLUSION: Findings indicating weak and strong domains were consistent with previous findings of studies that used the Program Sustainability Assessment Tool. The addition of qualitative probes yielded detailed data describing capacity strengths, weaknesses, and strategies to increase the likelihood that programs are sustained.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the absence of program evaluation, the presence of the other four domains may have provided compensatory strength for program sustainment.
“In the absence of program evaluation, the presence of the other four domains may have provided compensatory strength for program sustainment [30].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [30]
The Discussion says programs may use strengths in one sustainability domain to bolster others. It does not describe an absence of program evaluation or identify four other domains as compensating for it; instead, it reports that evaluation was rated highly at all four sites. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThese results reinforce the multidimensional and interactive nature of sustainability capacity domains and their role in program sustainment.
“These results reinforce the multidimensional and interactive nature of sustainability capacity domains and their role in program sustainment [21, 27, 30].”
The PSAT has been employed in several sustainability studies, but these studies have been primarily descriptive and seldom examined predictive strength between PSAT domains and program sustainment.
“Since its inception, the PSAT has been employed in several sustainability studies, though the studies have been primarily descriptive, seldom examining the predictive strength between the PSAT domains and program sustainment [30, 31].”
Although several studies have applied the Program Sustainability Assessment Tool measure or framework, it is not evident whether the tool captures all relevant domains.
“Although several studies have applied the PSAT measure or framework (62, 76, 138, 141), it is not evident whether the tool captures all relevant domains, and research is needed to validate the tool across diverse contexts.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [138]
Several measures exist for factors influencing sustainability, including the British National Health Service Sustainability Index, the Program Sustainability Assessment Tool, and the Sustained Implementation Support Scale.
“Other examples include the British National Health Service Sustainability Index, a 10-item measure of an organization's likelihood to sustain practices (78), the Program Sustainability Assessment Tool (76, 138), and the Sustained Implementation Support Scale, a scale that measures factors that affect the implementation process among practitioners (61).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [138]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · see citing works
Shediac-Rizkallah, M. C., & Bone, L. R. (1998) · Health Education Research · doi:10.1093/her/13.1.87 · Cited by 1,129 · see citing works
An Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · see citing works
Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability
Scheirer, M. A. (2005) · American Journal of Evaluation · doi:10.1177/1098214005278752 · Cited by 710 · see citing works
Public health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · see citing works
Toward the Effective and Efficient Measurement of Implementation Fidelity
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam‐Gerow, M. A. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0321-0 · Cited by 422 · see citing works
The Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364 · see citing works
Sustaining Community‐Based Programs for Families: Conceptualization and Measurement*
Mancini, J. A., & Marek, L. I. (2004) · Family Relations · doi:10.1111/j.0197-6664.2004.00040.x · Cited by 199 · see citing works
Linking Sustainability Research to Intervention Types
Scheirer, M. A. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.300976 · Cited by 161 · see citing works
Using the Program Sustainability Assessment Tool to Assess and Plan for Sustainability
Calhoun, A., Mainor, A., Moreland-Russell, S., Maier, R. C., Brossart, L., & Luke, D. A. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130185 · Cited by 100 · see citing works
Defining sustainability outcomes of health programs: Illustrations from an on-line survey
Scheirer, M. A., Hartling, G., & Hagerman, D. (2008) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2008.08.004 · Cited by 93 · see citing works
Understanding mis-implementation in Public Health practice
Brownson, R. C., Allen, P., Jacob, R. R., Harris, J. K., Duggan, K., Hipp, P. R., & Erwin, P. C. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.11.015 · Cited by 70 · see citing works
Fischer, M. (2014) · Sustainability · doi:10.3390/su7010294 · Cited by 61 · see citing works
TheYes We CanUrban Asthma Partnership: A Medical/Social Model for Childhood Asthma Management
Thyne, S. M., Rising, J. P., Legion, V., & Love, M. B. (2006) · Journal of Asthma · doi:10.1080/02770900600925288 · Cited by 52 · see citing works
Towards improved understanding of injury prevention program sustainability
Nilsén, P., Timpka, T., Nordenfelt, L., & Lindqvist, K. S. (2005) · Safety Science · doi:10.1016/j.ssci.2005.08.015 · Cited by 42 · see citing works
Mansfield, C. A., Viswanathan, M. S., Woodell, C., et al. (2011) · Health Promotion Practice · doi:10.1177/1524839911415665 · Cited by 24 · see citing works
Sarriot, E., Ricca, J., Ryan, L. J., Basnet, J., & Arscott‐Mills, S. (2009) · The International Journal of Health Planning and Management · doi:10.1002/hpm.1012 · Cited by 19 · see citing works
A Tool for Rating Chronic Disease Prevention and Public Health Interventions
Kelly, C., Scharff, D., LaRose, J., Dougherty, N. L., Hessel, A. S., & Brownson, R. C. (2013) · Preventing Chronic Disease · doi:10.5888/pcd10.130173 · Cited by 16 · see citing works
Translating Evidence-Based Interventions Into Practice
Viswanathan, M., Lux, L. J., Lohr, K. N., Evans, T. S., Smith, L. R., Woodell, C., Mansfield, C. A., Darcy, N., Ohadike, Y. U., Lesch, J. K., et al. (2011) · Health Promotion Practice · doi:10.1177/1524839911412594 · Cited by 16 · see citing works
Self-citationBlackford, J., & Street, A. F. (2012) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2012.04179.x · Cited by 13 · see citing works
Comprehensive asthma management for underserved children
Thyne, S. M., Marmor, A. K., Madden, N., et al. (2007) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2007.00882.x · Cited by 13 · see citing works
The Influence of Setting on Care Coordination for Childhood Asthma
Kelly, R. P., STOLL, S. C., Bryant-Stephens, T. C., et al. (2015) · Health Promotion Practice · doi:10.1177/1524839915598499 · Cited by 10 · see citing works
Self-citationChallenges and Lessons Learned From the Translation of Evidence-Based Childhood Asthma Interventions
Ohadike, Y. U., Malveaux, F. J., & Lesch, J. K. (2011) · Health Promotion Practice · doi:10.1177/1524839911414565 · Cited by 7 · see citing works
Self-citationThe complexities of home visitation for children with asthma in underserved communities
Thyne, S. M., & Fisher‐Owens, S. A. (2011) · Journal of Asthma · doi:10.3109/02770903.2011.554947 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [20]Patton MQ. Qualitative research and evaluation methods. 3rd edition. Thousand Oaks (CA): Sage Publications; 2002.
Cited by
Sorted by citations
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · see citing works
Hall, A., Shoesmith, A., Shelton, R. C., Lane, C., Wolfenden, L., & Nathan, N. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182111414 · Cited by 20 · see citing works
Assessing the sustainability capacity of evidence-based programs in community and health settings
Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1004167 · Cited by 22 · see citing works
Examination of sustainability indicators for fall prevention strategies in three states
Smith, M. L., Durrett, N. K., Schneider, E. C., Byers, I. N., Shubert, T. E., Wilson, A. D., Towne, S. D., & Ory, M. G. (2018) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2018.02.001 · Cited by 22 · see citing works
Malone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00656-y · Cited by 12 · see citing works
“The PSAT and CSAT have been used in organizational evaluation as well as within public health and implementation science research [10–12].”
MethodCooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00303-4 · Cited by 7 · see citing works
“Sustainment conditions Similar to past studies using the PSAT [30, 37], information on the conditions related to program sustainment was collected via individual self-report on the eight PSAT scales: environmental support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation…”
BackgroundMethodResultInfluential
- Authors
- Stoll, S., Janevic, M., Lara, M., Ramos-Valencia, G., Stephens, T. B., Persky, V., Uyeda, K., Ohadike, Y., & Malveaux, F.
- Affiliations
- University of Michigan (US)
- RAND Corporation (US)
- University of Puerto Rico System (PR)
- Children's Hospital of Philadelphia (US)
- University of Illinois Chicago (US)
- Los Angeles Unified School District (US)
- Merck & Co., Inc., Rahway, NJ, USA (United States) (US)
- Corresponding author
- Stoll, Shelley
- Venue
- Preventing Chronic Disease
- Published
- 2015-12-03
- Volume
- 12
- Pages
- E214-E214
- ISSN
- 1545-1151 (ISSN-L)
- PMC
- PMC4674447
- OpenAlex
- W2187762524
- Cited by
- 32
- References
- 25
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 2 · 2022: 10 · 2021: 2 · 2020: 4 · 2018: 4 · 2017: 2 · 2016: 3
- FWCI
- 5.49
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- public-domain
- Locations
- Preventing Chronic Disease (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Public Health Policies and Education
- Keywords
- program sustainability, childhood asthma, care coordination, low-income communities, funding stability, strategic planning, program evaluation, implementation barriers, evidence-based interventions
- MeSH terms
- Asthma, Child, Child Health Services, Humans, Poverty, United States, Program Evaluation, Community Networks, Qualitative Research, Evidence-Based Practice, Capacity Building
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
