The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index; 17 of the 22 sources its claims cite were checked for support.
Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT)
Open accessAbstract
Background: Although significant advances have been made in the conceptualization of sustainability, having pragmatic, psychometrically valid tools remains a need within the field. Our previous work has developed frameworks and tools to assess both program sustainability and clinical sustainability capacity. This work presents new, psychometrically tested short versions of the Program Sustainability Assessment Tool (PSAT) and the Clinical Sustainability Assessment Tool (CSAT). Methods: These methods were conducted in identical, parallel processes for the CSAT and PSAT. Previously collected data for these instruments was obtained across a variety of settings, contexts, and participants. We first conducted testing to determine cronbach’s alpha of shortened domains (3 items each) and then conducted Confirmatory Factor Analysis to ensure that the domains were still appropriate for the tool. After, the team met to review the results and determine the final versions of the short PSAT and short CSAT. Results: The short PSAT retained cronbach’s alpha’s of 0.82 – 0.91 for each domain of the tool, with which maintains excellent reliability for the tool. Confirmatory factor analysis highlights that the short PSAT retains conceptual distinction across the 8 domains, with CFI scores greater than 0.90, RMSEA scores below 0.6, and SRMR scores less than 0.08. The short CSAT had cronbach’s alpha of 0.84 – 0.92 for each of the domains of the tool, also suggesting excellent reliability of the domains within the measure after dropping two items/domain. Confirmatory factor analysis of the short CSAT meets the same specifications as above, again highlighting conceptual distinction across the domains. Conclusion: Each tool was able to be shortened to three items per domain while maintaining strong psychometric properties. This results in a tool that takes less time to complete, meeting one of the key calls for pragmatic measures within implementation science. This advances our abilities to measure and test sustainability within implementation science.
Overview
CoChat Summary
TL;DR
This brief report developed and psychometrically tested 3-item-per-domain short forms of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT), each 40% shorter than the original long forms. Both short tools retained strong reliability and acceptable confirmatory factor analysis fit, supporting less burdensome sustainability capacity assessment.
Key findings
- The short PSAT retained all 8 domains with 3 items each, resulting in a 24-item measure that is 40% shorter than the original. Cronbach’s alphas ranged from .825 to .912 across domains, and CFA fit for the short version was CFI = .969, RMSEA = .040, SRMR = .036 (N = 5,706).
- The short CSAT retained all 7 domains with 3 items each, resulting in a 21-item measure that is 40% shorter than the original. Cronbach’s alphas ranged from .842 to .924 across domains, and CFA fit for the short version was CFI = .946, RMSEA = .056, SRMR = .045 (N = 501).
- Both short versions maintained conceptual distinction across their domains, with the authors concluding they are reliable, precise, and pragmatic measures. The authors state the short forms are not intended to replace the longer versions, which may be preferable when comprehensive or nuanced domain assessment is needed.
Methods
- This was a parallel psychometric study using previously collected data. PSAT data came from sustaintool.org for 2014–2019 (N = 5,706); CSAT data combined website responses from 2018–2022 with prior studies of antimicrobial stewardship, cancer screening, and smoking cessation (N = 501).
- For each tool, the team calculated Cronbach’s alpha for the original 5-item domains, identified the two lowest item-total correlation items per domain, reviewed item drops for conceptual importance, recalculated alpha for the new 3-item domains, and conducted confirmatory factor analysis comparing long and short forms. Analyses were conducted in R, with “not applicable” treated as missing and test/incomplete data removed.
Limitations
- The authors state that all data were collected using the longer versions of the tools for other purposes, and future work should test data collected on both versions with a new sample.
- The authors state the short versions are not intended to replace the longer versions; some scenarios, especially cross-sectional designs, may call for the original, more comprehensive items.
- The text makes evident that the CSAT analyses rely on a smaller, combined sample (N = 501) from website and prior studies, while the PSAT uses a large online sample (N = 5,706); the authors also note that “stakeholder” terminology was replaced and future psychometric analyses with the new language are planned.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 2 Not valid · 1 Not checked · 28 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Both measures had been previously developed with established reliability and validity, and have been widely used across a variety of contexts and settings.”
§ Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) > Introduction
What the sources say
- Partly supportedcites [7] Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The program sustainability assessment tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11: 130184. sourceExperimental — accuracy not yet measured
The Discussion reports that the PSAT was reliable, and the Results report good internal consistency and preliminary evidence of validity. However, the Methods say the pilot study was not designed as an in-depth validation study, so describing validity as established overstates the evidence in this paper. (LLM, conf=0.86)
- Disputedcites [8] Schell SF, Luke DA, Schooley MW, et al. Public health program capacity for sustainability: a new framework. Implement Sci. 2013;8: 15. sourceExperimental — accuracy not yet measured
The Results (Literature review; page unavailable) say that existing program-sustainability tools had not been successfully tested for reliability or validity. The Conclusions (page unavailable) describe developing and validating a tool based on this framework as future work, rather than reporting an already validated PSAT. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Both measures had been previously developed with established reliability and validity, and have been widely used across a variety of contexts and settings.”
§ Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) > Introduction
What the sources say
- Not matchedcites [6] Malone S, Prewitt K, Luke D. Assessing clinical sustainability: a new, user-friendly tool for evaluating real-world practices. In American Evaluation Association’s annual conference. 2019.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [9] Agulnik A, Malone S, Puerto-Torres M, et al. Reliability and validity of a Spanish-language measure assessing clinical capacity to sustain Paediatric Early Warning Systems (PEWS) in resource-limited hospitals. BMJ Open. 2021;11(10): e053116. sourceExperimental — accuracy not yet measured
The Methods (CSAT adaptation; page not provided) state that the CSAT had previously been developed and tested, with excellent internal consistency. They describe only trends toward discriminant validity, however, so the claim that its validity was already established overstates the evidence. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“reduce the burden of assessment and facilitate measurement of sustainability capacity in research, evaluation, and practice setting [5, 17, 18, 22].”
§ Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) > Discussion
What the sources say
- Partly supportedcites [5] Hall A, Shoesmith A, Doherty E, et al. Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review. Implement Sci. 2022;17(1):81. sourceExperimental — accuracy not yet measured
The Introduction says the review provides guidance for selecting sustainability measures, and the Discussion > Limitations describes quantitative measures as an efficient way to measure sustainability and its determinants. The Results > Psychometric and pragmatic qualities lists short versions, but the paper does not establish that those versions specifically facilitate measurement of sustainability capacity across research, evaluation, and practice settings. No page numbers are available in our copy. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [17] Glasgow RE, Riley WT. Pragmatic Measures: What They Are and Why We Need Them. Am J Prev Med. 2013;45(2):237–43. source
- Partly supportedcites [18] Powell BJ, Stanick CF, Halko HM, et al. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118. sourceExperimental — accuracy not yet measured
The Background and Methods > Data analysis passages identify low burden and brevity as criteria for pragmatic measures, which supports the general rationale for reducing assessment burden. They do not discuss short versions of a sustainability-capacity measure or show that such versions facilitate its use in research, evaluation, and practice settings. No page number is available in our copy. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Despite this increased attention, there are still relatively few established measurement tools for assessing sustainability and sustainment [4, 5].”
§ Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) > Introduction
What the sources say
- Supportedcites [4] Moullin JC, Sklar M, Green A, et al. Advancing the pragmatic measurement of sustainment: a narrative review of measures. Implement Sci Commun. 2020;1(1):76. source
The Conclusion (page not provided) says there is a lack of pragmatic, psychometrically sound measures and a limited number of measures specifically addressing sustainment. This supports the claim that relatively few established tools are available for assessing sustainability and sustainment. (LLM, conf=0.90)
- Supportedcites [5] Hall A, Shoesmith A, Doherty E, et al. Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review. Implement Sci. 2022;17(1):81. source
The Discussion reports that only a small number of sustainability measures were consistently used in empirical studies. Its section on sustainability as an outcome says only three of the 28 included measures assessed that outcome, supporting the claim that relatively few established tools assess sustainability or sustainment. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Short version (3 items) | 5706 | 9.9 | .969 | .040 | .036”
§ Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) > Results > Program sustainability assessment tool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
6 more claims
Create a free account to see all 40 claims and every source verdict.
- Probably not a substantial claim. 20
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
Citations
Cites
Factor analysis in the development and refinement of clinical assessment instruments.
Floyd, F. J., & Widaman, K. F. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.286 · Cited by 3,970 · cited 1×
“The CFI scores greater than 0.90, RMSEA scores below 0.6, and SRMR scores less than 0.08 suggest that both the long and short version of the PSAT have excellent fit [21].”
BackgroundInfluentialWiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · cited 3×
“This work will aid in the development of interventions and strategies that respond to identified gaps based on these measures, which will serve to further advance sustainability of evidence informed programs and practices [1].”
BackgroundInfluentialPublic health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · cited 1×
“Our team developed two theoretically informed measures of sustainability capacity, the Program Sustainability Assessment Tool (PSAT) [7, 8], designed to assess sustainability in public health settings, and the Clinical Sus-tainability Assessment Tool (CSAT) [6, 9], designed to assess sustainability…”
MethodSelf-citationThe Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364 · cited 1×
“Our team developed two theoretically informed measures of sustainability capacity, the Program Sustainability Assessment Tool (PSAT) [7, 8], designed to assess sustainability in public health settings, and the Clinical Sus-tainability Assessment Tool (CSAT) [6, 9], designed to assess sustainability…”
MethodSelf-citationGlasgow, R. E., & Riley, W. T. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.010 · Cited by 318
Braithwaite, J., Ludlow, K., Testa, L., Herkes, J., Augustsson, H., Lamprell, G., McPherson, E., & Zurynski, Y. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-036453 · Cited by 147
Powell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C. (2017) · Implementation Science · doi:10.1186/s13012-017-0649-x · Cited by 144 · cited 3×
“We were able to reduce the number of items of both measures while maintaining strong psychometric properties, and, in doing so, reduce the burden of assessment and facilitate measurement of sustainability capacity in research, evaluation, and practice setting [5, 17, 18, 22].”
BackgroundStanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117 · cited 3×
“While the PAPERS scale has a cutoff of 10 items for a short tool [22], reducing the PSAT and CSAT still reduces the burden of assessment.”
BackgroundMethodUsing the Program Sustainability Assessment Tool to Assess and Plan for Sustainability
Calhoun, A., Mainor, A., Moreland-Russell, S., Maier, R. C., Brossart, L., & Luke, D. A. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130185 · Cited by 100 · cited 2×
“The PSAT and CSAT have been used in organizational evaluation as well as within public health and implementation science research [10–12].”
BackgroundMethodSelf-citation
Research works printed but not linked: 1
- Malone S, Prewitt K, Luke D. Assessing clinical sustainability: a new, user-friendly tool for evaluating real-world practices. In American Evaluation Association’s annual conference. 2019.
Cited by
Smith, J. D., Carroll, A. J., Tedla, Y. G., Sanuade, O. A., Merle, J. L., Heinrich, J., Bannon, J., Abramsohn, E., Ahmad, F. S., Lazar, D., et al. (2025) · Implementation Science · doi:10.1186/s13012-025-01431-w · Cited by 8
Wang, C., Sakita, F. M., Sumner, S. F., Shayo, F. M., Martin, Z., Msangi, W., Munisi, J. J., Mulesi, E. M., Aboud, A. M., Bettger, J. P., et al. (2026) · Implementation Science Communications · doi:10.1186/s43058-026-00860-y · Cited by 2
“The tool includes 21 items across seven domains: Engaged Staff and Leadership, Engaged Stakeholders, Organizational Context and Capacity, Workflow Integration, Implementation and Training, Monitoring and Evaluation, and Outcomes and Effectiveness [17, 23].”
InfluentialMerle, J. L., Kuo, M. C., Gacki‐Smith, J., Agrawal, A. H., Friedewald, J. J., Gordon, E. J., & Smith, J. D. (2025) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-025-04068-8 · Cited by 1
Raniti, M. B., Logan, N., Reavley, N. J., & Neelakantan, L. (2026) · Mental Health & Prevention · doi:10.1016/j.mhp.2026.200476 · Cited by 1
Merle, J. L., Kuharić, M., Cella, D. F., Mitchell, S. A., Austin, J. D., Ridgeway, J. L., Hassett, M. J., Paudel, R., Flores, A. M., DiMartino, L., et al. (2026) · Implementation Science Communications · doi:10.1186/s43058-026-00867-5 · Cited by 1
“Each domain is evaluated by 7 items; however, a recently validated short form of the CSAT (CSAT Short) reduced the number of items to 3 per domain [24].”
Influential
Metadata
- Authors
- Malone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A.
- Affiliations
- Washington University in St. Louis (US)
- Corresponding author
- Malone, Sara
- Venue
- Implementation Science Communications
- Published
- 2024-10-10
- Volume
- 5
- Issue
- 1
- Pages
- 113-113
- ISSN
- 2662-2211 (ISSN-L)
- PMC
- PMC11468075
- OpenAlex
- W4403288069
- Semantic Scholar
- eebea8e7d9afc5000dc521251cd8e532bf078653
- Cited by
- 12
- References
- 20
- Citations by year
- 2026: 7 · 2025: 5
- FWCI
- 7.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Implementation Science Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Cancer Institute (P50CA244431, R37CA276215)
- Centers for Disease Control and Prevention (75D301-20-R-68063)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- program sustainability, short-form instrument, psychometric validation, reliability statistics, factor structure, burden reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT). https://app.cochat.ai/papers/10.1186/s43058-024-00656-y
- The paper
- Malone, S., Prewitt, K., McKay, V., Zabotka, L., Bacon, C., & Luke, D. A. (2024). Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT). Implementation Science Communications. https://doi.org/10.1186/s43058-024-00656-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.