Advancing the pragmatic measurement of sustainment: a narrative review of measures
Open accessAbstract
Background: Sustainment, an outcome indicating an intervention continues to be implemented over time, has been comparatively less studied than other phases of the implementation process. This may be because of methodological difficulties, funding cycles, and minimal attention to theories and measurement of sustainment. This review synthesizes the literature on sustainment measures, evaluates the qualities of each measure, and highlights the strengths and gaps in existing sustainment measures. Results of the review will inform recommendations for the development of a pragmatic, valid, and reliable measure of sustainment. Methods: A narrative review of published sustainment outcome and sustainability measures (i.e., factors that influence sustainment) was conducted, including appraising measures in the Society of Implementation Research Collaboration (SIRC) instrument review project (IRP) and the Dissemination and Implementation Grid-Enabled Measures database initiative (GEM-D&I). The narrative review used a snowballing strategy by searching the reference sections of literature reviews and definitions of sustainability and sustainment. Measures used frequently and judged to be comprehensive and/or validated by a team of implementation scientists were extracted for analysis. Results: Eleven measures were evaluated. Three of the included measures were found in the SIRC-IRP, three in the GEM-D&I database, (one measure was in both databases) and six were identified in our additional searches. Thirteen constructs relating to sustainment were coded from selected measures. Measures covered a range of determinants for sustainment (i.e., construct of sustainability) as well as constructs of sustainment as an outcome. Strengths of the measures included, development by expert panels knowledgeable about particular interventions, fields or contexts, and utility in specific scenarios. A number of limitations were found in the measures analyzed including inadequate assessment of psychometric characteristics, being overly intervention or context specific, being lengthy and/or complex, and focusing on outer context factors. Conclusion: There is a lack of pragmatic and psychometrically sound measures of sustainment that can be completed by implementation stakeholders within inner context settings (e.g., frontline providers, supervisors).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More quantitative prior studies rarely utilized validated instruments to measure predictors, and few existing studies model the interaction of factors and multiple pathways to sustainment.
“Those that were more quantitative in nature rarely utilized validated instruments to measure predictors [23], and despite acknowledgement of the likely interaction of these factors and multiple pathways to sustainment, few existing studies model this complexity [24, 25].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [23]
The citation to this paper [23] supports only the sentence’s first clause, not the later claim about modeling multiple pathways [24, 25]. The Background reports a lack of psychometrically sound, pragmatic sustainment measures, but the paper does not establish that quantitative studies rarely used validated instruments specifically to measure predictors. Its Discussion says that the majority of measures it reviewed were validated questionnaires. (LLM, conf=0.72)
Experimental — accuracy not yet measuredDespite an increase in the number of studies focusing on program sustainment, there remains a relative dearth of empirical evidence on the combinations of factors associated with sustainment, especially those guided by established sustainment frameworks and measurement tools.
“Despite an increase in the number of studies focusing on program sustainment, there remains a relative dearth of empirical evidence on the combinations of factors associated with sustainment [15, 16, 59], especially those guided by established sustainment frameworks and measurement tools [23].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [23]
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThere are still relatively few established measurement tools for assessing sustainability and sustainment.
“Despite this increased attention, there are still relatively few established measurement tools for assessing sustainability and sustainment [4, 5].”
From Lowering the burden: Shorter versions of the Program Sustainability Assessment Tool (PSAT) and Clinical Sustainability Assessment Tool (CSAT) (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 76 · Cited by 72
Research works printed but not linked: 2
- [31]Society for Implementation Research Collaboration. The SIRC Instrument Review Project (IRP): a systematic review and synthesis of implementation science instruments. https://societyforimplementationresearchcollaboration.org/sirc-instrument-project/.
- [40]Stirman SW, Kimberly J, Cook N, Calloway A, Castro F. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7(17). PMC3317864 22417162
- Authors
- Moullin, J. C., Sklar, M., Green, A., Dickson, K. S., Stadnick, N. A., Reeder, K., & Aarons, G. A.
- Affiliations
- Curtin University (AU)
- University of California San Diego (US)
- San Diego State University (US)
- Corresponding author
- Aarons, Gregory A.
- ORCID
- Venue
- Implementation Science Communications
- Published
- 2020-09-17
- Volume
- 1
- Issue
- 1
- Pages
- 76-76
- ISSN
- 2662-2211 (ISSN-L)
- PMC
- PMC7499830
- OpenAlex
- W3087727097
- Cited by
- 72
- References
- 76
- Citations by year
- 2026: 5 · 2025: 12 · 2024: 18 · 2023: 13 · 2022: 20 · 2021: 4
- FWCI
- 1.76
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Institute of Mental Health (R03MH117493-01, K23MH115100, K23MH110602)
- National Institute on Drug Abuse (R01DA038466, R01DA049891)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- program sustainability, sustainability, reliability and validity, psychometric properties, implementation research, frontline health workers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
