The DOI matches its Crossref record; 58 of the 59 references it cites were found in a registry or an index; 48 of the 70 sources its claims cite were checked for support.
Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention
Open accessAbstract
Background: In order to achieve wide-scale impact in community settings, programs must be sustained. Theory and empirical evidence suggest that intervention characteristics, organizational context, capacity for program implementation, and processes related to implementation are associated with continued program delivery. However, few studies examine how combinations of these factors work together in different settings to influence program sustainment. Methods: Using scales specified in the Program Sustainability Assessment Tool (PSAT), the current cross-sectional study aims to identify the necessary and sufficient conditions for the sustainment of the Strengthening Families Program for Parents and Youth 10-14 (SFP 10-14). Staff (n = 59) at SFP 10-14 implementation sites across Washington State completed an online survey reporting on their current level of SFP 10-14 sustainment. They also completed PSAT, with eight scales designed to assess conditions that consistently produce sustainment. Data were analyzed using qualitative comparative analysis. Results: Environmental support was the only necessary condition for sustainment success. Four solutions sufficient to achieve sustainment were also identified. These included the combined presence of (1) environmental support, organizational capacity, and funding stability; (2) environmental support, organizational capacity, communication, and program evaluation, in the absence of strategic planning; (3) environmental support, organizational capacity, program evaluation, and partnerships, in the absence of strategic planning; and (4) environmental support, communication, partnerships, and funding stability, in the absence of program evaluation. Conclusions: Environmental support in combination with organizational capacity appeared to most consistently produce sustainment of SFP 10-14 programs in Washington State. Program providers will benefit from a focusing on enhancing those conditions to increase program sustainment.
Overview
CoChat Summary
TL;DR
This cross-sectional crisp-set QCA study of 32 SFP 10-14 implementation staff in Washington State found environmental support was the only necessary condition for program sustainment and identified four sufficient combinations of PSAT conditions. Environmental support plus organizational capacity appeared in three of four pathways, suggesting multiple routes to sustainment.
Key findings
- Environmental support was the only necessary condition: 94% of cases with average or above environmental support reported routinized sustainment (consistency .94, coverage .84); the absence of environmental support was the only condition sufficient for sustainment failure.
- Four sufficient intermediate solutions were identified: (1) environmental support + organizational capacity + funding stability (consistency .90, raw coverage .53, unique coverage .41); (2) environmental support + organizational capacity + communication + program evaluation + absence of strategic planning (consistency 1.00, raw coverage .12); (3) environmental support + organizational capacity + program evaluation + partnerships + absence of strategic planning (consistency 1.00, raw coverage .06); and (4) environmental support + communication + partnerships + funding stability + absence of program evaluation (consistency 1.00, raw coverage .18).
- Of the 32 analytic cases, 14 (43.7%) achieved sustainment. Funding stability was not necessary and appeared in only two of four solutions; strategic planning absence appeared in two solutions; program evaluation appeared in two solutions but was required to be absent in one.
Methods
- Cross-sectional online survey conducted January–February 2015 with SFP 10-14 coordinators, facilitators, and other staff across Washington State; 119 contacts were emailed, 59 completed the survey (49.6%), and the final QCA analytic sample was 32 cases with complete data.
- Sustainment was measured with one item: Level 4 “routinized” was coded as sustained/present, while Levels 1–3 were coded absent. Conditions were the eight PSAT scales (five items each, 7-point response scale); program adaptation was excluded due to lack of face validity and empirical support, and remaining conditions were mean-split dichotomized.
- Data were analyzed using crisp-set qualitative comparative analysis, including truth-table construction, necessity analysis with a consistency threshold of 0.9, sufficiency analysis using the intermediate solution with a consistency cutoff of 0.8, and model fit assessed via solution coverage and consistency.
Limitations
- Stated: the small sample may have limited identification of all possible combinations; the QCA complete-data requirement and lack of missing-data techniques may bias results; dichotomization and mean-split calibration may obscure partial contributions and limit generalizability, though sensitivity analyses with other cutpoints were highly similar; single-reporter measurement of both conditions and sustainment, and possible multiple staff from the same site, may have inflated associations; implementer characteristics and other intervention characteristics were not included.
- Evident: the cross-sectional design cannot establish temporal or causal ordering, and the sample is limited to SFP 10-14 sites in Washington State, restricting generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
27 Valid · 1 Not valid · 7 Not checked · 58 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The program has demonstrated consistent positive impacts on youth behavior problems, delinquency, and alcohol and drug abuse in clinical trials [10–12].”
§ Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention > Background > Strengthening Families for Parents and Youth 10-14
What the sources say
- Partly supportedcites [10] Spoth RL, Randall GK, Trudeau L, Shin C, Redmond C. Substance use outcomes 5 ½ years past baseline for partnership-based, family-school preventive interventions. Drug Alcohol Depend. 2008;96(1–2):57–68. sourceExperimental — accuracy not yet measured
The Discussion (page not provided) reports positive substance-initiation outcomes for the combined LST + SFP 10–14 intervention, though similar outcomes occurred with LST alone. The paper identifies reducing other problem behaviors as an SFP 10–14 goal, but does not report trial outcomes for behavior problems or delinquency. It therefore does not establish consistent positive impacts across all the claimed outcomes for SFP 10–14 itself. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [11] Spoth R, Redmond C, Lepper H. Alcohol initiation outcomes of universal family-focused preventive interventions: one- and two-year follow-ups of a controlled study. J Stud Alcohol. 1999;60(Suppl. 13):103–11 Available from: https://www.jsad.com/doi/abs/10.15288/jsas.1999.s13.103 [cited 30 Jul 2020]. source
- Not checked — no full text of the sourcecites [12] Spoth R, Reyes ML, Redmond C, Shin C. Assessing a public health approach to delay onset and progression of adolescent substance use: latent transition and log-linear analyses of longitudinal family preventive intervention outcomes. J Consult Clin Psychol. 1999;67(5):619–30 Available from: /record/1999-11785-001 [cited 30 Jul 2020]. source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In the absence of program evaluation, the presence of the other four domains may have provided compensatory strength for program sustainment [30].”
§ Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention > Discussion
What the sources say
- Partly supportedcites [30] Stoll S, Janevic M, Lara M, Ramos-Valencia G, Stephens TB, Persky V, et al. A mixed-method application of the program sustainability assessment tool to evaluate the sustainability of 4 pediatric asthma care coordination programs. Prev Chronic Dis. 2015;12(12):1–10. sourceExperimental — accuracy not yet measured
The Discussion says programs may use strengths in one sustainability domain to bolster others. It does not describe an absence of program evaluation or identify four other domains as compensating for it; instead, it reports that evaluation was rated highly at all four sites. (LLM, conf=0.87)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Thus, the results support the assertion that strengthening other sustainability domains may help contribute to program sustainment even in the face of weak financial stability [27, 42].”
§ Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention > Discussion
What the sources say
- Partly supportedcites [27] Welsh JA, Chilenski SM, Johnson L, Greenberg MT, Spoth RL. Pathways to sustainability: 8-year follow-up from the PROSPER Project. J Prim Prev. 2016;37(3):263–86. sourceExperimental — accuracy not yet measured
The Discussion says infrastructure support and technical assistance helped PROSPER teams sustain programs and navigate threats to their viability. It does not show that strengthening these areas sustained programs despite weak financial stability; it instead reports that the two teams with the lowest fundraising discontinued prevention activities by Year 8. Our copy does not provide page numbers for these passages. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [42] King DK, Gonzalez SJ, Hartje JA, Hanson BL, Edney C, Snell H, et al. Examining the sustainability potential of a multisite pilot to integrate alcohol screening and brief intervention within three primary care systems. Transl Behav Med. 2018;8(5):776–84. source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“As was documented in a systematic review of 125 empirical studies of health-focused interventions, most sustainability research focuses on categorizing factors presumably associated with program continuation—known as program sustainment [15].”
§ Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention > Background > Factors associated with program sustainment
What the sources say
- Could not verifycites [15] Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7(1):1–19 Available from: http://www.implementationscience.com/content/7/1/17. sourceExperimental — accuracy not yet measured
The Methods (Data analysis) and Results (Associations between QI team factors and performance indicators) examine factors associated with the impact of mental-healthcare quality-improvement teams. Searches for a systematic review of 125 studies, sustainability, and program continuation found no passage in our copy addressing the attributed claim. (LLM, conf=0.77)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“This finding appears to be in line with a study by Tabak and colleagues [31] that found both sites with high and low sustainment capacity reporting limited funding stability.”
§ Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention > Discussion
What the sources say
- Supportedcites [31] Tabak RG, Duggan K, Smith C, Aisaka K, Moreland-Russell S, Brownson RC. Assessing capacity for sustainability of effective programs and policies in local health departments. J Public Heal Manag Pract. 2016;22(2):129–37 Available from: /pmc/articles/PMC4635064/?report=abstract [cited 28 Jul 2020]. source
In the Results section (“Funding Stability”), both high- and low-capacity local health departments described limited funding and limited funding duration, which constrained sustainability. The paper also notes that high-capacity departments had greater funding flexibility; that distinction does not undermine the citing sentence. (LLM, conf=0.94)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
46 more claims · 4 with their cited sources checked
Create a free account to see all 93 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 26
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Environmental support was the only condition that met this criterion, with 94% of the cases with average or above environmental support also reporting routinized SFP 10-14 sustainment (see Table 4).”
§ Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention > Results > Necessity analysis
Reported 94 · recomputed 84.210526
Citations
Cites
Sawyer, S. M., Azzopardi, P. S., Wickremarathne, D., & Patton, G. C. (2018) · The Lancet Child & Adolescent Health · doi:10.1016/s2352-4642(18)30022-1 · Cited by 3,666
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 1×
“The Exploration, Preparation, Implementation, and Sustainment (EPIS) model of implementation [50] and the Integrated Sustainability Framework [16] have recognized inner (e.”
Johnston, L., Miech, R., & O'Malley, P. (2020) · doi:10.3998/2027.42/162579 · Cited by 1,611
Standards of Good Practice in Qualitative Comparative Analysis (QCA) and Fuzzy-Sets
Schneider, C. Q., & Wagemann, C. (2010) · Comparative Sociology · doi:10.1163/156913210x12493538729793 · Cited by 1,473 · cited 5×
“8 is accepted as a suitable benchmark for initial inclusion in QCA analyses, a significantly greater value should be observed when making claims about the necessity of a condition [44].”
BackgroundMethodInfluentialWiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355
Economic costs of excessive alcohol consumption in the U.S., 2006
Bouchery, E. E., Harwood, H. J., Sacks, J. J., Simon, C. J., & Brewer, R. D. (2011) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2011.06.045 · Cited by 1,041
Qualitative Comparative Analysis using Fuzzy Sets (fsQCA)
Ragin, C. C. (2009) · Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques · doi:10.4135/9781452226569.n5 · Cited by 890 · cited 1×
“8 is recommended as a minimum guideline for claims of causality [48] and therefore is used in this study.”
MethodThe Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806
Self-citationAn Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · cited 2×
“Sustainability, or program sustainment, is defined as the “continued use of the program components and activities for the continued achievement of desirable program and population outcomes” [14].”
BackgroundPublic health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · cited 1×
“Conceptual framework The present study used the Capacity for Sustainability Framework and its resultant Program Sustainability Assessment Tool (PSAT) to guide data collection, analysis, and interpretation [28, 29].”
Method
Research works printed but not linked: 3
- National Academies of Sciences, Engineering, Medicine. Fostering healthy mental, emotional. In: and behavioral development in children and youth: a national agenda. Washington, DC: National Academies Press; 2019.
- U.S. Department of Health and Human Services (HHS). Facing addiction in America: the Surgeon General’s Report on alcohol, drugs, and health. Washington, DC; 2016. Available from: https://pubmed.ncbi.nlm.nih.gov/28252892/.
- Legewie N. An introduction to applied data analysis with qualitative comparative analysis. Forum Qual Soc Res/Forum Qual Sozialforsch. 2013;14(3):45 Available from: http://www.qualitative-research.net/ [cited 28 Jul 2020].
Cited by
Identification and development of implementation strategies: the important role of codesign
van der Laag, P. J., Dorhout, B. G., Heeren, A. A., Veenhof, C., Barten, D., & Schoonhoven, L. (2024) · Frontiers in Health Services · doi:10.3389/frhs.2024.1305955 · Cited by 17
“Moreover, the use of multicomponent strategies is highlighted by Cooper et al. (45), where various combinations of strategies were found effective for sustaining the implementation of an EBP (4, 43).”
BackgroundPars, E., Hirzalla, F., VanDerNagel, J., Dijkstra, B. A. G., & Schellekens, A. (2024) · Substance Abuse and Rehabilitation · doi:10.2147/sar.s447560 · Cited by 5
van der Laag, P. J., Dorhout, B., Veenhof, C., Barten, D., & Schoonhoven, L. (2025) · Global Implementation Research and Applications · doi:10.1007/s43477-025-00162-w · Cited by 2
“Moreover, because of the differences between settings, other or multiple barriers should be addressed (Cooper et al., 2022 ).”
Dopp, A. R., North, M. N., Gilbert, M., Ringel, J. S., Silovsky, J. F., Blythe, M., Edwards, D., Schmidt, S. R., & Funderburk, B. W. (2024) · Implementation Research and Practice · doi:10.1177/26334895241249394 · Cited by 4
“…multiple combinations of sustainment capacities were adequate for sustainment, depending on the context (consistent with our case examples, e.g., improving strategic planning was less relevant for the service agency that already had high sustainment capacity in other domains; Cooper et al., 2022).”
MethodMylonas, N., Scoles, L. H., Priyam, A., Rodgers, A., Holcroft, D., Carey, J., Myton, T., O'Donnell, A. J., Drummond, C., Lovell, K., et al. (2025) · Drugs Education Prevention and Policy · doi:10.1080/09687637.2025.2574263 · Cited by 2
“…precondition for implementation, research has shown that other domains of sustainability (organizational infrastructure, leadership and stakeholder involvement) are also needed, and can often compensate for the lack of stability in funding resources (Cooper et al., 2022; Scheirer & Dearing, 2011).”
Metadata
- Authors
- Cooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A.
- Affiliations
- Washington State University (US)
- Washington State Department of Health (US)
- Corresponding author
- Cooper, Brittany Rhoades
- Venue
- Implementation Science Communications
- Published
- 2022-05-26
- Volume
- 3
- Issue
- 1
- Pages
- 55-55
- ISSN
- 2662-2211 (ISSN-L)
- PMC
- PMC9137145
- OpenAlex
- W4281678781
- Cited by
- 7
- References
- 59
- Citations by year
- 2025: 3 · 2024: 4
- FWCI
- 2.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Implementation Science Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- State of Washington Initiative Measure No. 171
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Epidemiology › Substance Abuse Treatment and Outcomes
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- program sustainment, community-based prevention, Strengthening Families Program, environmental support, organizational capacity, funding stability, program evaluation, strategic planning, partnership, communication, necessary and sufficient conditions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention. https://app.cochat.ai/papers/10.1186/s43058-022-00303-4
- The paper
- Cooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A. (2022). Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention. Implementation Science Communications. https://doi.org/10.1186/s43058-022-00303-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.