System-Level Influences on the Sustainability of a Cognitive Therapy Program in a Community Behavioral Health Network
Open accessAbstract
OBJECTIVE: The purpose of this study was to examine influences on the sustainability of a program to implement an evidence-based psychotherapy in a mental health system. METHODS: Interviews with program administrators, training consultants, agency administrators, and supervisors (N=24), along with summaries of program evaluation data and program documentation, were analyzed with a directed content-analytic approach. RESULTS: Findings suggested a number of interconnected and interacting influences on sustainability, including alignment with emerging sociopolitical influences and system and organizational priorities; program-level adaptation and evolution; intervention flexibility; strong communication, collaboration, planning, and support; and perceived benefit. These individual factors appeared to mutually influence one another and contribute to the degree of program sustainability achieved at the system level. Although most influences were positive, financial planning and support emerged as potentially both facilitator and barrier, and evaluation of benefits at the patient level remained a challenge. CONCLUSIONS: Several factors appeared to contribute to the sustainability of a psychosocial intervention in a large urban mental health system and warrant further investigation. Understanding interconnections between multiple individual facilitators and barriers appears critical to advancing understanding of sustainability in dynamic systems and adds to emerging recommendations for other implementation efforts. In particular, implications of the findings include the importance of implementation strategies, such as long-term planning, coalition building, clarifying roles and expectations, planned adaptation, evaluation, diversification of financing strategies, and incentivizing implementation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Program evaluation has been consistently recognized in the literature as a key ingredient for program sustainment.
“Program evaluation has been consistently recognized in the literature as a key ingredient for program sustainment [18, 21, 56].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [21]
The Discussion (page unavailable) links evaluation to sustainability frameworks and suggests that evaluation processes may help sustain implementation programs. The Results also describe evaluation in this program. The paper does not establish the broader claim that the literature has consistently recognized program evaluation as a key ingredient for sustainment. (LLM, conf=0.83)
Experimental — accuracy not yet measuredMost prior sustainability studies included small, homogeneous samples and based conclusions mostly on qualitative retrospective assessments rather than statistical tests of association.
“However, this knowledge is greatly limited because most studies included small, homogeneous samples, and their conclusions were mostly based on qualitative retrospective assessments of individuals’ beliefs about what led to sustainment success or failure, rather than based on statistical tests of their association [17–22].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [21]
The Results describe a small sample of 24 administrators, consultants, and supervisors. Methods > Analysis describes qualitative coding of interviews and documents, while the Discussion says the design cannot establish associations with quantitative sustainability measures. This paper illustrates much of the cited limitation, but its participants held varied roles, some interviews occurred as training began, and a single study cannot establish that most prior studies had these characteristics. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThese results reinforce the multidimensional and interactive nature of sustainability capacity domains and their role in program sustainment.
“These results reinforce the multidimensional and interactive nature of sustainability capacity domains and their role in program sustainment [21, 27, 30].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601 · see citing works
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · see citing works
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 4× · see citing works
“Many findings, particularly those related to the outer context (sociopolitical influences, system needs, funding, and collaboration), as well as intervention fit and the need for ongoing support, were consistent with Aarons and colleagues’model (10), and our findings also support Pluye and colleagues’ (18) emphasis on…”
BackgroundMethodResultInfluentialThe dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · see citing works
Self-citationShediac-Rizkallah, M. C., & Bone, L. R. (1998) · Health Education Research · doi:10.1093/her/13.1.87 · Cited by 1,129 · see citing works
A compilation of strategies for implementing clinical innovations in health and mental health
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E., & York, J. L. (2012) · Medical Care Research and Review · doi:10.1177/1077558711430690 · Cited by 1,069 · see citing works
McHugh, R. K., & Barlow, D. H. (2010) · American Psychologist · doi:10.1037/a0018121 · Cited by 909 · see citing works
Making public health programs last: conceptualizing sustainability
Pluye, P., Potvin, L., & Denis, J.-L. (2004) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2004.01.001 · Cited by 374 · see citing works
Karlin, B. E., & Cross, G. (2014) · American Psychologist · doi:10.1037/a0033888 · Cited by 283 · see citing works
Measuring Use of Evidence Based Psychotherapy for Posttraumatic Stress Disorder
Shiner, B. R., DʼAvolio, L., Nguyen, T. M., Zayed, M. H., Young‐Xu, Y., Desai, R. A., Schnurr, P. P., Fiore, L. D., & Watts, B. V. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0421-0 · Cited by 140 · see citing works
Ehlers, A., Grey, N., Wild, J., Stott, R., Liness, S., Deale, A., Handley, R., Albert, I., Cullen, D., Hackmann, A., et al. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.08.006 · Cited by 134 · see citing works
Martin, G. P., Weaver, S., Currie, G., Finn, R., & McDonald, R. (2012) · Health Services Management Research · doi:10.1177/0951484812474246 · Cited by 116 · see citing works
Stewart, R. E., Adams, D. R., Mandell, D. S., Hadley, T. R., Evans, A. C., Rubin, R. M., Erney, J., Neimark, G. B., Hurford, M. O., & Beidas, R. S. (2015) · Psychiatric Services · doi:10.1176/appi.ps.201500392 · Cited by 86 · see citing works
Self-citationApplying the Policy Ecology Framework to Philadelphia’s Behavioral Health Transformation Efforts
Powell, B. J., Beidas, R. S., Rubin, R. M., Stewart, R. E., Wolk, C. B., Matlin, S. L., Weaver, S., Hurford, M. O., Evans, A. C., Hadley, T. R., et al. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0733-6 · Cited by 82 · see citing works
Self-citationUse of Evidence-Based Treatment for Posttraumatic Stress Disorder in Army Behavioral Healthcare
Wilk, J. E., West, J. C., Duffy, F. F., Herrell, R. K., Rae, D. S., & Hoge, C. W. (2013) · Psychiatry · doi:10.1521/psyc.2013.76.4.336 · Cited by 50 · see citing works
Stirman, S. W., Buchhofer, R. L., McLaulin, J. B., Evans, A. C., & Beck, A. T. (2009) · Psychiatric Services · doi:10.1176/ps.2009.60.10.1302 · Cited by 46 · see citing works
Self-citationNakamura, B. J., Chorpita, B. F., Hirsch, M., Daleiden, E. L., Slavin, L. A., Amundson, M. J., Rocco, S. A. L., Mueller, C. W., Osiecki, S., Southam‐Gerow, M. A., et al. (2011) · Clinical Psychology Science and Practice · doi:10.1111/j.1468-2850.2010.01231.x · Cited by 42 · see citing works
The Sustainability of Systems of Care for Children’s Mental Health: Lessons Learned
Stroul, B. A., & Manteuffel, B. A. (2007) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-007-9065-3 · Cited by 36 · see citing works
Meta-Inquiry: An Approach to Interview Success
Carlson, N., & McCaslin, M. (2015) · The Qualitative Report · doi:10.46743/2160-3715/2003.1867 · Cited by 24 · see citing works
Schoenwald, S. K., Garland, A. F., Southam‐Gerow, M. A., Chorpita, B. F., & Chapman, J. E. (2011) · Clinical Psychology Science and Practice · doi:10.1111/j.1468-2850.2011.01264.x · Cited by 20 · see citing works
Doing More with What We Know: Introduction to the Special Issue
Chorpita, B. F., & Daleiden, E. L. (2014) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2013.869751 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [23]Institute of Medicine Treatment for Posttraumatic Stress Disorder in Military and Veteran Populations: Final Assessment Washington, DC National Academies Press 2014 25077185
Cited by
Sorted by citations
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · see citing works
Self-citationRosen, C. S., Matthieu, M. M., Stirman, S. W., Cook, J. M., Landes, S. J., Bernardy, N. C., Chard, K. M., Crowley, J. J., Eftekhari, A., Finley, E. P., et al. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0755-0 · Cited by 140 · see citing works
Self-citationZurynski, Y. A., Ludlow, K., Testa, L., Augustsson, H., Herkes, J., Hutchinson, K., Lamprell, G., McPherson, E., Carrigan, A., Ellis, L. A., et al. (2023) · Implementation Science · doi:10.1186/s13012-023-01315-x · Cited by 109 · see citing works
Stirman, S. W., Gutner, C. A., Langdon, K., & Graham, J. R. (2016) · Behavior Therapy · doi:10.1016/j.beth.2015.12.001 · Cited by 103 · see citing works
Self-citationStewart, R. E., Adams, D. R., Mandell, D. S., Hadley, T. R., Evans, A. C., Rubin, R. M., Erney, J., Neimark, G. B., Hurford, M. O., & Beidas, R. S. (2015) · Psychiatric Services · doi:10.1176/appi.ps.201500392 · Cited by 86 · see citing works
Self-citationApplying the Policy Ecology Framework to Philadelphia’s Behavioral Health Transformation Efforts
Powell, B. J., Beidas, R. S., Rubin, R. M., Stewart, R. E., Wolk, C. B., Matlin, S. L., Weaver, S., Hurford, M. O., Evans, A. C., Hadley, T. R., et al. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0733-6 · Cited by 82 · see citing works
Self-citationA Systematic Review of the Literature on the Sustainability of Community Health Collaboratives
Hearld, L. R., Bleser, W. K., Alexander, J. A., & Wolf, L. J. (2015) · Medical Care Research and Review · doi:10.1177/1077558715607162 · Cited by 49 · see citing works
Sustaining alcohol and opioid use disorder treatment in primary care: a mixed methods study
Hunter, S. B., Ober, A. J., McCullough, C., Storholm, E. D., Iyiewuare, P. O., Pham, C., & Watkins, K. E. (2018) · Implementation Science · doi:10.1186/s13012-018-0777-y · Cited by 22 · see citing works
Lui, J. H. L., Brookman-Frazee, L. I., Lind, T., Le, K., Roesch, S. C., Aarons, G. A., Innes-Gomberg, D., Pesanti, K., & Lau, A. S. (2021) · Implementation Science · doi:10.1186/s13012-021-01149-5 · Cited by 22 · see citing works
Cooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00303-4 · Cited by 7 · see citing works
Chicoine, G., & Straus, S. E. (2025) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000496 · Cited by 7 · see citing works
A Model to Transform Psychosis Milieu Treatment Using CBT-Informed Interventions
Riggs, S. E., & Creed, T. A. (2016) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2016.08.001 · Cited by 7 · see citing works
Self-citationShenberger-Trujillo, J. M., & Kurinec, C. A. (2016) · Military Behavioral Health · doi:10.1080/21635781.2016.1181580 · Cited by 5 · see citing works
McCabe, L. A., Ruberti, M. R., & Endres, T. (2022) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2022.102049 · Cited by 5 · see citing works
A Commentary on Evidence-Based Public Policy to Promote Child Well-Being
Berliner, L. (2017) · Child and Adolescent Social Work Journal · doi:10.1007/s10560-017-0489-1 · Cited by 3 · see citing works
Zurynski, Y. A., Ludlow, K., Testa, L., Augustsson, H., Herkes, J., Hutchinson, K., Lamprell, G., McPherson, E., Carrigan, A., Ellis, L. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26639073
Zurynski, Y. A., Ludlow, K., Testa, L., Augustsson, H., Herkes, J., Hutchinson, K., Lamprell, G., McPherson, E., Carrigan, A., Ellis, L. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26639061
Zurynski, Y. A., Ludlow, K., Testa, L., Augustsson, H., Herkes, J., Hutchinson, K., Lamprell, G., McPherson, E., Carrigan, A., Ellis, L. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26639064
Zurynski, Y. A., Ludlow, K., Testa, L., Augustsson, H., Herkes, J., Hutchinson, K., Lamprell, G., McPherson, E., Carrigan, A., Ellis, L. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26639070
Zurynski, Y. A., Ludlow, K., Testa, L., Augustsson, H., Herkes, J., Hutchinson, K., Lamprell, G., McPherson, E., Carrigan, A., Ellis, L. A., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26639067
Barbosa, G. S. (2026) · Revista Psicologia Jurídica Forense · doi:10.56492/issn.27647374.2025n2a6
Voorhees, B. W. V., Glassgow, A. E., Martin, M. A., Touchette, D. R., Siegel, M., Gerges, M., Rasinski, K. A., & Caskey, R. N. (2026) · Healthcare · doi:10.1016/j.hjdsi.2026.100781
- Authors
- Stirman, S. W., Matza, A., Gamarra, J., Toder, K., Xhezo, R., Evans, A. C., Hurford, M., Beck, A. T., Crits-Christoph, P., & Creed, T.
- Affiliations
- Boston University (US)
- ORCID
- Venue
- Psychiatric Services
- Published
- 2015-07-01
- Volume
- 66
- Issue
- 7
- Pages
- 734-742
- ISSN
- 0022-1597 (ISSN-L)
- 1075-2730
- 1557-9700
- PMC
- PMC4490058
- OpenAlex
- W2008294770
- Cited by
- 20
- References
- 22
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 1 · 2022: 2 · 2021: 1 · 2018: 2 · 2017: 1 · 2016: 4 · 2015: 3
- FWCI
- 6.08
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Psychiatric Services (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R00MH080100, K99/R00 MH 080100, K99MH080100, R25MH080916)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Psychology › Clinical Psychology › Psychiatric care and mental health services
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Keywords
- program sustainability, cognitive therapy, evidence-based psychotherapy, implementation strategies, organizational priorities, sociopolitical influences, intervention adaptation, financial planning, coalition building
- MeSH terms
- Cooperative Behavior, Female, Health Personnel, Humans, Interviews as Topic, Male, Pennsylvania, Program Evaluation, Cognitive Behavioral Therapy, Community Networks, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
