Predicting the Long-Term Sustainability of Evidence-Based Practices in Mental Health Care: An 8-Year Longitudinal Analysis
Abstract
Few studies have examined predictors of long-term sustainability of evidence-based practices in mental health. This study used assessments of five evidence-based practices implemented in 49 sites in eight states at baseline and years 2, 4, and 8. Program characteristics, implementation characteristics, reinforcement activities, and sustainability factors were used to predict program survival status. The majority of predictors were not significant. Supervisor turnover in year 4 predicted survival status in year 8, but site characteristics, fidelity at implementation, quality improvement activities, and post-implementation activities had little impact on long-term program survival. This study extends previous sustainability research by examining the long-term impact of internal program factors over a substantial period of time using longitudinal prediction. Future research should also consider the influence of external factors such as financial policies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More quantitative prior studies rarely utilized validated instruments to measure predictors, and few existing studies model the interaction of factors and multiple pathways to sustainment.
“Those that were more quantitative in nature rarely utilized validated instruments to measure predictors [23], and despite acknowledgement of the likely interaction of these factors and multiple pathways to sustainment, few existing studies model this complexity [24, 25].”
Prospective, multilevel, and mixed-methods study designs are ideal for studying sustainability.
“Prospective, multilevel, and mixed-methods study designs are ideal for studying sustainability (94, 98).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [98]
Many sustainability studies in clinical/health care and mental health settings suggest that funding, organizational factors and support, and practitioner/workforce characteristics are particularly influential.
“Consistent with organizational contexts in community settings (e.g., schools), many of these studies, particularly those in clinics or hospitals, suggest that funding, organizational factors and support (e.g., champions, supervision), and practitioner/workforce characteristics (e.g., turnover) are particularly influential (13, 98, 125).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [98]
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · Routledge eBooks · doi:10.4324/9780203710753-35 · Cited by 37,557 · see citing works
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,555 · see citing works
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
(2005) · Cited by 3,748 · see citing works
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · see citing works
Glanz, K., & Bishop, D. B. (2010) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.012809.103604 · Cited by 2,336 · see citing works
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · see citing works
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · see citing works
Shediac-Rizkallah, M. C., & Bone, L. R. (1998) · Health Education Research · doi:10.1093/her/13.1.87 · Cited by 1,129 · see citing works
A guiding framework and approach for implementation research in substance use disorders treatment.
Damschroder, L. J., & Hagedorn, H. J. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022284 · Cited by 427 · see citing works
Building capacity and sustainable prevention innovations: a sustainability planning model
Johnson, K., Hays, C., Center, H., & Daley, C. (2004) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2004.01.002 · Cited by 411 · see citing works
Sustainability science: an integrated approach for health-programme planning
Gruen, R. L., Elliott, J. H., Nolan, M. L., Lawton, P. D., Parkhill, A., McLaren, C. J., & Lavis, J. N. (2008) · The Lancet · doi:10.1016/s0140-6736(08)61659-1 · Cited by 395 · see citing works
Making public health programs last: conceptualizing sustainability
Pluye, P., Potvin, L., & Denis, J.-L. (2004) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2004.01.001 · Cited by 374 · see citing works
Fidelity Outcomes in the National Implementing Evidence-Based Practices Project
McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., Goldman, H. H., Lutz, W. J., & Finnerty, M. T. (2007) · Psychiatric Services · doi:10.1176/ps.2007.58.10.1279 · Cited by 315 · see citing works
Self-citationThe Sustainability of Evidence-Based Practices in Routine Mental Health Agencies
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2009) · Community Mental Health Journal · doi:10.1007/s10597-009-9202-y · Cited by 193 · see citing works
Self-citationLong-Term Sustainability of Evidence-Based Practices in Community Mental Health Agencies
Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0461-5 · Cited by 172 · see citing works
Self-citationWhat Is Implementation Research?
Bhattacharyya, O. K., Reeves, S., & Zwarenstein, M. (2009) · Research on Social Work Practice · doi:10.1177/1049731509335528 · Cited by 162 · see citing works
Massatti, R. R., Sweeney, H. A., Panzano, P. C., & Roth, D. (2007) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0141-z · Cited by 153 · see citing works
Program sustainability begins with the first events
Pluye, P., Potvin, L., Denis, J.-L., Pelletier, J., & Mannoni, C. (2005) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2004.10.003 · Cited by 147 · see citing works
Implementing Evidence-Based Practices for People With Schizophrenia
Drake, R. E., Bond, G. R., & Essock, S. M. (2009) · Schizophrenia Bulletin · doi:10.1093/schbul/sbp041 · Cited by 119 · see citing works
Self-citationMancini, A. D., Moser, L. L., Whitley, R., McHugo, G. J., Bond, G. R., Finnerty, M. T., & Burns, B. J. (2009) · Psychiatric Services · doi:10.1176/ps.2009.60.2.189 · Cited by 117 · see citing works
Self-citationLife Histories of Innovations: How New Practices Become Routinized
Yin, R. K. (1981) · Public Administration Review · doi:10.2307/975720 · Cited by 113 · see citing works
Torrey, W. C., Bond, G. R., McHugo, G. J., & Swain, K. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0357-9 · Cited by 108 · see citing works
Self-citationMcFarlane, W. R., McNary, S. W., Dixon, L. B., Hornby, H., & Cimett, E. (2001) · Psychiatric Services · doi:10.1176/appi.ps.52.7.935 · Cited by 87 · see citing works
Moser, L. L., DeLuca, N. L., Bond, G. R., & Rollins, A. L. (2004) · CNS Spectrums · doi:10.1017/s1092852900009780 · Cited by 81 · see citing works
Self-citationTorrey, W. C., Lynde, D., & Gorman, P. (2005) · Child and Adolescent Psychiatric Clinics of North America · doi:10.1016/j.chc.2004.05.004 · Cited by 56 · see citing works
Best Practices: A National Mental Health Learning Collaborative on Supported Employment
Becker, D. R., Drake, R. E., Bond, G. R., Nawaz, S., Haslett, W. R., & Martínez, R. A. (2011) · Psychiatric Services · doi:10.1176/ps.62.7.pss6207_0704 · Cited by 44 · see citing works
Self-citationStrategies for implementing evidence-based practices in routine mental health settings
Drake, R. E., Torrey, W. C., & McHugo, G. J. (2003) · Evidence-Based Mental Health · doi:10.1136/ebmh.6.1.6 · Cited by 42 · see citing works
Self-citationCritical strategies for implementing supported employment
Becker, D. R., Baker, S. R., Carlson, L. S., Flint, L., Howell, R. F., Lindsay, S., Moore, M., Reeder, S. A., & Drake, R. E. (2007) · Journal of Vocational Rehabilitation · doi:10.3233/jvr-2007-00380 · Cited by 38 · see citing works
Self-citationThe Experts Rate: Supervisory Behaviors That Impact the Implementation of Evidence-Based Practices
Carlson, L. S., Rapp, C. A., & Eichler, M. S. (2010) · Community Mental Health Journal · doi:10.1007/s10597-010-9367-4 · Cited by 36 · see citing works
Mancini, A., Moser, L. L., Whitley, R., McHugo, G. J., Bond, G. R., Finnerty, M. T., & Burns, B. J. (2009) · Psychiatric Services · doi:10.1176/appi.ps.60.2.189 · Cited by 33 · see citing works
Self-citationThe Mental Health Treatment Study.
Frey, W. D., Azrin, S. T., Goldman, H. H., Kalasunas, S. A., Salkever, D. S., Miller, A. L., Bond, G. R., & Drake, R. E. (2008) · Psychiatric Rehabilitation Journal · doi:10.2975/31.4.2008.306.312 · Cited by 30 · see citing works
Self-citationBond, G. R., Drake, R. E., Rapp, C. A., McHugo, G. J., & Xie, H. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-009-0226-y · Cited by 30 · see citing works
Self-citationBjorklund, R. W., Monroe‐DeVita, M., Reed, D., Toulon, A., & Morse, G. A. (2008) · Psychiatric Services · doi:10.1176/appi.ps.60.1.24 · Cited by 25 · see citing works
Wilson, D. C., & Crisanti, A. S. (2008) · Community Mental Health Journal · doi:10.1007/s10597-008-9167-2 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · see citing works
Proctor, E., Luke, D., Calhoun, A., McMillen, C., Brownson, R., McCrary, S., & Padek, M. (2015) · Implementation Science · doi:10.1186/s13012-015-0274-5 · Cited by 429 · see citing works
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2017) · Implementation Science · doi:10.1186/s13012-017-0606-8 · Cited by 56 · see citing works
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2015) · Implementation Science · doi:10.1186/s13012-015-0364-4 · Cited by 58 · see citing works
Johnson, K., Collins, D., Shamblen, S., Kenworthy, T., & Wandersman, A. (2017) · Prevention Science · doi:10.1007/s11121-017-0784-2 · Cited by 23 · see citing works
Popowich, A. D., Mushquash, A. R., Pearson, E., Schmidt, F., & Mushquash, C. J. (2020) · Counselling and Psychotherapy Research · doi:10.1002/capr.12250 · Cited by 15 · see citing works
Miller, C. J., Kim, B., Connolly, S. L., Spitzer, E. G., Brown, M., Bailey, H. M., Weaver, K., & Sullivan, J. L. (2023) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01231-0 · Cited by 10 · see citing works
Cooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00303-4 · Cited by 7 · see citing works
“Those that were more quantitative in nature rarely utilized validated instruments to measure predictors [23], and despite acknowledgement of the likely interaction of these factors and multiple pathways to sustainment, few existing studies model this complexity [24, 25].”
Background
- Authors
- Peterson, A. E., Bond, G. R., Drake, R. E., McHugo, G. J., Jones, A. M., & Williams, J. R.
- Affiliations
- Dartmouth Psychiatric Research Center (US)
- Manila Consulting Group (United States) (US)
- University of Miami (US)
- Corresponding author
- Williams, Jessica R.
- Venue
- The Journal of Behavioral Health Services & Research
- Published
- 2013-05-25
- Volume
- 41
- Issue
- 3
- Pages
- 337-346
- ISSN
- 1094-3412 (ISSN-L)
- 1556-3308
- OpenAlex
- W2001681776
- Cited by
- 60
- References
- 35
- Citations by year
- 2026: 1 · 2024: 8 · 2023: 2 · 2022: 4 · 2021: 12 · 2020: 4 · 2019: 6 · 2018: 3 · 2017: 9 · 2016: 3 · 2015: 5 · 2014: 3
- FWCI
- 6.61
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- The Journal of Behavioral Health Services & Research (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Substance Abuse and Mental Health Services Administration
- U.S. Public Health Service (HHSS283200700040I/HHSS28342003T)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Psychology › Clinical Psychology › Psychiatric care and mental health services
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- evidence-based practice, sustainability, mental health services, program survival, implementation fidelity, quality improvement, program characteristics, long-term forecasting
- MeSH terms
- Community Mental Health Services, Diffusion of Innovation, Forecasting, Humans, Longitudinal Studies, Program Evaluation, Outcome Assessment, Health Care, Guideline Adherence, Quality Indicators, Health Care, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
