The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research
Open accessThe DOI matches its Crossref record; 210 of the 211 references it cites were found in a registry or an index.
Abstract
Background: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research. Methods: Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized. Results: Although "sustainability" was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves. Conclusions: Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“findings indicating that incomplete penetration and sustainment are the norm (Stirman et al., 2012)”
From User‐centered design for psychosocial intervention development and implementation. (2016) · cites this paper as Stirman et al., 2012
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A number of literature reviews of implementation frameworks have also been conducted, concentrating on particular implementation concepts, such as a particular stage, or specifically on either the factors, strategies or evaluations, rather than addressing all the concepts that could affect an innovation’s implementation [13,17,19-26].”
From A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework (2015) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite the significant investment of staff time and other resources, many promising initiatives fail to sustain and do not produce long term benefits [1–6].”
From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overall, fewer than half of the programs continued at high levels of fidelity.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [135]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a systematic review of 125 empirical studies focused on the sustainability of public health and clinical interventions (135), researchers found that even when full implementation was originally achieved, partial sustainability was much more common than the continued implementation of the entire intervention.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [135]
Experimental — accuracy not yet measured
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–210 of 210
Lichtenstein, E. H., Thompson, B., Nettekoven, L., & Corbett, K. K. (1996) · Health Education Research · doi:10.1093/her/11.4.527 · Cited by 13 · cited 3×
“Figure 1 is a modified PRISMA (which stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses [33]) diagram summarizing the selection process, which includes reasons that potentially relevant papers were excluded.”
Hii, J., Chee, K. C., Vun, Y. S., Awang, J., Chin, K., & Kan, S. K. (1996) · PubMed · Cited by 12
Attributes of an Independently Self-sustaining Implementation
Knapp, H., Anaya, H. D., & Goetz, M. B. (2010) · Quality Management in Health Care · doi:10.1097/qmh.0b013e3181fa06f8 · Cited by 8
INATTENTION TO THE FIDELITY OF HEALTH CARE DELIVERY IS COSTING LIVES
Woolf, S. H., & Johnson, R. E. (2007) · American Journal of Public Health · doi:10.2105/ajph.2007.116707 · Cited by 8
Evaluation of Community-Based Health Projects: The Healthy Tomorrows Experience
Ruch‐Ross, H. S., Keller, D., Miller, N., Bassewitz, J. B., & Melinkovich, P. (2008) · PEDIATRICS · doi:10.1542/peds.2006-3273 · Cited by 7 · cited 3×
“Figure 1 is a modified PRISMA (which stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses [33]) diagram summarizing the selection process, which includes reasons that potentially relevant papers were excluded.”
Rapoport, A. (2011) · Education Research International · doi:10.1155/2011/591276 · Cited by 5 · cited 3×
“Figure 1 is a modified PRISMA (which stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses [33]) diagram summarizing the selection process, which includes reasons that potentially relevant papers were excluded.”
Kellie, S. M., Timmins, A., & Brown, C. (2011) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(11)37019-5 · Cited by 4
The Effects of a Classroom-based Intervention on Aggression-related Injuries
DeWein, M., & Miller, L. K. (2009) · Child & Youth Care Forum · doi:10.1007/s10566-009-9076-8 · Cited by 3
Cultural variations and the morphology of innovation
Kimberly, J. R., & Rye, C. B. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511581151.009 · Cited by 1
Self-citationRau, R., Rumpeltin, C., Hoop, R., Pfeiffer, H., Drees, J., Paas, B., Schmitz-Buhl, G., & Geraedts, M. (2009) · Journal of Public Health · doi:10.1007/s10389-009-0278-1 · Cited by 1 · cited 3×
“Figure 1 is a modified PRISMA (which stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses [33]) diagram summarizing the selection process, which includes reasons that potentially relevant papers were excluded.”
Research works printed but not linked: 21
- Greenhalgh T How to Spread Good Ideas: A systematic review of the literature on diffusion, dissemination, and sustainability of innovations in health service delivery and organization, in Report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R & D (NCCSDO) 2004 London: University College 362
- Maher L Gustafson DA Sustainability Model and Guide: Evans; 2007
- Stirman SW Sustainability: A systematic review of methods and conceptualizations. in Fourth Annual NIH Conference on the Science of Dissemination and Implementation 2011 Bethesda, MD
- Scheirer MA J Dearing An Agenda for Research on the Sustainability of Public Health Programs 2011 American Journal of Public Health in press PMC3222409 21940916
- Kimberly JR Starbuck WH Nystrom PC Managerial Innovation, in Handbook of Organizational Design 1981 New York: Oxford University Press 84 104
- Claquin P Sustainability of EPI: Utopia or Sine Qua Non Condition of Child Survival, in Resources for Child Health Project 1989 Arlington; REACH
- Lafond AK Improving the quality of investment in health: lessons on sustainability Health Policy Plan 1995 10 63 76
- Ovretveit J Making temporary quality improvement continuous 2003 Stockholm: Swedish Association of County Councils
- SW S CF R McMackin, et al Moving From the Laboratory to the Real World: Implementation Science and Effectiveness in Practice Settings, in Toward an Integrated Approach to Trauma Focused Therapy: Placing Evidence-Based Interventions in an Expanded Psychological Context American Psychological Association: Washington, D.C in press
- Davies B Tremblay D Edwards N Bick D, Ingram ID Sustaining evidence-based practice systems and measuring the impacts, in Evaluating the Impact of Implementing Evidence-Based Practice 2010 Oxford: Wiley Blackwell
- Blasinsky M Goldman HH Unutzer J Project IMPACT: a report on barriers and facilitators to sustainability Adm Policy Ment Health 2006 33 6 718 729 10.1007/s10488-006-0086-7 16967339 · DOI · Paper page
- Bowman C Sobo E Asch S Gifford A The H. I. V. Hepatitis Quality Enhancement Research Initiative: Measuring persistence of implementation: QUERI series Implement Sci 2008 3 1 21 10.1186/1748-5908-3-21 18430200 PMC2390585 · DOI · Paper page
- Gruen RL Elliott JH Nolan ML Lawton PD Parkhill A McLaren CJ Lavis JN Sustainability science: an integrated approach for health-programme planning Lancet 2008 372 9649 1579 1589 10.1016/S0140-6736(08)61659-1 18984192 · DOI · Paper page
- Massatti RR Sweeney HA Panzano PC Roth D The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP Adm Policy Ment Health 2008 35 1-2 50 65 10.1007/s10488-007-0141-z 17968652 · DOI
- Nilsen P Timpka T Nordenfelt L Lindqvist K Towards improved understanding of injury prevention program sustainability Saf Sci 2005 43 10 815 10.1016/j.ssci.2005.08.015 · DOI
- Plochg T Delnoij DM Hoogedoorn NP Klazinga NS Collaborating while competing? The sustainability of community-based integrated care initiatives through a health partnership BMC Health Serv Res 2006 20 6 37 PMC1464130 16549028
- Rabin B Nehl E Elliott T Deshpande A Brownson R Glanz K Individual and setting level predictors of the implementation of a skin cancer prevention program: a multilevel analysis Implement Sci 2010 5 40 53 10.1186/1748-5908-5-40 20513242 PMC2901365 · DOI
- Rohrbach LA Graham JW Hansen WB Diffusion of a school-based substance abuse prevention program: predictors of program implementation Prev Med 1993 22 2 237 260 10.1006/pmed.1993.1020 8483862 · DOI · Paper page
- Ruch-Ross H Keller D Miller N Bassewitz J Melinkovich P Evaluation of community-based health projects: the healthy tomorrows experience Pediatrics 2008 122 3 e564 e572 10.1542/peds.2006-3273 18762490 · DOI
- Scheirer MA The life cycle of an innovation: adoption versus discontinuation of the fluoride mouth rinse program in schools J Health Soc Behav 1990 31 2 203 215 10.2307/2137173 2102498 · DOI · Paper page
- Scheirer MA Hartling G Hagerman D Defining sustainability outcomes of health programs: Illustrations from an on-line survey Eval Program Plann 2008 31 4 335 346 10.1016/j.evalprogplan.2008.08.004 18835642 · DOI · Paper page
Cited by
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724
“Recent approaches to sustainability locate key efforts squarely in the implementation phase, arguing that once a practice has been implemented within a care system, those who manage the delivery of that practice should turn their attention to ensuring that the practice can be maintained over time [6,24].”
Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0898-y · Cited by 1,329
Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806
Self-citation
Metadata
- Authors
- Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M.
- Affiliations
- University of Pennsylvania (US)
- VA Boston Healthcare System (US)
- Boston University (US)
- Venue
- Implementation Science
- Published
- 2012-03-14
- Volume
- 7
- Issue
- 1
- Pages
- 17-17
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC3317864
- OpenAlex
- W2158467768
- Cited by
- 1,355
- References
- 211
- Citations by year
- 2026: 69 · 2025: 77 · 2024: 113 · 2023: 116 · 2022: 125 · 2021: 145 · 2020: 136 · 2019: 110 · 2018: 103 · 2017: 100 · 2016: 62 · 2015: 90 · 2014: 55 · 2013: 47 · 2012: 7
- FWCI
- 9.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion, cc-by)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Veterans Affairs (R25 MH080916-01A2)
- Washington University in St. Louis (R25 MH080916-01A2)
- National Institutes of Health
- Quality Enhancement Research Initiative (R25 MH080916-01A2)
- National Institute of Mental Health (MH080916, R00 MH 01800, R00MH080100, R25MH080916, T32MH019836)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- program sustainability, implementation outcomes, evidence-based programs, healthcare innovation, organizational context, organizational capacity, intervention fidelity, intervention adaptation, long-term implementation, sustainability factors, research recommendations, definition of sustainability
- MeSH terms
- Delivery of Health Care, Health Planning Guidelines, Humans, Organizational Innovation, Research Design, Program Evaluation, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.