The DOI matches its Crossref record; 210 of the 211 references it cites were found in a registry or an index.
The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research
Open accessAbstract
Background: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research. Methods: Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized. Results: Although "sustainability" was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves. Conclusions: Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“findings indicating that incomplete penetration and sustainment are the norm (Stirman et al., 2012)”
From User‐centered design for psychosocial intervention development and implementation. (2016) · cites this paper as Stirman et al., 2012
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A number of literature reviews of implementation frameworks have also been conducted, concentrating on particular implementation concepts, such as a particular stage, or specifically on either the factors, strategies or evaluations, rather than addressing all the concepts that could affect an innovation’s implementation [13,17,19-26].”
From A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework (2015) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite the significant investment of staff time and other resources, many promising initiatives fail to sustain and do not produce long term benefits [1–6].”
From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overall, fewer than half of the programs continued at high levels of fidelity.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [135]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a systematic review of 125 empirical studies focused on the sustainability of public health and clinical interventions (135), researchers found that even when full implementation was originally achieved, partial sustainability was much more common than the continued implementation of the entire intervention.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [135]
Experimental — accuracy not yet measured
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 2×
“61 to 1 ("moderate” to “almost perfect” [45]) on more specific categories, which were later collapsed into the three broad categories.”
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · cited 2×
“Figure 1 is a modified PRISMA (which stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses [33]) diagram summarizing the selection process, which includes reasons that potentially relevant papers were excluded.”
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
Exploration and Exploitation in Organizational Learning
March, J. G. (1991) · Organization Science · doi:10.1287/orsc.2.1.71 · Cited by 21,942
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Organizational culture and leadership
(1993) · Long Range Planning · doi:10.1016/0024-6301(93)90120-5 · Cited by 16,462
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 2×
“In contrast to the relatively consistent emphasis on characteristics of the innovation within conceptualizations of sustainability and the broader implementation literature, fewer studies than expected found that characteristics of the innovation were associated with sustainability [3,9,64].”
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 13,723
Research works printed but not linked: 20
- Greenhalgh T: How to Spread Good Ideas: A systematic review of the literature on diffusion, dissemination, and sustainability of innovations in health service delivery and organization, in Report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R & D (NCCSDO). 2004, London: University College, 362-
- Maher L, Gustafson DA: Sustainability Model and Guide: Evans;. 2007
- Stirman SW: Sustainability: A systematic review of methods and conceptualizations. in Fourth Annual NIH Conference on the Science of Dissemination and Implementation. 2011, Bethesda, MD
- Scheirer MA, J Dearing: An Agenda for Research on the Sustainability of Public Health Programs. 2011, American Journal of Public Health,
- Kimberly JR, Starbuck WH: Managerial Innovation, in Handbook of Organizational Design. Edited by: Nystrom PC. 1981, New York: Oxford University Press, 84-104.
- Claquin P: Sustainability of EPI: Utopia or Sine Qua Non Condition of Child Survival, in Resources for Child Health Project. 1989, Arlington; REACH
- Lafond AK: Improving the quality of investment in health: lessons on sustainability. Health Policy Plan. 1995, 10: 63-76.
- Ovretveit J: Making temporary quality improvement continuous. 2003, Stockholm: Swedish Association of County Councils
- Davies B, Tremblay D, Edwards N: Sustaining evidence-based practice systems and measuring the impacts, in Evaluating the Impact of Implementing Evidence-Based Practice. Edited by: Bick D, Ingram ID. 2010, Oxford: Wiley Blackwell
- Blasinsky M, Goldman HH, Unutzer J: Project IMPACT: a report on barriers and facilitators to sustainability. Adm Policy Ment Health. 2006, 33 (6): 718-729. 10.1007/s10488-006-0086-7. · DOI · Paper page
- Bowman C, Sobo E, Asch S, Gifford A: The H. I. V. Hepatitis Quality Enhancement Research Initiative: Measuring persistence of implementation: QUERI series. Implement Sci. 2008, 3 (1): 21-10.1186/1748-5908-3-21. · DOI · Paper page
- Gruen RL, Elliott JH, Nolan ML, Lawton PD, Parkhill A, McLaren CJ, Lavis JN: Sustainability science: an integrated approach for health-programme planning. Lancet. 2008, 372 (9649): 1579-1589. 10.1016/S0140-6736(08)61659-1. · DOI · Paper page
- Massatti RR, Sweeney HA, Panzano PC, Roth D: The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP. Adm Policy Ment Health. 2008, 35 (1-2): 50-65. 10.1007/s10488-007-0141-z. · DOI
- Nilsen P, Timpka T, Nordenfelt L, Lindqvist K: Towards improved understanding of injury prevention program sustainability. Saf Sci. 2005, 43 (10): 815-10.1016/j.ssci.2005.08.015. · DOI
- Plochg T, Delnoij DM, Hoogedoorn NP, Klazinga NS: Collaborating while competing? The sustainability of community-based integrated care initiatives through a health partnership. BMC Health Serv Res. 2006, 20: 6-37.
- Rabin B, Nehl E, Elliott T, Deshpande A, Brownson R, Glanz K: Individual and setting level predictors of the implementation of a skin cancer prevention program: a multilevel analysis. Implement Sci. 2010, 5: 40-53. 10.1186/1748-5908-5-40. · DOI
- Rohrbach LA, Graham JW, Hansen WB: Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993, 22 (2): 237-260. 10.1006/pmed.1993.1020. · DOI · Paper page
- Ruch-Ross H, Keller D, Miller N, Bassewitz J, Melinkovich P: Evaluation of community-based health projects: the healthy tomorrows experience. Pediatrics. 2008, 122 (3): e564-e572. 10.1542/peds.2006-3273. · DOI
- Scheirer MA: The life cycle of an innovation: adoption versus discontinuation of the fluoride mouth rinse program in schools. J Health Soc Behav. 1990, 31 (2): 203-215. 10.2307/2137173. · DOI · Paper page
- Scheirer MA, Hartling G, Hagerman D: Defining sustainability outcomes of health programs: Illustrations from an on-line survey. Eval Program Plann. 2008, 31 (4): 335-346. 10.1016/j.evalprogplan.2008.08.004. · DOI · Paper page
Cited by
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724
“Recent approaches to sustainability locate key efforts squarely in the implementation phase, arguing that once a practice has been implemented within a care system, those who manage the delivery of that practice should turn their attention to ensuring that the practice can be maintained over time [6,24].”
Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0898-y · Cited by 1,329
Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806
Self-citation
Metadata
- Authors
- Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M.
- Affiliations
- University of Pennsylvania (US)
- VA Boston Healthcare System (US)
- Boston University (US)
- Venue
- Implementation Science
- Published
- 2012-03-14
- Volume
- 7
- Issue
- 1
- Pages
- 17-17
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC3317864
- OpenAlex
- W2158467768
- Cited by
- 1,355
- References
- 211
- Citations by year
- 2026: 69 · 2025: 77 · 2024: 113 · 2023: 116 · 2022: 125 · 2021: 145 · 2020: 136 · 2019: 110 · 2018: 103 · 2017: 100 · 2016: 62 · 2015: 90 · 2014: 55 · 2013: 47 · 2012: 7
- FWCI
- 9.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion, cc-by)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Veterans Affairs (R25 MH080916-01A2)
- Washington University in St. Louis (R25 MH080916-01A2)
- National Institutes of Health
- Quality Enhancement Research Initiative (R25 MH080916-01A2)
- National Institute of Mental Health (MH080916, R00 MH 01800, R00MH080100, R25MH080916, T32MH019836)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- program sustainability, implementation outcomes, evidence-based programs, healthcare innovation, organizational context, organizational capacity, intervention fidelity, intervention adaptation, long-term implementation, sustainability factors, research recommendations, definition of sustainability
- MeSH terms
- Delivery of Health Care, Health Planning Guidelines, Humans, Organizational Innovation, Research Design, Program Evaluation, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. https://app.cochat.ai/papers/10.1186/1748-5908-7-17
- The paper
- Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science. https://doi.org/10.1186/1748-5908-7-17
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.