CoChat

The DOI matches its Crossref record; 210 of the 211 references it cites were found in a registry or an index.

The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research

Open access
Wiltsey Stirman, Shannon, Kimberly, John, Cook, Natasha, Calloway, Amber, Castro, Frank, Charns, Martin · 2012 · Implementation Science

Abstract

Background: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research. Methods: Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized. Results: Although "sustainability" was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves. Conclusions: Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1748-5908-7-17

Cites

210 of 210 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Diagram of Study Selection and Exclusion Process.
Figure 1 Diagram of Study Selection and Exclusion Process.CC BY · PubMed Central
Figure 2 Sustainability Outcomes By Field. Note: More rigorous studies are defined as having included independent or objective observation and a judgment of fidelity, quality, or level of implementation. Ranges are provided when multiple studies reported these rates.
Figure 2 Sustainability Outcomes By Field. Note: More rigorous studies are defined as having included independent or objective observation and a judgment of fidelity, quality, or level of implementation. Ranges are provided when multiple studies reported these rates.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

18 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 211 references

Research works printed but not linked: 20

  1. Greenhalgh T: How to Spread Good Ideas: A systematic review of the literature on diffusion, dissemination, and sustainability of innovations in health service delivery and organization, in Report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R & D (NCCSDO). 2004, London: University College, 362-
  2. Maher L, Gustafson DA: Sustainability Model and Guide: Evans;. 2007
  3. Stirman SW: Sustainability: A systematic review of methods and conceptualizations. in Fourth Annual NIH Conference on the Science of Dissemination and Implementation. 2011, Bethesda, MD
  4. Scheirer MA, J Dearing: An Agenda for Research on the Sustainability of Public Health Programs. 2011, American Journal of Public Health,
  5. Kimberly JR, Starbuck WH: Managerial Innovation, in Handbook of Organizational Design. Edited by: Nystrom PC. 1981, New York: Oxford University Press, 84-104.
  6. Claquin P: Sustainability of EPI: Utopia or Sine Qua Non Condition of Child Survival, in Resources for Child Health Project. 1989, Arlington; REACH
  7. Lafond AK: Improving the quality of investment in health: lessons on sustainability. Health Policy Plan. 1995, 10: 63-76.
  8. Ovretveit J: Making temporary quality improvement continuous. 2003, Stockholm: Swedish Association of County Councils
  9. Davies B, Tremblay D, Edwards N: Sustaining evidence-based practice systems and measuring the impacts, in Evaluating the Impact of Implementing Evidence-Based Practice. Edited by: Bick D, Ingram ID. 2010, Oxford: Wiley Blackwell
  10. Blasinsky M, Goldman HH, Unutzer J: Project IMPACT: a report on barriers and facilitators to sustainability. Adm Policy Ment Health. 2006, 33 (6): 718-729. 10.1007/s10488-006-0086-7. · DOI · Paper page
  11. Bowman C, Sobo E, Asch S, Gifford A: The H. I. V. Hepatitis Quality Enhancement Research Initiative: Measuring persistence of implementation: QUERI series. Implement Sci. 2008, 3 (1): 21-10.1186/1748-5908-3-21. · DOI · Paper page
  12. Gruen RL, Elliott JH, Nolan ML, Lawton PD, Parkhill A, McLaren CJ, Lavis JN: Sustainability science: an integrated approach for health-programme planning. Lancet. 2008, 372 (9649): 1579-1589. 10.1016/S0140-6736(08)61659-1. · DOI · Paper page
  13. Massatti RR, Sweeney HA, Panzano PC, Roth D: The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP. Adm Policy Ment Health. 2008, 35 (1-2): 50-65. 10.1007/s10488-007-0141-z. · DOI
  14. Nilsen P, Timpka T, Nordenfelt L, Lindqvist K: Towards improved understanding of injury prevention program sustainability. Saf Sci. 2005, 43 (10): 815-10.1016/j.ssci.2005.08.015. · DOI
  15. Plochg T, Delnoij DM, Hoogedoorn NP, Klazinga NS: Collaborating while competing? The sustainability of community-based integrated care initiatives through a health partnership. BMC Health Serv Res. 2006, 20: 6-37.
  16. Rabin B, Nehl E, Elliott T, Deshpande A, Brownson R, Glanz K: Individual and setting level predictors of the implementation of a skin cancer prevention program: a multilevel analysis. Implement Sci. 2010, 5: 40-53. 10.1186/1748-5908-5-40. · DOI
  17. Rohrbach LA, Graham JW, Hansen WB: Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993, 22 (2): 237-260. 10.1006/pmed.1993.1020. · DOI · Paper page
  18. Ruch-Ross H, Keller D, Miller N, Bassewitz J, Melinkovich P: Evaluation of community-based health projects: the healthy tomorrows experience. Pediatrics. 2008, 122 (3): e564-e572. 10.1542/peds.2006-3273. · DOI
  19. Scheirer MA: The life cycle of an innovation: adoption versus discontinuation of the fluoride mouth rinse program in schools. J Health Soc Behav. 1990, 31 (2): 203-215. 10.2307/2137173. · DOI · Paper page
  20. Scheirer MA, Hartling G, Hagerman D: Defining sustainability outcomes of health programs: Illustrations from an on-line survey. Eval Program Plann. 2008, 31 (4): 335-346. 10.1016/j.evalprogplan.2008.08.004. · DOI · Paper page

Cited by

See all 1,355

Metadata

Authors
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M.
Affiliations
  • University of Pennsylvania (US)
  • VA Boston Healthcare System (US)
  • Boston University (US)
Venue
Implementation Science
Published
2012-03-14
Volume
7
Issue
1
Pages
17-17
ISSN
1748-5908 (ISSN-L)
OpenAlex
W2158467768
Cited by
1,355
References
211
Citations by year
2026: 69 · 2025: 77 · 2024: 113 · 2023: 116 · 2022: 125 · 2021: 145 · 2020: 136 · 2019: 110 · 2018: 103 · 2017: 100 · 2016: 62 · 2015: 90 · 2014: 55 · 2013: 47 · 2012: 7
FWCI
9.51
Type
journal article
Language
en
Open access
Locations
  • Implementation Science (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ScholarlyCommons (University of Pennsylvania) (submittedVersion, cc-by)
  • ScholarlyCommons (University of Pennsylvania) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Veterans Affairs (R25 MH080916-01A2)
  • Washington University in St. Louis (R25 MH080916-01A2)
  • National Institutes of Health
  • Quality Enhancement Research Initiative (R25 MH080916-01A2)
  • National Institute of Mental Health (MH080916, R00 MH 01800, R00MH080100, R25MH080916, T32MH019836)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
program sustainability, implementation outcomes, evidence-based programs, healthcare innovation, organizational context, organizational capacity, intervention fidelity, intervention adaptation, long-term implementation, sustainability factors, research recommendations, definition of sustainability
MeSH terms
Delivery of Health Care, Health Planning Guidelines, Humans, Organizational Innovation, Research Design, Program Evaluation, Evidence-Based Practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. https://app.cochat.ai/papers/10.1186/1748-5908-7-17
The paper
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science. https://doi.org/10.1186/1748-5908-7-17

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.