CoChat

The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research

Open access
Wiltsey Stirman, Shannon, Kimberly, John, Cook, Natasha, Calloway, Amber, Castro, Frank, Charns, Martin · 2012 · Implementation Science

The DOI matches its Crossref record; 210 of the 211 references it cites were found in a registry or an index.

Abstract

Background: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research. Methods: Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized. Results: Although "sustainability" was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves. Conclusions: Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1748-5908-7-17

Cites

210 of 210 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Diagram of Study Selection and Exclusion Process.
Figure 1 Diagram of Study Selection and Exclusion Process.CC BY · PubMed Central
Figure 2 Sustainability Outcomes By Field. Note: More rigorous studies are defined as having included independent or objective observation and a judgment of fidelity, quality, or level of implementation. Ranges are provided when multiple studies reported these rates.
Figure 2 Sustainability Outcomes By Field. Note: More rigorous studies are defined as having included independent or objective observation and a judgment of fidelity, quality, or level of implementation. Ranges are provided when multiple studies reported these rates.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

18 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Items 201–210 of 210

Research works printed but not linked: 21

  1. Greenhalgh T How to Spread Good Ideas: A systematic review of the literature on diffusion, dissemination, and sustainability of innovations in health service delivery and organization, in Report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R & D (NCCSDO) 2004 London: University College 362
  2. Maher L Gustafson DA Sustainability Model and Guide: Evans; 2007
  3. Stirman SW Sustainability: A systematic review of methods and conceptualizations. in Fourth Annual NIH Conference on the Science of Dissemination and Implementation 2011 Bethesda, MD
  4. Scheirer MA J Dearing An Agenda for Research on the Sustainability of Public Health Programs 2011 American Journal of Public Health in press PMC3222409 21940916
  5. Kimberly JR Starbuck WH Nystrom PC Managerial Innovation, in Handbook of Organizational Design 1981 New York: Oxford University Press 84 104
  6. Claquin P Sustainability of EPI: Utopia or Sine Qua Non Condition of Child Survival, in Resources for Child Health Project 1989 Arlington; REACH
  7. Lafond AK Improving the quality of investment in health: lessons on sustainability Health Policy Plan 1995 10 63 76
  8. Ovretveit J Making temporary quality improvement continuous 2003 Stockholm: Swedish Association of County Councils
  9. SW S CF R McMackin, et al Moving From the Laboratory to the Real World: Implementation Science and Effectiveness in Practice Settings, in Toward an Integrated Approach to Trauma Focused Therapy: Placing Evidence-Based Interventions in an Expanded Psychological Context American Psychological Association: Washington, D.C in press
  10. Davies B Tremblay D Edwards N Bick D, Ingram ID Sustaining evidence-based practice systems and measuring the impacts, in Evaluating the Impact of Implementing Evidence-Based Practice 2010 Oxford: Wiley Blackwell
  11. Blasinsky M Goldman HH Unutzer J Project IMPACT: a report on barriers and facilitators to sustainability Adm Policy Ment Health 2006 33 6 718 729 10.1007/s10488-006-0086-7 16967339 · DOI · Paper page
  12. Bowman C Sobo E Asch S Gifford A The H. I. V. Hepatitis Quality Enhancement Research Initiative: Measuring persistence of implementation: QUERI series Implement Sci 2008 3 1 21 10.1186/1748-5908-3-21 18430200 PMC2390585 · DOI · Paper page
  13. Gruen RL Elliott JH Nolan ML Lawton PD Parkhill A McLaren CJ Lavis JN Sustainability science: an integrated approach for health-programme planning Lancet 2008 372 9649 1579 1589 10.1016/S0140-6736(08)61659-1 18984192 · DOI · Paper page
  14. Massatti RR Sweeney HA Panzano PC Roth D The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP Adm Policy Ment Health 2008 35 1-2 50 65 10.1007/s10488-007-0141-z 17968652 · DOI
  15. Nilsen P Timpka T Nordenfelt L Lindqvist K Towards improved understanding of injury prevention program sustainability Saf Sci 2005 43 10 815 10.1016/j.ssci.2005.08.015 · DOI
  16. Plochg T Delnoij DM Hoogedoorn NP Klazinga NS Collaborating while competing? The sustainability of community-based integrated care initiatives through a health partnership BMC Health Serv Res 2006 20 6 37 PMC1464130 16549028
  17. Rabin B Nehl E Elliott T Deshpande A Brownson R Glanz K Individual and setting level predictors of the implementation of a skin cancer prevention program: a multilevel analysis Implement Sci 2010 5 40 53 10.1186/1748-5908-5-40 20513242 PMC2901365 · DOI
  18. Rohrbach LA Graham JW Hansen WB Diffusion of a school-based substance abuse prevention program: predictors of program implementation Prev Med 1993 22 2 237 260 10.1006/pmed.1993.1020 8483862 · DOI · Paper page
  19. Ruch-Ross H Keller D Miller N Bassewitz J Melinkovich P Evaluation of community-based health projects: the healthy tomorrows experience Pediatrics 2008 122 3 e564 e572 10.1542/peds.2006-3273 18762490 · DOI
  20. Scheirer MA The life cycle of an innovation: adoption versus discontinuation of the fluoride mouth rinse program in schools J Health Soc Behav 1990 31 2 203 215 10.2307/2137173 2102498 · DOI · Paper page
  21. Scheirer MA Hartling G Hagerman D Defining sustainability outcomes of health programs: Illustrations from an on-line survey Eval Program Plann 2008 31 4 335 346 10.1016/j.evalprogplan.2008.08.004 18835642 · DOI · Paper page

Cited by

See all 1,355

Metadata

Authors
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M.
Affiliations
  • University of Pennsylvania (US)
  • VA Boston Healthcare System (US)
  • Boston University (US)
Venue
Implementation Science
Published
2012-03-14
Volume
7
Issue
1
Pages
17-17
ISSN
1748-5908 (ISSN-L)
OpenAlex
W2158467768
Cited by
1,355
References
211
Citations by year
2026: 69 · 2025: 77 · 2024: 113 · 2023: 116 · 2022: 125 · 2021: 145 · 2020: 136 · 2019: 110 · 2018: 103 · 2017: 100 · 2016: 62 · 2015: 90 · 2014: 55 · 2013: 47 · 2012: 7
FWCI
9.51
Type
journal article
Language
en
Open access
Locations
  • Implementation Science (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ScholarlyCommons (University of Pennsylvania) (submittedVersion, cc-by)
  • ScholarlyCommons (University of Pennsylvania) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Veterans Affairs (R25 MH080916-01A2)
  • Washington University in St. Louis (R25 MH080916-01A2)
  • National Institutes of Health
  • Quality Enhancement Research Initiative (R25 MH080916-01A2)
  • National Institute of Mental Health (MH080916, R00 MH 01800, R00MH080100, R25MH080916, T32MH019836)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
program sustainability, implementation outcomes, evidence-based programs, healthcare innovation, organizational context, organizational capacity, intervention fidelity, intervention adaptation, long-term implementation, sustainability factors, research recommendations, definition of sustainability
MeSH terms
Delivery of Health Care, Health Planning Guidelines, Humans, Organizational Innovation, Research Design, Program Evaluation, Evidence-Based Practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.