Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support
Open accessAbstract
BACKGROUND: Little is known about how well or under what conditions health innovations are sustained and their gains maintained once they are put into practice. Implementation science typically focuses on uptake by early adopters of one healthcare innovation at a time. The later-stage challenges of scaling up and sustaining evidence-supported interventions receive too little attention. This project identifies the challenges associated with sustainability research and generates recommendations for accelerating and strengthening this work. METHODS: A multi-method, multi-stage approach, was used: (1) identifying and recruiting experts in sustainability as participants, (2) conducting research on sustainability using concept mapping, (3) action planning during an intensive working conference of sustainability experts to expand the concept mapping quantitative results, and (4) consolidating results into a set of recommendations for research, methodological advances, and infrastructure building to advance understanding of sustainability. Participants comprised researchers, funders, and leaders in health, mental health, and public health with shared interest in the sustainability of evidence-based health care. RESULTS: Prompted to identify important issues for sustainability research, participants generated 91 distinct statements, for which a concept mapping process produced 11 conceptually distinct clusters. During the conference, participants built upon the concept mapping clusters to generate recommendations for sustainability research. The recommendations fell into three domains: (1) pursue high priority research questions as a unified agenda on sustainability; (2) advance methods for sustainability research; (3) advance infrastructure to support sustainability research. CONCLUSIONS: Implementation science needs to pursue later-stage translation research questions required for population impact. Priorities include conceptual consistency and operational clarity for measuring sustainability, developing evidence about the value of sustaining interventions over time, identifying correlates of sustainability along with strategies for sustaining evidence-supported interventions, advancing the theoretical base and research designs for sustainability research, and advancing the workforce capacity, research culture, and funding mechanisms for this important work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Despite an increase in the number of studies focusing on program sustainment, there remains a relative dearth of empirical evidence on the combinations of factors associated with sustainment, especially those guided by established sustainment frameworks and measurement tools.
“Despite an increase in the number of studies focusing on program sustainment, there remains a relative dearth of empirical evidence on the combinations of factors associated with sustainment [15, 16, 59], especially those guided by established sustainment frameworks and measurement tools [23].”
From Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention (2022) · cites this paper as [59]
Split into 2 atoms — 1 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredExperts have prioritized sustainability as an understudied area and identified it as one of the most significant translational research problems of our time.
“Recently, experts have prioritized sustainability as an understudied area (45), identifying it as “one of the most significant translational research problems of our time” (103, p. 2) and as a persistent challenge across a range of settings and service delivery sectors, and across health behaviors and outcomes.”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [103]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · Routledge eBooks · doi:10.4324/9780203710753-35 · Cited by 37,557 · see citing works
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 20,555 · see citing works
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
Self-citationThe quality of health care delivered to adults in the United States
McGlynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., & Kerr, E. A. (2003) · New England Journal of Medicine · doi:10.1056/nejmsa022615 · Cited by 5,079 · see citing works
Wang, P. S., Lane, M., Olfson, M., Pincus, H. A., Wells, K. B., & Kessler, R. C. (2005) · Archives of General Psychiatry · doi:10.1001/archpsyc.62.6.629 · Cited by 2,695 · see citing works
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
An introduction to concept mapping for planning and evaluation
Trochim, W. M. K. (1989) · Evaluation and Program Planning · doi:10.1016/0149-7189(89)90016-5 · Cited by 1,512 · see citing works
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · see citing works
Concept Mapping for Planning and Evaluation
Kane, M., & Trochim, W. M. K. (2006) · Sage eBooks · doi:10.4135/9781412983730 · Cited by 1,141 · see citing works
An Agenda for Research on the Sustainability of Public Health Programs
Scheirer, M. A., & Dearing, J. W. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300193 · Cited by 775 · see citing works
Damschroder, L. J., & Lowery, J. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-51 · Cited by 734 · see citing works
Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-65 · Cited by 691 · see citing works
Public health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · see citing works
Self-citationPronovost, P. J., Goeschel, C. A., Colantuoni, E., Watson, S., Lubomski, L. H., Berenholtz, S. M., Thompson, D. A., Sinopoli, D. J., Cosgrove, S. E., Sexton, J. B., et al. (2010) · BMJ · doi:10.1136/bmj.c309 · Cited by 519 · see citing works
Developing Robust, Sustainable, Implementation Systems Using Rigorous, Rapid and Relevant Science
Glasgow, R. E., & Chambers, D. (2012) · Clinical and Translational Science · doi:10.1111/j.1752-8062.2011.00383.x · Cited by 278 · see citing works
The Sustainability of Evidence-Based Practices in Routine Mental Health Agencies
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2009) · Community Mental Health Journal · doi:10.1007/s10597-009-9202-y · Cited by 193 · see citing works
Linking Sustainability Research to Intervention Types
Scheirer, M. A. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.300976 · Cited by 161 · see citing works
Massatti, R. R., Sweeney, H. A., Panzano, P. C., & Roth, D. (2007) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0141-z · Cited by 153 · see citing works
Sustainability of a practice-individualized preventive service delivery intervention
Stange, K. C., Goodwin, M. A., Zyzanski, S. J., & Dietrich, A. J. (2003) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(03)00219-8 · Cited by 131 · see citing works
Green, A. E., & Aarons, G. A. (2011) · Implementation Science · doi:10.1186/1748-5908-6-104 · Cited by 125 · see citing works
Advancing the application, quality and harmonization of implementation science measures
Rabin, B. A., Purcell, P., Naveed, S., Moser, R. P., Henton, M. D., Proctor, E. K., Brownson, R. C., & Glasgow, R. E. (2012) · Implementation Science · doi:10.1186/1748-5908-7-119 · Cited by 106 · see citing works
Self-citationBrownson, R. C., Kelly, C. M., Eyler, A. A., Carnoske, C., Grost, L., Handy, S., Maddock, J. E., Pluto, D. M., Ritacco, B. A., Sallis, J. F., et al. (2008) · Journal of Physical Activity and Health · doi:10.1123/jpah.5.4.488 · Cited by 101 · see citing works
Self-citationEasier Said than Done: Intervention Sustainability in an Urban After-School Program
Lyon, A. R., Frazier, S. L., Mehta, T. G., Atkins, M. S., & Weisbach, J. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0339-y · Cited by 87 · see citing works
Spoth, R. L., Guyll, M., Redmond, C. R., Greenberg, M. T., & Feinberg, M. E. (2011) · American Journal of Community Psychology · doi:10.1007/s10464-011-9430-5 · Cited by 78 · see citing works
Concept Mapping as a Technique for Program Theory Development
Rosas, S. R. (2005) · American Journal of Evaluation · doi:10.1177/1098214005278760 · Cited by 77 · see citing works
Systems Antecedents for Dissemination and Implementation: A Review and Analysis of Measures
Emmons, K. M., Weiner, B., Fernandez, M. E., & Tu, S.-P. (2012) · Health Education & Behavior · doi:10.1177/1090198111409748 · Cited by 72 · see citing works
Mayer, J., Mooney, B. R., Gundlapalli, A. V., Harbarth, S., Stoddard, G. J., Rubin, M. A., Eutropius, L., Brinton, B. G., & Samore, M. H. (2010) · Infection Control and Hospital Epidemiology · doi:10.1086/657666 · Cited by 71 · see citing works
Peterson, A. E., Bond, G. R., Drake, R. E., McHugo, G. J., Jones, A. M., & Williams, J. R. (2014) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-013-9347-x · Cited by 60 · see citing works
A framework for implementing sustainable oral health promotion interventions
Simpson, D. D. (2011) · Journal of Public Health Dentistry · doi:10.1111/j.1752-7325.2011.00234.x · Cited by 56 · see citing works
Druss, B. G., von Esenwein, S. A., Compton, M. T., Zhao, L., & LESLIE, D. L. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.11010071 · Cited by 56 · see citing works
Using concept mapping to develop a logic model for the Prevention Research Centers Program.
Anderson, L. A., Gwaltney, M. K., Wright, D. S., Brownson, R. C., Kane, M., CROSS, A. W., Mack, R., Schwartz, R. H., Sims, T., & White, C. (2006) · PubMed Central · Cited by 55 · see citing works
Self-citationExamining sustainability in a hospital setting: Case of smoking cessation
Campbell, S., Pieters, K., Mullen, K., Reece, R., & Reid, R. D. (2011) · Implementation Science · doi:10.1186/1748-5908-6-108 · Cited by 52 · see citing works
Seow, H. Y., Phillips, C. O., Rich, M. W., Spertus, J. A., Krumholz, H. M., & Lynn, J. (2006) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2005.00638.x · Cited by 43 · see citing works
Supported local implementation of clinical guidelines in psychiatry: a two-year follow-up
Forsner, T., Wistedt, A. Å., Brommels, M., Janszky, I., de León, A. P., & Forsell, Y. (2010) · Implementation Science · doi:10.1186/1748-5908-5-4 · Cited by 36 · see citing works
Nease, D. E., Nutting, P. A., Graham, D. G., Dickinson, W. P., Gallagher, K. M., & Jeffcott-Pera, M. (2010) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2010.05.090212 · Cited by 27 · see citing works
Defining the Components of the Research Pipeline
Kleinman, M. S., & Mold, J. W. (2009) · Clinical and Translational Science · doi:10.1111/j.1752-8062.2009.00119.x · Cited by 26 · see citing works
Culture Counts—Sustainable Inpatient Computerized Surveillance Across Duke University Health System
Cozart, H. B., Horvath, M. M., Long, A. M., et al. (2010) · Quality Management in Health Care · doi:10.1097/qmh.0b013e3181fa0680 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [29]AHRQ 2010 Annual Conference. Better care, better health: delivering on quality for all Americans. Rockville, MD: Agency for Healthcare Research and Quality; 2010. http://archive.ahrq.gov/news/events/conference/2010/.
- [37]Edwards N. Scaling-up health innovations and interventions in public health: a brief review of the current state-of-the-science. Conference to Advance the State of the Science and Practice on Scale-up and Spread of Effective Health Programs; 2010. p. 1–45
Cited by
Sorted by citations
The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care
Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014731 · Cited by 806 · see citing works
Developing a comprehensive definition of sustainability
Moore, J. E., Mascarenhas, A., Bain, J., & Straus, S. E. (2017) · Implementation Science · doi:10.1186/s13012-017-0637-1 · Cited by 630 · see citing works
Shelton, R. C., Chambers, D. A., & Glasgow, R. E. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00134 · Cited by 448 · see citing works
Means, A. R., Kemp, C. G., Gwayi-Chore, M.-C., Gimbel, S., Soi, C., Sherr, K., Wagenaar, B. H., Wasserheit, J. N., & Weiner, B. J. (2020) · Implementation Science · doi:10.1186/s13012-020-0977-0 · Cited by 418 · see citing works
Evidence-based intervention sustainability strategies: a systematic review
Hailemariam, M., Bustos, T., Montgomery, B., Barajas, R., Evans, L. B., & Drahota, A. (2019) · Implementation Science · doi:10.1186/s13012-019-0910-6 · Cited by 398 · see citing works
Iwelunmor, J., Blackstone, S., Veira, D., Nwaozuru, U., Airhihenbuwa, C., Munodawafa, D., Kalipeni, E., Jutal, A., Shelley, D., & Ogedegbe, G. (2015) · Implementation Science · doi:10.1186/s13012-016-0392-8 · Cited by 219 · see citing works
Malone, S., Prewitt, K., Hackett, R., Lin, J. C., McKay, V., Walsh-Bailey, C., & Luke, D. A. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00181-2 · Cited by 148 · see citing works
Self-citationBraithwaite, J., Ludlow, K., Testa, L., Herkes, J., Augustsson, H., Lamprell, G., McPherson, E., & Zurynski, Y. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-036453 · Cited by 147 · see citing works
Kemp, C. G., Jarrett, B. A., Kwon, C.-S., Song, L., Jetté, N., Sapag, J. C., Bass, J., Murray, L., Rao, D., & Baral, S. (2019) · BMC Medicine · doi:10.1186/s12916-018-1237-x · Cited by 138 · see citing works
Côté-Boileau, É., Denis, J.-L., Callery, B., & Sabean, M. (2019) · Health Research Policy and Systems · doi:10.1186/s12961-019-0482-6 · Cited by 133 · see citing works
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81 · see citing works
Advancing the pragmatic measurement of sustainment: a narrative review of measures
Moullin, J. C., Sklar, M., Green, A., Dickson, K. S., Stadnick, N. A., Reeder, K., & Aarons, G. A. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00068-8 · Cited by 72 · see citing works
Palinkas, L. A., Chou, C.-P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H. (2020) · Implementation Science · doi:10.1186/s13012-020-01030-x · Cited by 70 · see citing works
Johnson, A. M., Moore, J. E., Chambers, D. A., Rup, J., Dinyarian, C., & Straus, S. E. (2019) · Implementation Science · doi:10.1186/s13012-019-0895-1 · Cited by 53 · see citing works
The next phase in professional services research: From implementation to sustainability
Crespo-Gonzalez, C., Garcia-Cardenas, V., & Benrimoj, S. I. (2017) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2017.05.020 · Cited by 45 · see citing works
Nathan, N., Shelton, R. C., Laur, C. V., Hailemariam, M., & Hall, A. (2023) · Frontiers in Health Services · doi:10.3389/frhs.2023.1176023 · Cited by 36 · see citing works
Urquhart, R., Kendell, C., Cornelissen, E., Powell, B. J., Madden, L. L., Kissmann, G., Richmond, S. A., & Bender, J. L. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-042503 · Cited by 32 · see citing works
Mohr, D. C., Rosen, C. S., Schnurr, P. P., Orazem, R. J., Noorbaloochi, S., Clothier, B. A., Eftekhari, A., Bernardy, N. C., Chard, K. M., Crowley, J. J., et al. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700432 · Cited by 26 · see citing works
Scheirer, M. A., Santos, S. L. Z., Tagai, E. K., Bowie, J., Slade, J., Carter, R., & Holt, C. L. (2017) · Implementation Science · doi:10.1186/s13012-017-0576-x · Cited by 29 · see citing works
Hall, A., Shoesmith, A., Shelton, R. C., Lane, C., Wolfenden, L., & Nathan, N. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182111414 · Cited by 20 · see citing works
Assessing the sustainability capacity of evidence-based programs in community and health settings
Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1004167 · Cited by 22 · see citing works
“Many implementation science studies focus on the early stages of implementation and short-term outcomes; thus there is an important research and evaluation opportunity focusing on how effective programs are sustained over time after their initial adoption (1, 3, 4).”
BackgroundSelf-citationExamination of sustainability indicators for fall prevention strategies in three states
Smith, M. L., Durrett, N. K., Schneider, E. C., Byers, I. N., Shubert, T. E., Wilson, A. D., Towne, S. D., & Ory, M. G. (2018) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2018.02.001 · Cited by 22 · see citing works
Agulnik, A., Malone, S., Puerto-Torres, M., Gonzalez-Ruiz, A., Vedaraju, Y., Wang, H., Graetz, D., Prewitt, K., Villegas, C., Cardenas-Aguierre, A., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053116 · Cited by 20 · see citing works
Self-citationSustainability of Tobacco Treatment Programs in the Cancer Center Cessation Initiative
_, _. (2021) · Journal of the National Comprehensive Cancer Network · doi:10.6004/jnccn.2021.7093 · Cited by 11 · see citing works
Cooper, B. R., Hill, L. G., Parker, L., Jenkins, G. J., Shrestha, G., & Funaiole, A. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00303-4 · Cited by 7 · see citing works
“Despite an increase in the number of studies focusing on program sustainment, there remains a relative dearth of empirical evidence on the combinations of factors associated with sustainment [15, 16, 59], especially those guided by established sustainment frameworks and measurement tools [23].”
BackgroundMalone, S., Newland, J., Kudchadkar, S. R., Prewitt, K., McKay, V., Prusaczyk, B., Proctor, E., Brownson, R. C., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1005802 · Cited by 6 · see citing works
Self-citation
- Authors
- Proctor, E., Luke, D., Calhoun, A., McMillen, C., Brownson, R., McCrary, S., & Padek, M.
- Affiliations
- Washington University in St. Louis (US)
- Bridgespan Group (US)
- University of Chicago (US)
- Corresponding author
- Proctor, Enola
- Venue
- Implementation Science
- Published
- 2015-06-11
- Volume
- 10
- Issue
- 1
- Pages
- 88-88
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC4494699
- OpenAlex
- W1509030728
- Cited by
- 429
- References
- 37
- Citations by year
- 2026: 33 · 2025: 49 · 2024: 48 · 2023: 63 · 2022: 54 · 2021: 47 · 2020: 48 · 2019: 29 · 2018: 29 · 2017: 17 · 2016: 7 · 2015: 5
- FWCI
- 35.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion)
- Funders
- Washington University in St. Louis
- National Institutes of Health (5p30mh068579-05, CA155496, RR024992, UL1‐RR024992)
- Agency for Healthcare Research and Quality (1R13HS020775, R13HS020775)
- National Institute of Mental Health (P30-MH-068579, R25MH080916)
- National Cancer Institute (U54CA155496, THI1 U54 CA155496-01)
- National Center for Research Resources (UL1RR024992)
- Center for Mental Health Services
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- intervention sustainability, evidence-based care, scale-up, research agenda, sustainability measurement, research infrastructure, research funding, concept mapping, action planning
- MeSH terms
- Delivery of Health Care, Humans, Program Evaluation, Program Development, Evidence-Based Practice, Translational Research, Biomedical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
