Measuring persistence of implementation: QUERI Series
Open accessAbstract
As more quality improvement programs are implemented to achieve gains in performance, the need to evaluate their lasting effects has become increasingly evident. However, such long-term follow-up evaluations are scarce in healthcare implementation science, being largely relegated to the "need for further research" section of most project write-ups. This article explores the variety of conceptualizations of implementation sustainability, as well as behavioral and organizational factors that influence the maintenance of gains. It highlights the finer points of design considerations and draws on our own experiences with measuring sustainability, framed within the rich theoretical and empirical contributions of others. In addition, recommendations are made for designing sustainability analyses. This article is one in a Series of articles documenting implementation science frameworks and approaches developed by the U.S. Department of Veterans Affairs Quality Enhancement Research Initiative (QUERI).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Veterans Affairs Quality Enhancement Research Initiative describes a process for utilizing individual medical centers as drivers of change around specific service approaches in other VA centers.
“The Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI) describes a process for utilizing individual medical centers as drivers of change around specific service approaches in other VA centers”
From Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors (2011) · cites this paper as Bowman et al. 2008
Many promising improvement initiatives fail to sustain and do not produce long-term benefits.
“Despite the significant investment of staff time and other resources, many promising initiatives fail to sustain and do not produce long term benefits [1–6].”
From Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare (2018) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1980) · Medical Entomology and Zoology · Cited by 25,810 · see citing works
The Health Belief Model: a decade later.
Janz, N. K., & Becker, M. H. (1984) · Health Education Quarterly · doi:10.1177/109019818401100101 · Cited by 8,437 · see citing works
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · see citing works
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217 · see citing works
(2005) · Cited by 3,748 · see citing works
The Challenge of Innovation Implementation
Klein, K. J., & Sorra, J. S. (1996) · The Academy of Management Review · doi:10.2307/259164 · Cited by 2,113 · see citing works
Planning health promotion programs an intervention mapping approach
Bartholomew, L. K., Parcel, G. S., Kok, G., & Gottlieb, N. (2006) · Cited by 2,038 · see citing works
Disseminating Innovations in Health Care
Berwick, D. M. (2003) · JAMA · doi:10.1001/jama.289.15.1969 · Cited by 1,878 · see citing works
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836 · see citing works
Ferlie, E. B., & Shortell, S. M. (2001) · The Milbank Quarterly · doi:10.1111/1468-0009.00206 · Cited by 1,335 · see citing works
Process evaluation for public health interventions and research
Steckler, A., Linnan, L. A., & Israel, B. A. (2002) · Cited by 864 · see citing works
Complexity, leadership, and management in healthcare organisations
Plsek, P. E., & Wilson, T. (2001) · BMJ · doi:10.1136/bmj.323.7315.746 · Cited by 755 · see citing works
Evidence-Based Quality Improvement: The State Of The Science
Shojania, K. G., & Grimshaw, J. (2005) · Health Affairs · doi:10.1377/hlthaff.24.1.138 · Cited by 688 · see citing works
Shortell, S. M., O'Brien, J. L., Carman, J. M., FOSTER, R. W. G., Hughes, E. F., Boerstler, H., & O’Connor, E. J. (1995) · Europe PMC (PubMed Central) · Cited by 678 · see citing works
Process Evaluation for Public Health Interventions and Research
White, W. L. (2003) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2003.09.006 · Cited by 645 · see citing works
The role of formative evaluation in implementation research and the QUERI experience
Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L. (2006) · Journal of General Internal Medicine · doi:10.1007/s11606-006-0267-9 · Cited by 642 · see citing works
Self-citationProcess versus outcome indicators in the assessment of quality of health care
Mant, J. (2001) · International Journal for Quality in Health Care · doi:10.1093/intqhc/13.6.475 · Cited by 607 · see citing works
Quality collaboratives: lessons from research
Øvretveit, J. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.4.345 · Cited by 572 · see citing works
Building capacity and sustainable prevention innovations: a sustainability planning model
Johnson, K., Hays, C., Center, H., & Daley, C. (2004) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2004.01.002 · Cited by 411 · see citing works
From Research to “Best Practices” in Other Settings and Populations
Green, L. W. (2001) · American Journal of Health Behavior · doi:10.5993/ajhb.25.3.2 · Cited by 407 · see citing works
Knox, K. L., Litts, D. A., Talcott, G. W., Feig, J. C., & Caine, E. D. (2003) · BMJ · doi:10.1136/bmj.327.7428.1376 · Cited by 391 · see citing works
Harland, J., White, M. J., Drinkwater, C., Chinn, D. J., Farr, L., & Howel, D. (1999) · BMJ · doi:10.1136/bmj.319.7213.828 · Cited by 354 · see citing works
Evaluation of quality improvement programmes
Øvretveit, J. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.3.270 · Cited by 289 · see citing works
Lessons from Experienced Guideline Implementers: Attend to Many Factors and Use Multiple Strategies
Solberg, L. I., Brekke, M. L., Fazio, C. J., Fowles, J., Jacobsen, D. N., Kottke, T. E., Mosser, G., O’Connor, P. J., Ohnsorg, K. A., & Rolnick, S. J. (2000) · The Joint Commission Journal on Quality Improvement · doi:10.1016/s1070-3241(00)26013-6 · Cited by 259 · see citing works
A model for the institutionalization of health promotion programs
Goodman, R. M., & Steckler, A. (1989) · Family & Community Health · doi:10.1097/00003727-198902000-00009 · Cited by 224 · see citing works
Sanci, L. A. (2000) · BMJ · doi:10.1136/bmj.320.7229.224 · Cited by 203 · see citing works
Development of Level of Institutionalization Scales for Health Promotion Programs
Goodman, R. M., McLeroy, K. R., Steckler, A., & Hoyle, R. H. (1993) · Health Education Quarterly · doi:10.1177/109019819302000208 · Cited by 181 · see citing works
Stetler, C. B., Mittman, B. S., & Francis, J. (2008) · Implementation Science · doi:10.1186/1748-5908-3-8 · Cited by 176 · see citing works
Orleans, C. T. (2000) · Health Psychology · doi:10.1037/0278-6133.19.suppl1.76 · Cited by 174 · see citing works
Choosing Unsafe Sex: AIDS-Risk Denial Among Disadvantaged Women
Sobo, S. (1995) · Cited by 169 · see citing works
Understanding the Normalization of Telemedicine Services through Qualitative Evaluation: Table 1
May, C., Harrison, R., Finch, T., MacFarlane, A., Mair, F., & Wallace, P. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1145 · Cited by 155 · see citing works
Program sustainability: focus on organizational routines
Pluye, P. (2004) · Health Promotion International · doi:10.1093/heapro/dah411 · Cited by 144 · see citing works
Sustained Benefit of a Community and Professional Intervention to Increase Acute Stroke Therapy
Morgenstern, L. B., Bartholomew, L. K., Grotta, J. C., Savas, L. S., King, M., & Chan, W. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.18.2198 · Cited by 143 · see citing works
Commentary: applying the BMJ's guidelines on educational interventions.
Ker, J. (2000) · PubMed · Cited by 140 · see citing works
Demakis, J. G., McQueen, L., Kizer, K. W., & Feussner, J. R. (2000) · Medical Care · doi:10.1097/00005650-200006001-00003 · Cited by 131 · see citing works
Process Evaluation for Public Health Interventions and Research
Cissell, W. B. (2004) · Health Education Research · doi:10.1093/her/cyg084 · Cited by 120 · see citing works
Human Factors Barriers to the Effective Use of Ten HIV Clinical Reminders
Patterson, E. S., Nguyen, A. D., Halloran, J. P., & Asch, S. M. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1364 · Cited by 115 · see citing works
Self-citationMcQueen, L., Mittman, B. S., & Demakis, J. G. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1499 · Cited by 106 · see citing works
Theory-Based Evaluation: The Case of Social Funds
Carvalho, S., & White, H. (2004) · American Journal of Evaluation · doi:10.1177/109821400402500202 · Cited by 94 · see citing works
August, G. J., Bloomquist, M. L., Lee, S. S., Realmuto, G. M., & Hektner, J. M. (2006) · Prevention Science · doi:10.1007/s11121-005-0024-z · Cited by 83 · see citing works
Free school fruit--sustained effect 1 year later
Bere, E. T. (2005) · Health Education Research · doi:10.1093/her/cyh063 · Cited by 82 · see citing works
Sustaining the Implementation of an Evidence-Based Guideline for Bronchiolitis
Perlstein, P. H., Kotagal, U. R., Schoettker, P. J., Atherton, H. D., Farrell, M. K., Gerhardt, W. E., & Alfaro, M. P. (2000) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpedi.154.10.1001 · Cited by 80 · see citing works
Methods in Translation Science
Titler, M. G. (2004) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2004.04008.x · Cited by 71 · see citing works
Shepherd, J. (2000) · Health Education Research · doi:10.1093/her/15.6.681 · Cited by 71 · see citing works
Sustainability of Public Health Programs: The Example of Tobacco Treatment Services in Massachusetts
LaPelle, N. R., Zapka, J. G., & Ockene, J. K. (2006) · American Journal of Public Health · doi:10.2105/ajph.2005.067124 · Cited by 69 · see citing works
Methods for evaluation of small scale quality improvement projects
Harvey, G. (2003) · BMJ Quality & Safety · doi:10.1136/qhc.12.3.210 · Cited by 65 · see citing works
Clinical practice guidelines: barriers to durability after effective early implementation
Brand, C. A., Landgren, F., Hutchinson, A. F., Jones, C., MacGregor, L., & Campbell, D. A. (2005) · Internal Medicine Journal · doi:10.1111/j.1445-5994.2004.00763.x · Cited by 56 · see citing works
Reliable Effectiveness: A Theory on Sustaining and Replicating Worthwhile Innovations
Racine, D. P. (2006) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-006-0047-1 · Cited by 55 · see citing works
The Role of Services Researchers in Facilitating Intervention Research
Sullivan, G., Duan, N., Mukherjee, S., Kirchner, J. E., Perry, D. M., & Henderson, K. (2005) · Psychiatric Services · doi:10.1176/appi.ps.56.5.537 · Cited by 50 · see citing works
Stages in the Implementation of Innovative Clinical Programs in Complex Organizations
Rosenheck, R. (2001) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-200112000-00002 · Cited by 43 · see citing works
Choosing Unsafe Sex: AIDS-Risk Denial Among Disadvantaged Women.
Fursman, L., & Sobo, E. J. (1997) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2076642 · Cited by 40 · see citing works
Self-citationSanci, L. A., Coffey, C., Patton, G. C., & Bowes, G. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02172.x · Cited by 36 · see citing works
Sobo, E. J., & Sadler, B. L. (2002) · Human Organization · doi:10.17730/humo.61.3.fnk9rkekacak6mkx · Cited by 31 · see citing works
Self-citationFremont, A. M., Joyce, G., Anaya, H. D., et al. (2006) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(06)32042-9 · Cited by 18 · see citing works
Shye, D., Feldman, V., Hokanson, C. S., et al. (2004) · PubMed · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Anaya H B Results of a national comparative HIV quality-improvement initiative within the VA healthcare system. 5th International Conference on the Scientific Basis of Health Services: Global Evidence for Local Decisions 2003 Washington, D.C.
- Wheeler DJ Advanced Topics in Statistical Process Control 2004 2nd Ed. Knoxville, TN, SPC Press
Cited by
Sorted by citations
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · see citing works
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · see citing works
Public health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · see citing works
A Guide for applying a revised version of the PARIHS framework for implementation
Stetler, C. B., Damschroder, L. J., Helfrich, C. D., & Hagedorn, H. J. (2011) · Implementation Science · doi:10.1186/1748-5908-6-99 · Cited by 402 · see citing works
Lennox, L., Maher, L., & Reed, J. (2018) · Implementation Science · doi:10.1186/s13012-017-0707-4 · Cited by 421 · see citing works
The sustainability of healthcare innovations: a concept analysis
Fleiszer, A. R., Semenic, S. E., Ritchie, J. A., Richer, M., & Denis, J. (2015) · Journal of Advanced Nursing · doi:10.1111/jan.12633 · Cited by 166 · see citing works
Linking Sustainability Research to Intervention Types
Scheirer, M. A. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.300976 · Cited by 161 · see citing works
Sustainability of healthcare improvement: what can we learn from learning theory?
Hovlid, E., Bukve, O., Haug, K., Aslaksen, A. B., & von Plessen, C. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-235 · Cited by 101 · see citing works
Methods and metrics challenges of delivery-system research
Alexander, J. A., & Hearld, L. R. (2012) · Implementation Science · doi:10.1186/1748-5908-7-15 · Cited by 78 · see citing works
Communicable disease control programmes and health systems: an analytical approach to sustainability
Shigayeva, A., & Coker, R. J. (2015) · Health Policy and Planning · doi:10.1093/heapol/czu005 · Cited by 63 · see citing works
Chandler, J., Rycroft‐Malone, J., Hawkes, C., & Noyes, J. (2016) · Journal of Advanced Nursing · doi:10.1111/jan.12815 · Cited by 70 · see citing works
Measuring Sustainability Within the Veterans Administration Mental Health System Redesign Initiative
Ford, J. H., Krahn, D., Wise, M., & Oliver, K. A. (2011) · Quality Management in Health Care · doi:10.1097/qmh.0b013e3182314b20 · Cited by 60 · see citing works
Lennox, L., Doyle, C., Reed, J. E., & Bell, D. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014417 · Cited by 59 · see citing works
Hooker, L., Small, R., & Taft, A. (2016) · Journal of Advanced Nursing · doi:10.1111/jan.12851 · Cited by 47 · see citing works
The next phase in professional services research: From implementation to sustainability
Crespo-Gonzalez, C., Garcia-Cardenas, V., & Benrimoj, S. I. (2017) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2017.05.020 · Cited by 45 · see citing works
Assessing sustainability of Lifestyle Education for Activity Program (LEAP)
Saunders, R. P., Pate, R. R., Dowda, M., Ward, D. S., Epping, J. N., & Dishman, R. K. (2012) · Health Education Research · doi:10.1093/her/cyr111 · Cited by 29 · see citing works
Scheirer, M. A., Santos, S. L. Z., Tagai, E. K., Bowie, J., Slade, J., Carter, R., & Holt, C. L. (2017) · Implementation Science · doi:10.1186/s13012-017-0576-x · Cited by 29 · see citing works
Solberg, L. (2009) · Implementation Science · doi:10.1186/1748-5908-4-9 · Cited by 28 · see citing works
Francis, L., Dunt, D., & Cadilhac, D. A. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010944 · Cited by 24 · see citing works
Popowich, A. D., Mushquash, A. R., Pearson, E., Schmidt, F., & Mushquash, C. J. (2020) · Counselling and Psychotherapy Research · doi:10.1002/capr.12250 · Cited by 15 · see citing works
- Authors
- Initiative, T. H. Q. E. R., Bowman, C. C., Sobo, E. J., Asch, S. M., & Gifford, A. L.
- Affiliations
- Health Services Research & Development (US)
- San Diego State University (US)
- Center for the Study of Healthcare Provider Behavior (US)
- VA New England Healthcare System (US)
- Corresponding author
- Bowman, Candice C
- Venue
- Implementation Science
- Published
- 2008-04-22
- Volume
- 3
- Issue
- 1
- Pages
- 21-21
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC2390585
- OpenAlex
- W2113693281
- Cited by
- 100
- References
- 55
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 1 · 2023: 9 · 2022: 3 · 2021: 5 · 2020: 9 · 2019: 5 · 2018: 5 · 2017: 9 · 2016: 6 · 2015: 6 · 2014: 5 · 2013: 8 · 2012: 5
- FWCI
- 12.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Health Services Research and Development
- Quality Enhancement Research Initiative
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › Health Information Management › Healthcare Quality and Management
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- implementation sustainability, quality improvement programs, maintenance of gains, organizational factors, behavioral factors, Department of Veterans Affairs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
