The clinical sustainability assessment tool: measuring organizational capacity to promote sustainability in healthcare
Open accessPublished version of a preprint
Abstract
BACKGROUND: Few validated assessment tools are available to increase understanding and measure factors associated with sustainment of clinical practices, an increasingly recognized need among clinicians. We describe the development of the Clinical Sustainability Assessment Tool (CSAT), designed to assess factors that contribute to sustainable practices in clinical settings. METHODS: Sixty-four participants from clinical and research fields participated in concept mapping and were recruited to brainstorm factors that lead to sustained clinical practices. Once repeated factors were removed, participants sorted items based on similarity and rated them by importance and feasibility. Using concept mapping analyses, items were grouped into meaningful domains to develop an initial tool. We then recruited pilot sites and early adopters, for a total of 286 practicing clinicians, to pilot and evaluate the tool. Individuals were recruited from clinical settings across pediatric and adult medical and surgical subspecialties. The data were analyzed using confirmatory factor analysis (CFA) to test hypothesized subscale structure in the instrument. We used root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) to assess fit and thus the ability of CSAT to measure the identified domains. RESULTS: The concept mapping produced sorted statements that were edited into items that could be responded to, resulting in the creation of a tool with seven determinant domains and 47 items. The pilot and CFA testing resulted in a final CSAT instrument made up 35 items, five per domain. CFA results demonstrated very good fit of the seven domain structure of the CSAT (RMSEA = 0.049; SRMR = 0.049). Usability testing indicated the CSAT is brief, easy to use, easy to learn, and does not require extensive training. Additionally, the measure scored highly (18/20) on the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The seven final CSAT domains were engaged staff and leadership, engaged stakeholders, organizational readiness, workflow integration, implementation and training, monitoring and evaluation, and outcomes and effectiveness. CONCLUSIONS: The CSAT is a new reliable assessment tool which allows for greater practical and scientific understanding of contextual factors that enable sustainable clinical practices over time.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clinical settings often have been identified as having unique processes and structures to those in public health programs.
“Additionally, clinical settings often have been identified as having unique processes and structures to those in public health programs (6).”
From Assessing the sustainability capacity of evidence-based programs in community and health settings (2022) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788 · see citing works
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 79,888 · see citing works
<b>lavaan</b> : An <i>R</i> Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955 · see citing works
Hu, L.-T., & Bentler, P. M. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.424 · Cited by 12,298 · see citing works
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595 · see citing works
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989 · see citing works
Structural Equation Modeling: Concepts, Issues, and Applications.
Smith, M. D., & Hoyle, R. H. (1996) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2988418 · Cited by 6,868 · see citing works
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,400 · see citing works
Putnick, D. L., & Bornstein, M. H. (2016) · Developmental Review · doi:10.1016/j.dr.2016.06.004 · Cited by 4,369 · see citing works
Hu, L., & Bentler, P. M. (1995) · Cited by 4,008 · see citing works
Factor analysis in the development and refinement of clinical assessment instruments.
Floyd, F. J., & Widaman, K. F. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.286 · Cited by 3,970 · see citing works
Uses and abuses of coefficient alpha
Schmitt, N. (1996) · Psychological Assessment · doi:10.1037/1040-3590.8.4.350 · Cited by 2,588 · see citing works
An introduction to implementation science for the non-specialist
Bauer, M. S., Damschroder, L. J., Hagedorn, H., Smith, J. L., & Kilbourne, A. M. (2015) · BMC Psychology · doi:10.1186/s40359-015-0089-9 · Cited by 2,023 · see citing works
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · see citing works
Effect of the number of variables on measures of fit in structural equation modeling
Kenny, D. A., & McCoach, D. B. (2003) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1003_1 · Cited by 1,231 · see citing works
Beer, M., & Nohria, N. (2000) · PubMed · Cited by 1,182 · see citing works
Widaman, K. F., & Reise, S. P. (1997) · The science of prevention: Methodological advances from alcohol and substance abuse research. · doi:10.1037/10222-009 · Cited by 1,146 · see citing works
What is "quality improvement" and how can it transform healthcare?
Batalden, P. B., & Davidoff, F. (2007) · BMJ Quality & Safety · doi:10.1136/qshc.2006.022046 · Cited by 1,104 · see citing works
Nilsen, P., & Bernhardsson, S. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4015-3 · Cited by 1,014 · see citing works
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A. P., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M. M., & Streiner, D. L. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.01.016 · Cited by 997 · see citing works
Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability
Scheirer, M. A. (2005) · American Journal of Evaluation · doi:10.1177/1098214005278752 · Cited by 710 · see citing works
A Glossary for Dissemination and Implementation Research in Health
Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008) · Journal of Public Health Management and Practice · doi:10.1097/01.phh.0000311888.06252.bb · Cited by 676 · see citing works
Helfrich, C. D., Li, Y.-F., Sharp, N. D., & Sales, A. E. (2009) · Implementation Science · doi:10.1186/1748-5908-4-38 · Cited by 564 · see citing works
Public health program capacity for sustainability: a new framework
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B., & Bunger, A. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-15 · Cited by 563 · see citing works
Self-citationAarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014) · Implementation Science · doi:10.1186/1748-5908-9-45 · Cited by 559 · see citing works
Proctor, E., Luke, D., Calhoun, A., McMillen, C., Brownson, R., McCrary, S., & Padek, M. (2015) · Implementation Science · doi:10.1186/s13012-015-0274-5 · Cited by 429 · see citing works
Self-citationLennox, L., Maher, L., & Reed, J. (2018) · Implementation Science · doi:10.1186/s13012-017-0707-4 · Cited by 421 · see citing works
The Program Sustainability Assessment Tool: A New Instrument for Public Health Programs
Luke, D. A., Calhoun, A., Robichaux, C. B., Elliott, M. B., & Moreland-Russell, S. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.130184 · Cited by 364 · see citing works
Self-citationLee, T., & Shi, D. (2021) · Psychological Methods · doi:10.1037/met0000381 · Cited by 338 · see citing works
Development and assessment of the Alberta Context Tool
Estabrooks, C. A., Squires, J. E., Cummings, G. G., Birdsell, J. M., & Norton, P. G. (2009) · BMC Health Services Research · doi:10.1186/1472-6963-9-234 · Cited by 317 · see citing works
Flaspohler, P., Duffy, J., Wandersman, A., Stillman, L., & Maras, M. A. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9162-3 · Cited by 306 · see citing works
Aarons, G. A., Sommerfeld, D. H., & Walrath-Greene, C. M. (2009) · Implementation Science · doi:10.1186/1748-5908-4-83 · Cited by 276 · see citing works
Applied Psychometrics: The Steps of Scale Development and Standardization Process
Kyriazos, T. A., & Stalikas, A. (2018) · Psychology · doi:10.4236/psych.2018.911145 · Cited by 269 · see citing works
A model for the institutionalization of health promotion programs
Goodman, R. M., & Steckler, A. (1989) · Family & Community Health · doi:10.1097/00003727-198902000-00009 · Cited by 224 · see citing works
How to Sustain Change and Support Continuous Quality Improvement
Silver, S. A., McQuillan, R., Harel, Z., Weizman, A. V., Thomas, A., NESRALLAH, G. E., Bell, C. M., Chan, C. T., & Chertow, G. M. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.11501015 · Cited by 180 · see citing works
Braithwaite, J., Ludlow, K., Testa, L., Herkes, J., Augustsson, H., Lamprell, G., McPherson, E., & Zurynski, Y. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-036453 · Cited by 147 · see citing works
Redesigning Work Processes in Health Care: Lessons from the National Health Service
Ham, C., Kipping, R., & McLeod, H. (2003) · Milbank Quarterly · doi:10.1111/1468-0009.t01-3-00062 · Cited by 118 · see citing works
Chou, W. C., Tinetti, M. E., King, M. B., Irwin, K. S., & Fortinsky, R. H. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.00298.x · Cited by 118 · see citing works
Stanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., et al. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz164 · Cited by 117 · see citing works
Rosas, S. R., & Ridings, J. W. (2016) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2016.08.016 · Cited by 105 · see citing works
Defining Organizational Capacity for Public Health Services and Systems Research
Meyer, A., Davis, M. K., & Mays, G. P. (2012) · Journal of Public Health Management and Practice · doi:10.1097/phh.0b013e31825ce928 · Cited by 99 · see citing works
Chou, W. C., Tinetti, M. E., King, M. B., Irwin, K. S., & Fortinsky, R. H. (2006) · Journal of General Internal Medicine · doi:10.1007/s11606-006-0244-3 · Cited by 97 · see citing works
Palinkas, L. A., Chou, C.-P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H. (2020) · Implementation Science · doi:10.1186/s13012-020-01030-x · Cited by 70 · see citing works
Examining sustainability in a hospital setting: Case of smoking cessation
Campbell, S., Pieters, K., Mullen, K., Reece, R., & Reid, R. D. (2011) · Implementation Science · doi:10.1186/1748-5908-6-108 · Cited by 52 · see citing works
Luke, D. A., Ribisl, K. M., Walton, M. A., & Davidson, W. S. (2002) · Substance Use & Misuse · doi:10.1081/ja-120001498 · Cited by 46 · see citing works
Self-citationOrganizational Workflow and Its Impact on Work Quality
Cain, C. H., & Haque, S. N. (2008) · Europe PMC (PubMed Central) · Cited by 45 · see citing works
Hou, S., & Roberson, K. (2014) · Journal of Cancer Education · doi:10.1007/s13187-014-0723-x · Cited by 41 · see citing works
Assessing Quality Improvement in Health Care: Theory for Practice
Kleinman, L. C., & Dougherty, D. M. (2013) · PEDIATRICS · doi:10.1542/peds.2012-1427n · Cited by 37 · see citing works
Hanbury, A., Farley, K., Thompson, C. A., Wilson, P. M., & Chambers, D. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-422 · Cited by 22 · see citing works
Chambers, A., Mustard, C., Breslin, C., Holness, L. D., & Nichol, K. A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-9 · Cited by 20 · see citing works
An Implementation Science Approach to Handoff Redesign in a Cardiac Surgery Intensive Care Unit
Geoffrion, T. R., Lynch, I. P., Hsu, W., Phelps, E., Minhajuddin, A., Tsai, E., Timmons, A., & Greilich, P. E. (2019) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2019.09.047 · Cited by 19 · see citing works
Aarons, G. A., Reeder, K., Sam-Agudu, N. A., Vorkoper, S., & Sturke, R. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00156-3 · Cited by 18 · see citing works
Demski, D., & Arabie, K. (2008) · SPE International Conference on Health, Safety and Environment in Oil and Gas Exploration and Production · doi:10.2118/111857-ms · Cited by 10 · see citing works
Aarons, G. A., Reeder, K., Sam-Agudu, N. A., et al. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00167-0 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [14]Luke D, Malone, S., Prewitt, K., Hackett, R., Lin, J., editor The clinical sustainability assessment tool (CSAT): Assessing sustainability in clinical medicine settings. 11th Annual Conference on the Science of Dissemination and Implementation in Health; December 2018; Washington, D.C.
- [15]Malone S, Prewitt, K., Luke, DA, editor Assessing clinical sustainability: a new, user-friendly tool for evaluating real-world practices. American Evaluation Association’s annual conference; 2019; Minneapolis, MN.
- [26]Malone S, Prewitt K, Lin J, Luke D, editors. Using Concept Mapping for Measure Development: Examples from Sustainability. Conference on the Science of Dissemination and Implementation in Health; 2018; Washington, DC.
Cited by
Sorted by citations
Hall, A., Shoesmith, A., Doherty, E., McEvoy, B., Mettert, K., Lewis, C. C., Wolfenden, L., Yoong, S., Kingsland, M., Shelton, R. C., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01252-1 · Cited by 81 · see citing works
Assessing the sustainability capacity of evidence-based programs in community and health settings
Bacon, C., Malone, S., Prewitt, K., Hackett, R., Hastings, M., Dexter, S., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1004167 · Cited by 22 · see citing works
“Additionally, clinical settings often have been identified as having unique processes and structures to those in public health programs (6).”
BackgroundSelf-citationSustainability of Tobacco Treatment Programs in the Cancer Center Cessation Initiative
_, _. (2021) · Journal of the National Comprehensive Cancer Network · doi:10.6004/jnccn.2021.7093 · Cited by 11 · see citing works
Malone, S., Newland, J., Kudchadkar, S. R., Prewitt, K., McKay, V., Prusaczyk, B., Proctor, E., Brownson, R. C., & Luke, D. A. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.1005802 · Cited by 6 · see citing works
Self-citation
- Authors
- Malone, S., Prewitt, K., Hackett, R., Lin, J. C., McKay, V., Walsh-Bailey, C., & Luke, D. A.
- Affiliations
- Washington University in St. Louis (US)
- Corresponding author
- Malone, Sara
- Venue
- Implementation Science Communications
- Published
- 2021-07-17
- Volume
- 2
- Issue
- 1
- Pages
- 77-77
- ISSN
- 2662-2211 (ISSN-L)
- PMC
- PMC8285819
- OpenAlex
- W3123192473
- Cited by
- 148
- References
- 55
- Citations by year
- 2026: 34 · 2025: 29 · 2024: 34 · 2023: 31 · 2022: 17 · 2021: 3
- FWCI
- 23.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science Communications (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Cancer Institute (P50CA244431)
- National Center for Advancing Translational Sciences (UL1TR002345)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Health Professions › Health Information Management › Healthcare Quality and Management
- Keywords
- sustainability in healthcare, organizational capacity, scale development, concept mapping, instrument validation, employee engagement, leadership, stakeholder engagement, organizational readiness, workflow integration, monitoring and evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
