Change in Patient Outcomes After Augmenting a Low-level Implementation Strategy in Community Practices That Are Slow to Adopt a Collaborative Chronic Care Model: A Cluster Randomized Implementation Trial
Open accessAbstract
Background: Implementation strategies are essential for promoting the uptake of evidence-based practices and for patients to receive optimal care. Yet strategies differ substantially in their intensity and feasibility. Lower-intensity strategies (eg, training and technical support) are commonly used but may be insufficient for all clinics. Limited research has examined the comparative effectiveness of augmentations to low-level implementation strategies for nonresponding clinics. Objectives: To compare 2 augmentation strategies for improving uptake of an evidence-based collaborative chronic care model (CCM) on 18-month outcomes for patients with depression at community-based clinics nonresponsive to lower-level implementation support. Research Design: Providers initially received support using a low-level implementation strategy, Replicating Effective Programs (REP). After 6 months, nonresponsive clinics were randomized to add either external facilitation (REP+EF) or external and internal facilitation (REP+EF/IF). Measures: The primary outcome was patient 12-item short form survey (SF-12) mental health score at month 18. Secondary outcomes were patient health questionnaire (PHQ-9) depression score at month 18 and receipt of the CCM during months 6 through 18. Results: Twenty-seven clinics were nonresponsive after 6 months of REP. Thirteen clinics (N=77 patients) were randomized to REP+EF and 14 (N=92) to REP+EF/IF. At 18 months, patients in the REP+EF/IF arm had worse SF-12 [diff, 8.38; 95% confidence interval (CI), 3.59–13.18] and PHQ-9 scores (diff, 1.82; 95% CI, –0.14 to 3.79), and lower odds of CCM receipt (odds ratio, 0.67; 95% CI, 0.30–1.49) than REP+EF patients. Conclusions: Patients at sites receiving the more intensive REP+EF/IF saw less improvement in mood symptoms at 18 months than those receiving REP+EF and were no more likely to receive the CCM. For community-based clinics, EF augmentation may be more feasible than EF/IF for implementing CCMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other studies have found that more is not always better.
“others have found that “more is not always better” [63–65]”
From A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022 (2024) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910 · see citing works
Self-efficacy: Toward a unifying theory of behavioral change
Bandura, A. (1977) · Psychological Review · doi:10.1037/0033-295x.84.2.191 · Cited by 43,925 · see citing works
A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.10.1092 · Cited by 31,765 · see citing works
WARE, J. E., KOSINSKI, M., & KELLER, S. D. (1996) · Medical Care · doi:10.1097/00005650-199603000-00003 · Cited by 17,392 · cited 1× · see citing works
“777 Ever homeless 101 (60) 50 (65) 51 (55) 0.”
Spitzer, R. L. (1999) · JAMA · doi:10.1001/jama.282.18.1737 · Cited by 9,802 · see citing works
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015) · Implementation Science · doi:10.1186/s13012-015-0209-1 · Cited by 4,963 · see citing works
Global mortality, disability, and the contribution of risk factors: Global Burden of Disease Study
Murray, C. J., & Lopez, A. D. (1997) · The Lancet · doi:10.1016/s0140-6736(96)07495-8 · Cited by 4,492 · see citing works
The world health organization composite international diagnostic interview short-form (CIDI-SF)
Kessler, R. C., Andrews, G., Mroczek, D., Ustun, B., & Wittchen, H. (1998) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.47 · Cited by 2,242 · see citing works
Enabling the implementation of evidence based practice: a conceptual framework.
Kitson, A., Harvey, G., & McCormack, B. (1998) · Quality in Health Care · doi:10.1136/qshc.7.3.149 · Cited by 1,985 · see citing works
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · see citing works
Kitson, A. L., Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K., & Titchen, A. (2008) · Implementation Science · doi:10.1186/1748-5908-3-1 · Cited by 1,314 · see citing works
A compilation of strategies for implementing clinical innovations in health and mental health
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E., & York, J. L. (2012) · Medical Care Research and Review · doi:10.1177/1077558711430690 · Cited by 1,069 · see citing works
Minimization: A new method of assigning patients to treatment and control groups
Taves, D. R. (1974) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt1974155443 · Cited by 891 · see citing works
Getting evidence into practice: the role and function of facilitation
Harvey, G., Loftus‐Hills, A., Rycroft‐Malone, J., Titchen, A., Kitson, A., McCormack, B., & Seers, K. (2002) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2002.02126.x · Cited by 730 · see citing works
Kilbourne, A. M., Neumann, M. S., Pincus, H. A., Bauer, M. S., & Stall, R. (2007) · Implementation Science · doi:10.1186/1748-5908-2-42 · Cited by 653 · see citing works
Self-citationThe method of minimization for allocation to clinical trials
Scott, N. W., McPherson, G. C., Ramsay, C. R., & Campbell, M. K. (2002) · Controlled Clinical Trials · doi:10.1016/s0197-2456(02)00242-8 · Cited by 651 · see citing works
Woltmann, E., Grogan-Kaylor, A., Perron, B., Georges, H., Kilbourne, A. M., & Bauer, M. S. (2012) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.11111616 · Cited by 603 · see citing works
Self-citationA "SMART" Design for Building Individualized Treatment Sequences
Lei, H., Nahum-Shani, I., Lynch, K., Oslin, D., & Murphy, S. A. (2012) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032511-143152 · Cited by 557 · see citing works
Insel, T. R. (2009) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2008.540 · Cited by 459 · see citing works
Stetler, C. B., Legro, M. W., Rycroft‐Malone, J., Bowman, C., Curran, G. M., Guihan, M., Hagedorn, H., Piñeros, S. L., & Wallace, C. M. (2006) · Implementation Science · doi:10.1186/1748-5908-1-23 · Cited by 409 · see citing works
A Guide for applying a revised version of the PARIHS framework for implementation
Stetler, C. B., Damschroder, L. J., Helfrich, C. D., & Hagedorn, H. J. (2011) · Implementation Science · doi:10.1186/1748-5908-6-99 · Cited by 402 · see citing works
Collaborative care for bipolar disorder, Part II: impact on clinical outcome, function, and costs.
Bauer, M. S., McBride, L., Williford, W. O., Glick, H., Kinosian, B., Altshuler, L., Beresford, T., Kilbourne, A. M., & Sajatovic, M. (2006) · Psychiatric Services · doi:10.1176/ps.2006.57.7.937 · Cited by 312 · see citing works
Self-citationLong-term Effectiveness and Cost of a Systematic Care Program for Bipolar Disorder
Simon, G. E., Ludman, E. J., Bauer, M. S., Unützer, J., & Operskalski, B. H. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.500 · Cited by 264 · see citing works
Self-citationOutcomes of a Partnered Facilitation Strategy to Implement Primary Care–Mental Health
Kirchner, J. E., Ritchie, M. J., Pitcock, J. A., Parker, L. E., Curran, G. M., & Fortney, J. C. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3027-2 · Cited by 256 · see citing works
Bridging Community Intervention and Mental Health Services Research
Wells, K., Miranda, J., Bruce, M. L., Alegria, M., & Wallerstein, N. (2004) · American Journal of Psychiatry · doi:10.1176/appi.ajp.161.6.955 · Cited by 237 · see citing works
Almirall, D., Compton, S. N., Gunlicks‐Stoessel, M., Duan, N., & Murphy, S. A. (2012) · Statistics in Medicine · doi:10.1002/sim.4512 · Cited by 223 · see citing works
Self-citationThree questions can detect hazardous drinkers.
Gordon, A., Maisto, S. A., McNeil, M., Kraemer, K. L., Conigliaro, R. L., Kelley, M. E., & Conigliaro, J. C. (2001) · PubMed · Cited by 217 · see citing works
Kilbourne, A. M., Post, E. P., Nossek, A., Drill, L., Cooley, S., & Bauer, M. S. (2008) · Psychiatric Services · doi:10.1176/ps.2008.59.7.760 · Cited by 142 · see citing works
Self-citationBauer, M. S., McBride, L., Williford, W. O., Glick, H., Kinosian, B., Altshuler, L., Beresford, T., Kilbourne, A. M., & Sajatovic, M. (2006) · Psychiatric Services · doi:10.1176/ps.2006.57.7.927 · Cited by 138 · see citing works
Self-citationKilbourne, A. M., Almirall, D., Eisenberg, D., Waxmonsky, J., Goodrich, D. E., Fortney, J. C., Kirchner, J. E., Solberg, L. I., Main, D., Bauer, M. S., et al. (2014) · Implementation Science · doi:10.1186/s13012-014-0132-x · Cited by 130 · see citing works
Self-citationCollaborative Chronic Care Models for Mental Health Conditions
Miller, C. J., Grogan‐Kaylor, A. C., Perron, B. E., Kilbourne, A. M., Woltmann, E. M., & Bauer, M. S. (2013) · Medical Care · doi:10.1097/mlr.0b013e3182a3e4c4 · Cited by 130 · see citing works
Self-citationBauer, M. S., Vojta, C., Kinosian, B. P., Altshuler, L. L., & Glick, H. A. (2000) · Bipolar Disorders · doi:10.1034/j.1399-5618.2000.020409.x · Cited by 123 · see citing works
Self-citationBauer, A. M., Azzone, V., Goldman, H. H., Alexander, L., Unützer, J., Coleman-Beattie, B., & Frank, R. G. (2011) · Psychiatric Services · doi:10.1176/ps.62.9.pss6209_1047 · Cited by 120 · see citing works
Translating Evidence‐Based Depression Management Services to Community‐Based Primary Care Practices
Kilbourne, A. M., Schulberg, H. C., Post, E. P., Rollman, B. L., Belnap, B. H., & Pincus, H. A. (2004) · Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00326.x · Cited by 106 · see citing works
Self-citationMoving Treatment Research From Clinical Trials to the Real World
Roy‐Byrne, P., Sherbourne, C. D., Craske, M. G., Stein, M. B., Katon, W. J., Sullivan, G., Means‐Christensen, A. J., & Bystritsky, A. (2003) · Psychiatric Services · doi:10.1176/appi.ps.54.3.327 · Cited by 105 · see citing works
Kilbourne, A. M., Abraham, K. M., Goodrich, D. E., Bowersox, N. W., Almirall, D., Lai, Z., & Nord, K. M. (2013) · Implementation Science · doi:10.1186/1748-5908-8-136 · Cited by 102 · see citing works
Self-citationReplicating effective programs: HIV/AIDS prevention technology transfer.
Neumann, M. S., & Sogolow, E. D. (2000) · PubMed · Cited by 90 · see citing works
Rossom, R. C., Solberg, L. I., Magnan, S. J., Crain, A. L., Beck, A. L., Coleman, K. J., Katzelnick, D. J., Williams, M. D., Neely, C., Ohnsorg, K. A., et al. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2016.05.006 · Cited by 85 · see citing works
Implementation of Evidence-Based Practice and Organizational Performance
Hovmand, P. S., & Gillespie, D. F. (2008) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-008-9154-y · Cited by 63 · see citing works
Life Goals Collaborative Care for Patients With Bipolar Disorder and Cardiovascular Disease Risk
Kilbourne, A. M., Goodrich, D. E., Lai, Z., Clogston, J., Waxmonsky, J. A., & Bauer, M. S. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201100528 · Cited by 61 · see citing works
Self-citationKilbourne, A. M., Almirall, D., Goodrich, D. E., Lai, Z., Abraham, K. M., Nord, K. M., & Bowersox, N. W. (2014) · Implementation Science · doi:10.1186/s13012-014-0163-3 · Cited by 50 · see citing works
Self-citationDesign and implementation of a randomized trial evaluating systematic care for bipolar disorder
Simon, G. E., Ludman, E. J., Unützer, J., & Bauer, M. S. (2002) · Bipolar Disorders · doi:10.1034/j.1399-5618.2002.01190.x · Cited by 48 · see citing works
Self-citationUsing a multi-level approach to implement a primary care mental health (PCMH) program.
Kirchner, J. E., Edlund, C. N., Henderson, K., Daily, L., Parker, L. E., & Fortney, J. C. (2010) · Families Systems & Health · doi:10.1037/a0020250 · Cited by 45 · see citing works
Kilbourne, A. M., Neumann, M. S., Waxmonsky, J. A., Bauer, M. S., Kim, H. M., Pincus, H. A., & Thomas, M. R. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201200032 · Cited by 40 · see citing works
Self-citationA manic‐depressive symptom self‐report in optical scanable format
Glick, H. A., McBride, L., & Bauer, M. S. (2003) · Bipolar Disorders · doi:10.1034/j.1399-5618.2003.00043.x · Cited by 34 · see citing works
Self-citationKilbourne, A. M., Goodrich, D. E., Nord, K. M., Poppelen, C. V., Kyle, J., Bauer, M. S., Waxmonsky, J. A., Lai, Z., Kim, H. M., Eisenberg, D., et al. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0598-5 · Cited by 33 · see citing works
Self-citationNeCamp, T., Kilbourne, A. M., & Almirall, D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217708654 · Cited by 33 · see citing works
Self-citationKilbourne, A. M., Goodrich, D. E., Lai, Z., Almirall, D., Nord, K. M., Bowersox, N. W., & Abraham, K. M. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201300497 · Cited by 32 · see citing works
Self-citationKirchner, J. E., Kearney, L. K., Ritchie, M. J., Dollar, K. M., Swensen, A. B., & Schohn, M. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201400054 · Cited by 30 · see citing works
Primary Care and Behavioral Health Practice Size
Bauer, M. S., Leader, D., Un, H., Lai, Z., & Kilbourne, A. M. (2012) · Medical Care · doi:10.1097/mlr.0b013e31825f2864 · Cited by 29 · see citing works
Self-citationKilbourne, A., Bramlet, M., Barbaresso, M. M., Nord, K. M., Goodrich, D. E., Lai, Z.-S., Post, E., Almirall, D., Verchinina, L., Duffy, S., et al. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.07.007 · Cited by 28 · see citing works
Self-citationColeman, K. J., Hemmila, T., Valenti, M. D., Smith, N., Quarrell, R., Ruona, L. K., Brandenfels, E., Hann, B., Hinnenkamp, T., Parra, M. D., et al. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2016.03.003 · Cited by 24 · see citing works
Kilbourne, A. M., Greenwald, D. E., Hermann, R. C., Charns, M. P., McCarthy, J. F., & Yano, E. M. (2010) · Psychiatric Services · doi:10.1176/ps.2010.61.1.38 · Cited by 20 · see citing works
Self-citationKilbourne, A. M., Li, D., Lai, Z., Waxmonsky, J. A., & Ketter, T. A. (2012) · Depression and Anxiety · doi:10.1002/da.22003 · Cited by 18 · see citing works
Self-citationManagement of Mental Disorders in VA Primary Care Practices
Kilbourne, A. M., Pincus, H. A., Schutte, K. K., Kirchner, J. E., Haas, G. L., & Yano, E. M. (2006) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-006-0034-6 · Cited by 18 · see citing works
Self-citationKilbourne, A. M., Prenovost, K. M., Liebrecht, C., Eisenberg, D., Kim, H. M., Un, H., & Bauer, M. S. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201800336 · Cited by 16 · see citing works
Self-citationOrganizational contexts of primary care approaches for managing problem drinking
Schutte, K. K., Yano, E. M., Kilbourne, A. M., Wickrama, B., Kirchner, J. E., & Humphreys, K. N. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2008.09.002 · Cited by 15 · see citing works
Self-citationGoodrich, D. E., Bowersox, N. W., Abraham, K. M., Burk, J. P., Visnic, S., Lai, Z., & Kilbourne, A. M. (2012) · Depression Research and Treatment · doi:10.1155/2012/325249 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [25]Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci. 2007; 2: 42. 18067681 PMC2248206
- [30]Kilbourne AM, Goodrich DE, Lai Z, Reengaging veterans with serious mental illness into care: preliminary results from a national randomized trial. Psychiatric Services. 2014; 66(1): 90– 93. 25554233 PMC4640185
Cited by
Sorted by citations
Ashcraft, L. E., Goodrich, D. E., Hero, J., Phares, A., Bachrach, R. L., Quinn, D. A., Qureshi, N., Ernecoff, N. C., Lederer, L. G., Scheunemann, L. P., et al. (2024) · Implementation Science · doi:10.1186/s13012-024-01369-5 · Cited by 89 · see citing works
Kilbourne, A. M., Geng, E., Eshun-Wilson, I., Sweeney, S., Shelley, D., Cohen, D. J., Kirchner, J. E., Fernandez, M. E., & Parchman, M. L. (2023) · Implementation Science Communications · doi:10.1186/s43058-023-00435-1 · Cited by 75 · see citing works
Self-citationMiller, C. J., Kim, B., Connolly, S. L., Spitzer, E. G., Brown, M., Bailey, H. M., Weaver, K., & Sullivan, J. L. (2023) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01231-0 · Cited by 10 · see citing works
- Authors
- Smith, S. N., Almirall, D., Prenovost, K., Liebrecht, C., Kyle, J., Eisenberg, D., Bauer, M. S., & Kilbourne, A. M.
- Affiliations
- University of Michigan (US)
- United States Department of Veterans Affairs (US)
- Corresponding author
- Smith, Shawna N.
- Prenovost, Katherine
- Liebrecht, Celeste
- Kyle, Julia
- Kilbourne, Amy M.
- ORCID
- Venue
- Medical Care
- Published
- 2019-07
- Volume
- 57
- Issue
- 7
- Pages
- 503-511
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC6684247
- OpenAlex
- W2946668669
- Cited by
- 40
- References
- 59
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 14 · 2023: 3 · 2022: 8 · 2021: 4 · 2020: 4
- FWCI
- 5.44
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R01MH099898)
- National Institute on Drug Abuse (R01DA039901)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- implementation strategies, depression, community clinics, external facilitation, evidence-based practice uptake, PHQ-9, SF-12
- MeSH terms
- Adult, Colorado, Community Mental Health Services, Depression, Female, Humans, Male, Michigan, Middle Aged, Psychiatric Status Rating Scales, Program Evaluation, Cluster Analysis, Program Development, Outcome Assessment, Health Care, Models, Organizational, Evidence-Based Medicine, Intention to Treat Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
