Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial
Open accessAbstract
BACKGROUND: Much is to be learned about what implementation strategies are the most beneficial to communities attempting to adopt evidence-based practices. This paper presents outcomes from a randomized implementation trial of Multidimensional Treatment Foster Care (MTFC) in child public service systems in California and Ohio, including child welfare, juvenile justice, and mental health. METHODS: Fifty-one counties were assigned randomly to one of two different implementation strategies (Community Development Teams (CDT) or independent county implementation strategy (IND)) across four cohorts after being matched on county characteristics. We compared these two strategies on implementation process, quality, and milestone achievements using the Stages of Implementation Completion (SIC) (Implement Sci 6(1):1-8, 2011). RESULTS: A composite score for each county, combining the final implementation stage attained, the number of families served, and quality of implementation, was used as the primary outcome. No significant difference between CDT and IND was found for the composite measure. Additional analyses showed that there was no evidence that CDT increased the proportion of counties that started-up programs (i.e., placed at least one family in MTFC). For counties that did implement MTFC, those in the CDT condition served over twice as many youth during the study period as did IND. Of the counties that successfully achieved program start-up, those in the CDT condition completed the implementation process more thoroughly, as measured by the SIC. We found no significant differences by implementation condition on the time it took for first placement, achieving competency, or number of stages completed. CONCLUSIONS: This trial did not lead to higher rates of implementation or faster implementation but did provide evidence for more robust implementation in the CDT condition compared to IND implementation once the first family received MTFC services. This trial was successful from a design perspective in that no counties dropped out, even though this study took place during an economic recession. We believe that this methodologic approach of measurement utilizing the SIC, which is comprised of the three dimensions of quality, quantity, and timing, is appropriate for a wide range of implementation and translational studies. TRIAL REGISTRATION: Trial ID: NCT00880126 (ClinicalTrials.gov).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The original COINS study presented a head-to-head trial of two implementation approaches; although one approach was significantly more costly, the implementation outcomes achieved were superior enough to warrant the additional resources.
“For example, the original COINS study presented a head-to-head trial of two implementation approaches; although one approach was significantly more costly, the implementation outcomes achieved were superior enough to warrant the additional resources (91).”
From Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda (2019) · cites this paper as [91]
A head-to-head randomized implementation trial compares two active, qualitatively distinct strategies to see which better implements a clinical/preventive intervention.
“A head-to-head randomized implementation trial is a comparative effectiveness implementation trial that tests which of two active, qualitatively different implementation strategies is more successful in implementing a clinical/preventive intervention.”
From An Overview of Research and Evaluation Designs for Dissemination and Implementation (2017) · cites this paper as [11]
A primary outcome for most of these designs can be a composite process and output score based on speed, quality, and quantity.
“a primary outcome for most of these designs can be constructed as a composite process and output score, using dimensions of speed, quality and quantity (11, 25, 101).”
From An Overview of Research and Evaluation Designs for Dissemination and Implementation (2017) · cites this paper as [11]
Broad systems for measuring implementation process through key milestones, quality, and quantity, such as the Stages of Implementation Completion or similar unobtrusive measures, can likely be used to measure implementation fidelity and progress in diverse conditions.
“Nevertheless, it is likely that broad systems for measuring implementation process through key milestones, quality, and quantity, such as those identified in the Stages of Implementation Completion [86–88], or similar unobtrusive measures [89] can be used to measure implementation fidelity and progress in diverse conditions.”
From “Scaling-out” evidence-based interventions to new populations or new health care delivery systems (2017) · cites this paper as [88]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · Routledge eBooks · doi:10.4324/9780203710753-35 · Cited by 37,557 · see citing works
An Introduction to Categorical Data Analysis
Agresti, A. (2006) · Wiley series in probability and statistics · doi:10.1002/0470114754 · Cited by 6,699 · see citing works
Linear Mixed-Effects Models using 'Eigen' and S4
Bates, D. M., Maechler, M., Bolker, B. M., & Walker, S. C. (2015) · Cited by 5,924 · see citing works
(2005) · Cited by 3,748 · see citing works
An Introduction to Categorical Data Analysis
Musiol, G. (1997) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(97)84760-3 · Cited by 1,981 · see citing works
Continuous Improvement as an Ideal in Health Care
Berwick, D. M. (1989) · New England Journal of Medicine · doi:10.1056/nejm198901053200110 · Cited by 1,734 · see citing works
Dissemination and Implementation Research in HealthTranslating Science to Practice
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2012) · doi:10.1093/acprof:oso/9780199751877.001.0001 · Cited by 1,355 · see citing works
Individual determinants of research utilization by nurses: a systematic review update
Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-1 · Cited by 798 · see citing works
Issues in Disseminating and Replicating Effective Prevention Programs
Elliott, D. S., & Mihalic, S. (2004) · Prevention Science · doi:10.1023/b:prev.0000013981.28071.52 · Cited by 794 · see citing works
Some Conceptual and Statistical Issues in Analysis of Longitudinal Psychiatric Data
Gibbons, R. D. (1993) · Archives of General Psychiatry · doi:10.1001/archpsyc.1993.01820210073009 · Cited by 719 · see citing works
Pescher, C., & Tellis, G. J. (2019) · Diffusion fundamentals. · doi:10.62721/diffusion-fundamentals.32.1058 · Cited by 689 · see citing works
Design and analysis of group-randomized trials: a review of recent methodological developments
Murray, D. M., Varnell, S. P., & Blitstein, J. L. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.423 · Cited by 677 · see citing works
The CASE for USING INDUSTRIAL QUALITY MANAGEMENT SCIENCE in HEALTH CARE ORGANIZATIONS
Laffel, G. L., & Blumenthal, D. (1993) · CRC Press eBooks · doi:10.4324/9781498702881-02 · Cited by 401 · see citing works
Understanding the Components of Quality Improvement Collaboratives: A Systematic Literature Review
NADEEM, E., OLIN, S. S., HILL, L. C., HOAGWOOD, K. E., & HORWITZ, S. M. (2013) · The Milbank Quarterly · doi:10.1111/milq.12016 · Cited by 332 · see citing works
Glasgow, R. E., Magid, D. J., Beck, A., Ritzwoller, D., & Estabrooks, P. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000163645.41407.09 · Cited by 308 · see citing works
The Case for Using Industrial Quality Management Science in Health Care Organizations
Laffel, G. L. (1989) · JAMA · doi:10.1001/jama.1989.03430200113036 · Cited by 306 · see citing works
Society for Prevention Research Type 2 Translational Task Force Members and Contributing Authors, Spoth, R., Rohrbach, L. A., Greenberg, M., Leaf, P., Brown, C. H., Fagan, A., Catalano, R. F., Pentz, M. A., Sloboda, Z., et al. (2013) · Prevention Science · doi:10.1007/s11121-012-0362-6 · Cited by 283 · see citing works
Self-citationAn Analysis of Transformations Revisited, Rebutted
Box, G. E. P., & Cox, D. R. (1982) · Journal of the American Statistical Association · doi:10.2307/2287791 · Cited by 281 · see citing works
An Analysis of Transformations Revisited, Rebutted
Box, G. E. P., & Cox, D. R. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477788 · Cited by 278 · see citing works
Chamberlain, P., Brown, C. H., & Saldana, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-116 · Cited by 262 · see citing works
Self-citationChamberlain, P. (2003) · American Psychological Association eBooks · doi:10.1037/10596-000 · Cited by 250 · see citing works
Self-citationLeve, L. D., Harold, G. T., Chamberlain, P., Landsverk, J. A., Fisher, P. A., & Vostanis, P. (2012) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2012.02594.x · Cited by 240 · see citing works
Self-citationCollins, C. B., Harshbarger, C. L., Sawyer, R., & Hamdallah, M. (2006) · AIDS Education and Prevention · doi:10.1521/aeap.2006.18.supp.5 · Cited by 194 · see citing works
Saldana, L. (2014) · Implementation Science · doi:10.1186/1748-5908-9-43 · Cited by 188 · see citing works
Self-citationAdaptive Designs for Randomized Trials in Public Health
Brown, C. H., Ten Have, T. R., Jo, B., Dagne, G., Wyman, P. A., Muthén, B., & Gibbons, R. D. (2009) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.031308.100223 · Cited by 184 · see citing works
Self-citationEvaluating Translational Research: A Process Marker Model
Trochim, W. M. K., Kane, C., Graham, M. J., & Pincus, H. A. (2011) · Clinical and Translational Science · doi:10.1111/j.1752-8062.2011.00291.x · Cited by 177 · see citing works
Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., Poduska, J., Ialongo, N., Wyman, P. A., Chamberlain, P., et al. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2007.11.013 · Cited by 177 · see citing works
Self-citationSaldana, L., Chamberlain, P., Bradford, W. D., Campbell, M., & Landsverk, J. (2014) · Children and Youth Services Review · doi:10.1016/j.childyouth.2013.10.006 · Cited by 175 · see citing works
Self-citationChinman, M., Hannah, G., Wandersman, A. H., Ebener, P. A., Hunter, S. B., Imm, P., & Sheldon, J. (2005) · American Journal of Community Psychology · doi:10.1007/s10464-005-3390-6 · Cited by 167 · see citing works
PromisingInterventions, O’Connell, M. E., Boat, T. F., & Warner, K. E. (2009) · Cited by 159 · see citing works
Chinman, M., Hunter, S. B., Ebener, P., Paddock, S. M., Stillman, L., Imm, P., & Wandersman, A. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9163-2 · Cited by 154 · see citing works
Predicting program start-up using the stages of implementation measure
Saldana, L., Chamberlain, P., Wang, W., & Hendricks Brown, C. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0363-y · Cited by 148 · see citing works
Self-citationThe case for using industrial quality management science in health care organizations.
Laffel, G. L., & Blumenthal, D. (1989) · PubMed · Cited by 141 · see citing works
Chinman, M., Imm, P., & Wandersman, A. (2004) · Cited by 132 · see citing works
Design and Analysis in Dissemination and Implementation Research
Landsverk, J., Brown, C. H., Chamberlain, P., Palinkas, L., Ogihara, M., Czaja, S., Goldhaber-Fiebert, J. D., Rolls Reutz, J. A., & Horwitz, S. M. (2012) · Dissemination and Implementation Research in HealthTranslating Science to Practice · doi:10.1093/acprof:oso/9780199751877.003.0012 · Cited by 114 · see citing works
Self-citationChamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., Marsenich, L., Sosna, T., Padgett, C., & Bouwman, G. (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0167-x · Cited by 100 · see citing works
Self-citationPractical behavioral trials to advance evidence-based behavioral medicine
Glasgow, R. E., Davidson, K. W., Dobkin, P. L., Ockene, J. K., & Spring, B. J. (2006) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3101_3 · Cited by 76 · see citing works
Three Collaborative Models for Scaling Up Evidence-Based Practices
Chamberlain, P., Roberts, R., Jones, H., Marsenich, L., Sosna, T., & Price, J. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0349-9 · Cited by 66 · see citing works
Self-citationSupporting Implementation: The Role of Community Development Teams to Build Infrastructure
Saldana, L., & Chamberlain, P. (2012) · American Journal of Community Psychology · doi:10.1007/s10464-012-9503-0 · Cited by 65 · see citing works
Self-citationBerwick, D. M. (1996) · Annals of Internal Medicine · doi:10.7326/0003-4819-125-10-199611150-00009 · Cited by 56 · see citing works
Wandersman, A. (2013) · American Journal of Evaluation · doi:10.1177/1098214013500705 · Cited by 37 · see citing works
Chinman, M., Tremain, B., Imm, P., & Wandersman, A. (2009) · American Journal of Orthopsychiatry · doi:10.1037/a0016705 · Cited by 29 · see citing works
Getting To Outcomes: a community-based participatory approach to preventive interventions
Chinman, M., Early, D., Ebener, P., Hunter, S., Imm, P., Jenkins, P., Sheldon, J., & Wandersman, A. (2004) · Journal of Interprofessional Care · doi:10.1080/13561820400011727 · Cited by 25 · cited 1× · see citing works
“We note that a broad-based prevention support system, Getting to Outcomes (GTO), has used a non-randomized design to evaluate their implementation strategy [37] in six organizations that received GTO support contrasting these to four that did not [37-42].”
MethodLarsen, K. R. (2005) · Biometrics · doi:10.1111/j.1541-0420.2005.00374.x · Cited by 20 · see citing works
Palinkas, L. A., Holloway, I. W., Rice, E., Brown, C. H., Valente, T. W., & Chamberlain, P. (2013) · Implementation Science · doi:10.1186/1748-5908-8-133 · Cited by 20 · see citing works
Self-citationPals, S. L., Wiegand, R. E., & Murray, D. M. (2011) · AIDS · doi:10.1097/qad.0b013e3283467198 · Cited by 17 · see citing works
(2009)
Flagged first, then by citations
Cited by
Sorted by citations
Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda
Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., & Weiner, B. J. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00003 · Cited by 775 · see citing works
An Overview of Research and Evaluation Designs for Dissemination and Implementation
Brown, C. H., Curran, G., Palinkas, L. A., Aarons, G. A., Wells, K. B., Jones, L., Collins, L. M., Duan, N., Mittman, B. S., Wallace, A., et al. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044215 · Cited by 509 · see citing works
Self-citation“Scaling-out” evidence-based interventions to new populations or new health care delivery systems
Aarons, G. A., Sklar, M., Mustanski, B., Benbow, N., & Brown, C. H. (2017) · Implementation Science · doi:10.1186/s13012-017-0640-6 · Cited by 399 · see citing works
Self-citationTracking implementation strategies: a description of a practical approach and early findings
Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H., Birken, S. A., & Shea, C. (2017) · Health Research Policy and Systems · doi:10.1186/s12961-017-0175-y · Cited by 208 · see citing works
Aarons, G. A., Ehrhart, M. G., Moullin, J. C., Torres, E. M., & Green, A. E. (2017) · Implementation Science · doi:10.1186/s13012-017-0562-3 · Cited by 159 · see citing works
Stirman, S. W., Gutner, C. A., Langdon, K., & Graham, J. R. (2016) · Behavior Therapy · doi:10.1016/j.beth.2015.12.001 · Cited by 103 · see citing works
Alley, Z. M., Chapman, J. E., Schaper, H., & Saldana, L. (2023) · Implementation Science · doi:10.1186/s13012-023-01285-0 · Cited by 86 · see citing works
Self-citationWyman, P. A., Henry, D., Knoblauch, S., & Brown, C. H. (2015) · Prevention Science · doi:10.1007/s11121-014-0535-6 · Cited by 42 · see citing works
Self-citationAutomatic classification of communication logs into implementation stages via text analysis
Wang, D., Ogihara, M., Gallo, C., Villamar, J. A., Smith, J. D., Vermeer, W., Cruden, G., Benbow, N., & Brown, C. H. (2015) · Implementation Science · doi:10.1186/s13012-016-0483-6 · Cited by 17 · see citing works
Self-citationHuang, S., MacKinnon, D. P., Perrino, T., Gallo, C., Cruden, G., & Brown, C. H. (2016) · Statistical Methods & Applications · doi:10.1007/s10260-016-0354-y · Cited by 12 · see citing works
Self-citation
- Authors
- Brown, C. H., Chamberlain, P., Saldana, L., Padgett, C., Wang, W., & Cruden, G.
- Affiliations
- Northwestern University (US)
- Oregon Social Learning Center (US)
- University of South Florida (US)
- Corresponding author
- Brown, C Hendricks
- Venue
- Implementation Science
- Published
- 2014-10-14
- Volume
- 9
- Issue
- 1
- Pages
- 134-134
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC4201704
- OpenAlex
- W2122357259
- Cited by
- 89
- References
- 48
- Citations by year
- 2026: 2 · 2025: 11 · 2024: 5 · 2023: 4 · 2022: 14 · 2021: 8 · 2020: 11 · 2019: 4 · 2018: 6 · 2017: 11 · 2016: 5 · 2015: 6 · 2014: 2
- FWCI
- 14.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Health and Human Services
- William T. Grant Foundation
- National Institute of Mental Health (R01MH076158, DA023920, R01 MH0977748, R01MH097748)
- National Institute on Drug Abuse (P30DA027828, P30DA023920, DA027828, DA023920, K23DA021603, P50DA035763, DA035763)
- U.S. Children's Bureau
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Safety Research › Child Welfare and Adoption
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- Multidimensional Treatment Foster Care, implementation strategies, child welfare, juvenile justice, child mental health, evidence-based practice adoption, implementation quality
- MeSH terms
- California, Child, Child Welfare, Clinical Competence, Community Health Services, Delivery of Health Care, Foster Home Care, Health Plan Implementation, Humans, Ohio, Patient Care Team, Cluster Analysis, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
