Redesigning implementation measurement for monitoring and quality improvement in community delivery settings
Open accessAbstract
The field of prevention has established the potential to promote child adjustment across a wide array of outcomes. However, when evidence-based prevention programs have been delivered at scale in community settings, declines in implementation and outcomes have resulted. Maintaining high quality implementation is a critical challenge for the field. We describe steps towards the development of a practical system to monitor and support the high-quality implementation of evidence-based prevention programs in community settings. Research on the implementation of an evidence-based parenting program for divorcing families called the "New Beginnings Program" serves as an illustration of the promise of such a system. As a first step, we describe a multidimensional theoretical model of implementation that links aspects of program delivery with improvements in participant outcomes. We then describe research on the measurement of each of these implementation dimensions and test their relations to intended program outcomes. As a third step, we develop approaches to the assessment of these implementation constructs that are feasible to use in community settings and to establish their reliability and validity. We focus on the application of machine learning algorithms and web-based data collection systems to assess implementation and provide support for high quality delivery and positive outcomes. Examples are presented to demonstrate that valid and reliable measures can be collected using these methods. Finally, we envision how these measures can be used to develop an unobtrusive system to monitor implementation and provide feedback and support in real time to maintain high quality implementation and program outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such methods can reduce measurement burden, recall and reporting biases, and missing data.
“in addition to increasing scalability, they can reduce measurement burden, recall and reporting biases, and missing data (Berkel et al., 2019).”
From Measuring the active elements of cognitive-behavioral therapies (2023) · cites this paper as Berkel et al., 2019
In the Introduction’s “Towards Feasible Assessment of Participant Responsiveness” section (page unavailable), the paper says online data entry could reduce facilitator burden and that technology could help achieve complete data collection. It does not say these methods reduce recall or reporting biases; its discussion of transfer errors is a different point. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Durlak, J. A., & DuPre, E. P. (2008) · American Journal of Community Psychology · doi:10.1007/s10464-008-9165-0 · Cited by 5,304 · see citing works
MacArthur, C. A., Graham, S. W., & Fitzgerald, J. (2005) · Cited by 2,327 · see citing works
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · see citing works
Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices
Prasad, V., & Ioannidis, J. P. (2014) · Implementation Science · doi:10.1186/1748-5908-9-1 · Cited by 723 · see citing works
Putting the Pieces Together: An Integrated Model of Program Implementation
Berkel, C., Mauricio, A. M., Schoenfelder, E. N., & Sandler, I. N. (2010) · Prevention Science · doi:10.1007/s11121-010-0186-1 · Cited by 532 · see citing works
Self-citationSandler, I. N., Schoenfelder, E. N., Wolchik, S. A., & MacKinnon, D. P. (2011) · Annual Review of Psychology · doi:10.1146/annurev.psych.121208.131619 · Cited by 455 · see citing works
Self-citationToward the Effective and Efficient Measurement of Implementation Fidelity
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam‐Gerow, M. A. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0321-0 · Cited by 422 · see citing works
Implementation of the CALM intervention for anxiety disorders: a qualitative study
Curran, G. M., Sullivan, G., Mendel, P. J., Craske, M. G., Sherbourne, C. D., Stein, M. B., McDaniel, A., & Roy‐Byrne, P. (2012) · Implementation Science · doi:10.1186/1748-5908-7-14 · Cited by 349 · see citing works
Aarons, G. A., & Sawitzky, A. C. (2006) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-006-0039-1 · Cited by 346 · see citing works
Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005) · Behavior Therapy · doi:10.1016/s0005-7894(05)80049-8 · Cited by 320 · see citing works
Glasgow, R. E., & Riley, W. T. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.010 · Cited by 318 · see citing works
Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005) · Health Education Research · doi:10.1093/her/cyg134 · Cited by 278 · see citing works
An experimental evaluation of theory-based mother and mother–child programs for children of divorce
Wolchik, S. A., West, S. G., Sandler, I. N., Tein, J.-Y., Coatsworth, D., Lengua, L., Weiss, L., Anderson, E. R., Greene, S. M., & Griffin, W. A. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.68.5.843 · Cited by 263 · see citing works
Self-citationGroup environment scale manual
Moos, R. H. (1986) · Consulting Psychologists Press eBooks · Cited by 212 · see citing works
A "Fundamental Theorem" of Biomedical Informatics
Friedman, C. P. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3092 · Cited by 201 · see citing works
Martino, S., Ball, S. A., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009) · Psychotherapy Research · doi:10.1080/10503300802688460 · Cited by 164 · see citing works
The association of therapist empathy and synchrony in vocally encoded arousal.
Imel, Z. E., Barco, J. S., Brown, H. J., Baucom, B. R., Baer, J. S., Kircher, J. C., & Atkins, D. C. (2014) · Journal of Counseling Psychology · doi:10.1037/a0034943 · Cited by 159 · see citing works
Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., Glisson, C., Rolls-Reutz, J., & The Research Network on Youth Mental Health (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-007-0147-6 · Cited by 149 · see citing works
Treatment adherence process research in family therapy: A rationale and some practical guidelines.
Hogue, A., Liddle, H. A., & Rowe, C. L. (1996) · Psychotherapy · doi:10.1037/0033-3204.33.2.332 · Cited by 141 · see citing works
von Thiele Schwarz, U., Aarons, G. A., & Hasson, H. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4668-y · Cited by 134 · see citing works
Atkins, D. C., Mark Steyvers, B., Imel, Z. E., & Smyth, P. (2014) · Implementation Science · doi:10.1186/1748-5908-9-49 · Cited by 130 · see citing works
Handbook of parent training : helping parents prevent and solve problem behaviors
Briesmeister, J. M., & Schaefer, C. E. (2007) · Cited by 121 · see citing works
Xiao, B., Imel, Z. E., Georgiou, P., Atkins, D. C., & Narayanan, S. S. (2015) · PLoS ONE · doi:10.1371/journal.pone.0143055 · Cited by 111 · see citing works
Advancing the application, quality and harmonization of implementation science measures
Rabin, B. A., Purcell, P., Naveed, S., Moser, R. P., Henton, M. D., Proctor, E. K., Brownson, R. C., & Glasgow, R. E. (2012) · Implementation Science · doi:10.1186/1748-5908-7-119 · Cited by 106 · see citing works
Wolchik, S. A., Sandler, I. N., Tein, J.-Y., Mahrer, N. E., Millsap, R. E., Winslow, E., Vélez, C., Porter, M. M., Luecken, L. J., & Reed, A. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0033235 · Cited by 91 · see citing works
Self-citationIt’s a bird, it’s a plane, it’s … fidelity measurement in the real world.
Schoenwald, S. K. (2011) · Clinical Psychology Science and Practice · doi:10.1111/j.1468-2850.2011.01245.x · Cited by 72 · see citing works
Can, D., Marín, R. A., Georgiou, P., Imel, Z. E., Atkins, D. C., & Narayanan, S. S. (2016) · Journal of Counseling Psychology · doi:10.1037/cou0000111 · Cited by 61 · see citing works
Brown, C. H., Mohr, D. C., Gallo, C. G., Mader, C., Palinkas, L., Wingood, G., Prado, G., Kellam, S. G., Pantin, H., Poduska, J., et al. (2013) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/qai.0b013e31829372bd · Cited by 55 · see citing works
Self-citationPerrino, T., Beardslee, W., Bernal, G., Brincks, A., Cruden, G., Howe, G., Murry, V., Pantin, H., Prado, G., Sandler, I., et al. (2015) · Prevention Science · doi:10.1007/s11121-014-0518-7 · Cited by 51 · see citing works
Self-citationHerman, P. M., Mahrer, N. E., Wolchik, S. A., Porter, M. M., Jones, S., & Sandler, I. N. (2015) · Prevention Science · doi:10.1007/s11121-014-0527-6 · Cited by 50 · see citing works
Self-citationBerkel, C., Sandler, I. N., Wolchik, S. A., Brown, C. H., Gallo, C. G., Chiapa, A., Mauricio, A. M., & Jones, S. L. (2016) · Prevention Science · doi:10.1007/s11121-016-0738-0 · Cited by 37 · see citing works
Self-citationEffects of a Program to Promote High Quality Parenting by Divorced and Separated Fathers
Sandler, I. N., Gunn, H., Mazza, G. L., Tein, J., Wolchik, S. A., Berkel, C., Jones, S. L., & Porter, M. M. (2017) · Prevention Science · doi:10.1007/s11121-017-0841-x · Cited by 37 · see citing works
Self-citationBerkel, C., Mauricio, A. M., Sandler, I. N., Wolchik, S. A., Gallo, C. G., & Brown, C. H. (2017) · Prevention Science · doi:10.1007/s11121-017-0855-4 · Cited by 35 · see citing works
Self-citationMurry, V. M., Berkel, C., & Liu, N. (2018) · Prevention Science · doi:10.1007/s11121-018-0863-z · Cited by 30 · see citing works
Self-citationSchoenfelder, E. N., Sandler, I. N., Millsap, R. E., Wolchik, S. A., Berkel, C., & Ayers, T. S. (2013) · Prevention Science · doi:10.1007/s11121-012-0337-7 · Cited by 25 · see citing works
Self-citationGallo, C. G., Pantin, H. M., Villamar, J. A., Prado, G. J., Tapia, M. I., Ogihara, M., Cruden, G., & Brown, C. H. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0538-4 · Cited by 24 · see citing works
Self-citationChacko, A., Isham, A., Cleek, A. F., & McKay, M. M. (2016) · Pilot and Feasibility Studies · doi:10.1186/s40814-016-0097-4 · Cited by 19 · see citing works
Effectiveness Trial of the New Beginnings Program for Divorcing Parents
Sandler, I. N., Wolchik, S. A., Berkel, C., Jones, S. L., Mauricio, A. M., Tein, J., & Winslow, E. B. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316104453.006 · Cited by 13 · see citing works
Self-citation(2009)
Flagged first, then by citations
Research works printed but not linked: 2
- Gallo CG, Berkel C, Sandler IN, & Brown CH (2015). Improving implementation of behavioral interventions by monitoring quality of delivery in speech Paper presented at the annual Conference on the Science of Dissemination & Implementation, Washington, DC.
- Gallo CG, Li Y, Berkel C, Mehrotra S, Liu L, Benbow N, & Brown CH (under review). Recognizing emotion in speech for assessing the implementing behavioral interventions.
Cited by
Sorted by citations
Bickman, L. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01065-8 · Cited by 184 · see citing works
von Thiele Schwarz, U., Aarons, G. A., & Hasson, H. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4668-y · Cited by 134 · see citing works
Palinkas, L. A., Chou, C.-P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H. (2020) · Implementation Science · doi:10.1186/s13012-020-01030-x · Cited by 70 · see citing works
Self-citationMeasuring the active elements of cognitive-behavioral therapies
Cohen, Z. D., Barnes-Horowitz, N. M., Forbes, C. N., & Craske, M. G. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104364 · Cited by 58 · see citing works
- Authors
- Berkel, C., Gallo, C. G., Sandler, I. N., Mauricio, A. M., Smith, J. D., & Brown, C. H.
- Affiliations
- Arizona State University (US)
- Northwestern University (US)
- Corresponding author
- Berkel, Cady
- ORCID
- Venue
- The Journal of Primary Prevention
- Published
- 2019-01-18
- Volume
- 40
- Issue
- 1
- Pages
- 111-127
- ISSN
- 0278-095X (ISSN-L)
- 1573-6547
- PMC
- PMC6404547
- OpenAlex
- W2910727329
- Cited by
- 27
- References
- 40
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 9 · 2022: 2 · 2021: 6 · 2020: 5 · 2019: 2
- FWCI
- 5.11
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- The Journal of Primary Prevention (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute on Drug Abuse (R01DA033991, R01DA026874, P30-DA027828, R01DA033991-03S1)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Keywords
- implementation measurement, implementation monitoring, quality improvement, evidence-based prevention programs, parent training, divorce, child adjustment, implementation fidelity, machine learning, online data collection, real-time feedback, reliability and validity
- MeSH terms
- Machine Learning, Implementation Science, Adolescent, Adult, Child, Curriculum, Data Collection, Divorce, Female, Humans, Male, Models, Theoretical, Parent-Child Relations, Program Evaluation, Evidence-Based Practice, Quality Improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
