Automatic classification of communication logs into implementation stages via text analysis
Open accessAbstract
BACKGROUND: To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes. RESULTS: We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage. CONCLUSIONS: This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Broad systems for measuring implementation process through key milestones, quality, and quantity, such as the Stages of Implementation Completion or similar unobtrusive measures, can likely be used to measure implementation fidelity and progress in diverse conditions.
“Nevertheless, it is likely that broad systems for measuring implementation process through key milestones, quality, and quantity, such as those identified in the Stages of Implementation Completion [86–88], or similar unobtrusive measures [89] can be used to measure implementation fidelity and progress in diverse conditions.”
From “Scaling-out” evidence-based interventions to new populations or new health care delivery systems (2017) · cites this paper as [89]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · see citing works
(2005) · Cited by 3,748 · see citing works
Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0327-7 · Cited by 3,240 · cited 2× · see citing works
“The detection of the quality, speed, and milestone attainment of the implementation process of effective programs is of paramount concern for implementation scientists, policy makers, and practitioners [1, 3].”
BackgroundMethodFeature Selection for Knowledge Discovery and Data Mining
Liu, H., & Motoda, H. (1998) · Springer eBooks · doi:10.1007/978-1-4615-5689-3 · Cited by 2,216 · see citing works
Page, E. B., Webb, E. J., Campell, D. T., Schwart, R. D., & SECHREST, L. B. (1966) · American Educational Research Journal · doi:10.2307/1162043 · Cited by 1,970 · see citing works
Orthogonal nonnegative matrix t-factorizations for clustering
Ding, C. H. Q., Li, T., Peng, W., & Park, H. (2006) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1150402.1150420 · Cited by 1,273 · see citing works
Individual determinants of research utilization by nurses: a systematic review update
Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-1 · Cited by 798 · see citing works
Development, Evaluation, and Multinational Dissemination of the Triple P-Positive Parenting Program
Sanders, M. R. (2012) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032511-143104 · Cited by 619 · see citing works
Toward the Effective and Efficient Measurement of Implementation Fidelity
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam‐Gerow, M. A. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0321-0 · Cited by 422 · see citing works
Pai, M., Lloyd, N. S., Cheng, J., Thabane, L., Spencer, F. A., Cook, D. J., Haynes, R. B., Schünemann, H. J., & Douketis, J. D. (2013) · Implementation Science · doi:10.1186/1748-5908-8-1 · Cited by 376 · see citing works
Society for Prevention Research Type 2 Translational Task Force Members and Contributing Authors, Spoth, R., Rohrbach, L. A., Greenberg, M., Leaf, P., Brown, C. H., Fagan, A., Catalano, R. F., Pentz, M. A., Sloboda, Z., et al. (2013) · Prevention Science · doi:10.1007/s11121-012-0362-6 · Cited by 283 · see citing works
Self-citationChamberlain, P., Brown, C. H., & Saldana, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-116 · Cited by 262 · see citing works
Self-citationSaldana, L. (2014) · Implementation Science · doi:10.1186/1748-5908-9-43 · Cited by 188 · see citing works
Adaptive Designs for Randomized Trials in Public Health
Brown, C. H., Ten Have, T. R., Jo, B., Dagne, G., Wyman, P. A., Muthén, B., & Gibbons, R. D. (2009) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.031308.100223 · Cited by 184 · see citing works
Self-citationKorfmacher, J., O'Brien, R., Hiatt, S. W., & Olds, D. L. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.12.1847 · Cited by 156 · see citing works
Predicting program start-up using the stages of implementation measure
Saldana, L., Chamberlain, P., Wang, W., & Hendricks Brown, C. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0363-y · Cited by 148 · see citing works
Spoth, R. L., Guyll, M., Lillehoj, C. J., Redmond, C. R., & Greenberg, M. T. (2007) · Journal of Community Psychology · doi:10.1002/jcop.20207 · Cited by 145 · see citing works
Taking preventive intervention to scale: The nurse-family partnership
Olds, D. L., Hill, P. L., O'Brien, R., Racine, D., & Moritz, P. (2003) · Cognitive and Behavioral Practice · doi:10.1016/s1077-7229(03)80046-9 · Cited by 140 · see citing works
Atkins, D. C., Mark Steyvers, B., Imel, Z. E., & Smyth, P. (2014) · Implementation Science · doi:10.1186/1748-5908-9-49 · Cited by 130 · see citing works
Handbook of Family Therapy: The Science and Practice of Working with Families and Couples
Geist, R. (2005) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.2005.59.2.170 · Cited by 121 · see citing works
Leve, L. D., Fisher, P. A., & Chamberlain, P. (2009) · Journal of Personality · doi:10.1111/j.1467-6494.2009.00603.x · Cited by 119 · see citing works
Brown, C. H., & Liao, J. (1999) · American Journal of Community Psychology · doi:10.1023/a:1022142021441 · Cited by 116 · see citing works
Self-citationHorwitz, S. M., Chamberlain, P., Landsverk, J. A., & Mullican, C. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0274-3 · Cited by 111 · see citing works
Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., Marsenich, L., Sosna, T., Padgett, C., & Bouwman, G. (2008) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0167-x · Cited by 100 · see citing works
Self-citationLandsverk, J. A., Brown, C. H., Rolls-Reutz, J., Palinkas, L. A., & Horwitz, S. M. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0315-y · Cited by 89 · see citing works
Self-citationBrown, C. H., Chamberlain, P., Saldana, L., Padgett, C., Wang, W., & Cruden, G. (2014) · Implementation Science · doi:10.1186/s13012-014-0134-8 · Cited by 89 · see citing works
Self-citationSupporting Implementation: The Role of Community Development Teams to Build Infrastructure
Saldana, L., & Chamberlain, P. (2012) · American Journal of Community Psychology · doi:10.1007/s10464-012-9503-0 · Cited by 65 · see citing works
Privacy Enhancing Technologies
Weis, S. A. (2006) · IEEE Security & Privacy · doi:10.1109/msp.2006.130 · Cited by 61 · see citing works
Brown, C. H., Mohr, D. C., Gallo, C. G., Mader, C., Palinkas, L., Wingood, G., Prado, G., Kellam, S. G., Pantin, H., Poduska, J., et al. (2013) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/qai.0b013e31829372bd · Cited by 55 · see citing works
Self-citationPisani, A. R., Wyman, P. A., Mohr, D. C., Perrino, T., Gallo, C. G., Villamar, J. A., Kendziora, K. T., Howe, G. W., Sloboda, Z., & Brown, C. H. (2016) · Prevention Science · doi:10.1007/s11121-016-0664-1 · Cited by 26 · see citing works
Self-citationGallo, C. G., Pantin, H. M., Villamar, J. A., Prado, G. J., Tapia, M. I., Ogihara, M., Cruden, G., & Brown, C. H. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0538-4 · Cited by 24 · see citing works
Self-citationPalinkas, L. A., Holloway, I. W., Rice, E., Brown, C. H., Valente, T. W., & Chamberlain, P. (2013) · Implementation Science · doi:10.1186/1748-5908-8-133 · Cited by 20 · see citing works
Self-citationUnderstanding the art and science of implementation in the SAAF efficacy trial
Berkel, C., Murry, V. M., Roulston, K., & Brody, G. H. (2013) · Health Education · doi:10.1108/09654281311329240 · Cited by 19 · see citing works
Institute of Medicine and National Research Council (2014) · National Academies Press eBooks · doi:10.17226/18808 · Cited by 19 · see citing works
Fosco, G. M., Seeley, J. R., Dishion, T. J., Smolkowski, K., Stormshak, E. A., Downey-McCarthy, R. R., Falkenstein, C. A., Moore, K. J., & Strycker, L. A. (2013) · Issues in clinical child psychology · doi:10.1007/978-1-4614-7624-5_18 · Cited by 15 · see citing works
O’Brien, R. A., & Baca, R. P. (1997) · Journal of Community Psychology · doi:10.1002/(sici)1520-6629(199701)25:1 · Cited by 13 · see citing works
Traceable and Automatic Compliance of Privacy Policies in Federated Digital Identity Management
Squicciarini, A., Bhargav-Spantzel, A., Czeskis, A., et al. (2006) · Lecture notes in computer science · doi:10.1007/11957454_5 · Cited by 12 · see citing works
Scaling Up Treatment Foster Care Oregon: A Randomized Trial of Two Implementation Strategies
Chamberlain, P., & Saldana, L. (2015) · Psychology Press eBooks · doi:10.4324/9781315764917-16 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [9]Chamberlain P Brown CH Saldana L Reid J Wang W Marsenich L Sosna T Padgett C Bouwman G Engaging and recruiting counties in an experiment on implementing evidence-based practice in california Adm Policy Mental Health Mental Health Serv Res 2008 35 4 250 60 10.1007/s10488-008-0167-x PMC2562896 18302015 · DOI · Paper page
- [17]Gallo C Pantin H Villamar J Prado G Tapia M Ogihara M Cruden G Brown CH Blending qualitative and computational linguistics methods for fidelity assessment: experience with the familias unidas preventive intervention Adm Policy Mental Health 2015 42 5 574 85 10.1007/s10488-014-0538-4 PMC4125546 24500022 · DOI
- [32]Saldana L Chamberlain P Wang W Brown CH Predicting program start-up using the stages of implementation measure Admin Policy Mental Health Mental Health Serv Res 2012 39 6 419 25 10.1007/s10488-011-0363-y PMC3212640 21710257 · DOI · Paper page
- [40]Wang W Saldana L Chamberlain P Brown C To proceed or not to proceed: predicting implementation success 4th Annual NIH Conference on the Science of Dissemination and Implementation 2011 Washington, DC National Institutes of Health and Academy Health
- [42]Winkler WE Data cleaning methods Proceedings ACM SIGKDD workshop on data cleaning, record linkage, and object consolidation 2003 Washington, D.C. American Computing Machinery
Cited by
Sorted by citations
“Scaling-out” evidence-based interventions to new populations or new health care delivery systems
Aarons, G. A., Sklar, M., Mustanski, B., Benbow, N., & Brown, C. H. (2017) · Implementation Science · doi:10.1186/s13012-017-0640-6 · Cited by 399 · see citing works
Self-citationImplementation strategies for digital mental health interventions in health care settings.
Graham, A. K., Lattie, E. G., Powell, B. J., Lyon, A. R., Smith, J. D., Schueller, S. M., Stadnick, N. A., Brown, C. H., & Mohr, D. C. (2020) · American Psychologist · doi:10.1037/amp0000686 · Cited by 236 · see citing works
Self-citationPalinkas, L. A., Chou, C.-P., Spear, S. E., Mendon, S. J., Villamar, J., & Brown, C. H. (2020) · Implementation Science · doi:10.1186/s13012-020-01030-x · Cited by 70 · see citing works
Self-citation
- Authors
- Wang, D., Ogihara, M., Gallo, C., Villamar, J. A., Smith, J. D., Vermeer, W., Cruden, G., Benbow, N., & Brown, C. H.
- Affiliations
- Florida Atlantic University (US)
- University of Miami (US)
- Northwestern University (US)
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Brown, C. Hendricks
- ORCID
- Venue
- Implementation Science
- Published
- 2015-12
- Volume
- 11
- Issue
- 1
- Pages
- 119-119
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC5011842
- OpenAlex
- W2512901908
- Cited by
- 17
- References
- 39
- Citations by year
- 2026: 1 · 2023: 1 · 2022: 2 · 2021: 3 · 2020: 6 · 2019: 1 · 2018: 2 · 2017: 1
- FWCI
- 1.83
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute on Drug Abuse (DA027828)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- implementation monitoring, communication logs, text analysis, Multidimensional Treatment Foster Care, non-negative matrix factorization, decision rules, automated monitoring, implementation fidelity
- MeSH terms
- Machine Learning, California, Communication, Computer Simulation, Diffusion of Innovation, Feasibility Studies, Foster Home Care, Humans, Mathematics, Medical Informatics, Records, Sensitivity and Specificity, Information Storage and Retrieval, Translational Research, Biomedical, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
