CoChat

Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification

Open access
Atkins, David C, Steyvers, Mark, Imel, Zac E, Smyth, Padhraic · 2014 · Implementation Science
Cited by 127

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Behavioral interventions such as psychotherapy are leading, evidence-based practices for a variety of problems (e.g., substance abuse), but the evaluation of provider fidelity to behavioral interventions is limited by the need for human judgment. The current study evaluated the accuracy of statistical text classification in replicating human-based judgments of provider fidelity in one specific psychotherapy--motivational interviewing (MI). METHOD: Participants (n = 148) came from five previously conducted randomized trials and were either primary care patients at a safety-net hospital or university students. To be eligible for the original studies, participants met criteria for either problematic drug or alcohol use. All participants received a type of brief motivational interview, an evidence-based intervention for alcohol and substance use disorders. The Motivational Interviewing Skills Code is a standard measure of MI provider fidelity based on human ratings that was used to evaluate all therapy sessions. A text classification approach called a labeled topic model was used to learn associations between human-based fidelity ratings and MI session transcripts. It was then used to generate codes for new sessions. The primary comparison was the accuracy of model-based codes with human-based codes. RESULTS: Receiver operating characteristic (ROC) analyses of model-based codes showed reasonably strong sensitivity and specificity with those from human raters (range of area under ROC curve (AUC) scores: 0.62 - 0.81; average AUC: 0.72). Agreement with human raters was evaluated based on talk turns as well as code tallies for an entire session. Generated codes had higher reliability with human codes for session tallies and also varied strongly by individual code. CONCLUSION: To scale up the evaluation of behavioral interventions, technological solutions will be required. The current study demonstrated preliminary, encouraging findings regarding the utility of statistical text classification in bridging this methodological gap.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1748-5908-9-49

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 20

  1. Glynn LH, Hallgren KA, Houck JM, Moyers TB: CACTI: free, open-source software for the sequential coding of behavioral interactions. PLoS One. 2012, 7 (7): e39740-10.1371/journal.pone.0039740. doi: 10.1371/journal.pone.0039740 · DOI
  2. Blei D, Ng AY, Jordan MI: Latent Dirichlet allocation. J Mach Learn Res. 2003, 3: 993-1022.
  3. Steyvers M, Griffiths T: Probabilistic topic models. Latent Semantic Analysis: A Road to Meaning. Edited by: Landauer T, McNamara D, Dennis S, Kintsch W. 2006, Mahwah, NJ: Erlbaum, 427-448.
  4. Atkins DC, Rubin TN, Steyvers M, Doeden M, Baucom B, Christensen A: Topic models: a novel method for modeling couple and family text data. J Fam Psychol. 2012, 26: 816-827. · DOI · Paper page
  5. Rubin T, Chambers A, Smyth P, Steyvers M: Statistical topic models for multi-label document classification. J Mach Learn. 2012, 88 (1): 157-208. · DOI · Paper page
  6. Krupski A, Joesch JM, Dunn C, Donovan D, Bumgardner K, Lord SP, Ries R, Roy-Byrne P: Testing the effects of brief intervention in primary care for problem drug use in a randomized controlled trial: rationale, design, and methods. Addict Sci Clin Pract. 2012, 7 (1): 27-10.1186/1940-0640-7-27. · DOI · Paper page
  7. Lee CM, Neighbors C, Lewis MA, Kaysen D, Mittman A, Geisner IM, Atkins DC, Zheng C, Garberson LA, Kilmer JR, Larimer ME: Randomized controlled trial of a spring break intervention to reduce high-risk drinking. J Consult Clin Psychol. 2014, 82 (2): 189-201. · DOI
  8. Neighbors C, Lee CM, Atkins DC, Lewis MA, Kaysen D, Mittmann A, Fossos N, Geisner IM, Zheng C, Larimer ME: A randomized controlled trial of event-specific prevention strategies for reducing problematic drinking associated with 21st birthday celebrations. J Consult Clin Psychol. 2012, 80 (5): 850-862. · DOI · Paper page
  9. Tollison SJ, Lee CM, Neighbors C, Neil TA, Olson ND, Larimer ME: Questions and reflections: The use of motivational interviewing microskills in a peer-led brief alcohol intervention for college students. Behav Ther. 2008, 39 (2): 183-194. 10.1016/j.beth.2007.07.001. · DOI
  10. Miller WR, Moyers TB, Ernst DB, Amrhein PC: Manual for the Motivational Interviewing Skill Code (MISC), Version 2.1. 2008, New Mexico: Center on Alcoholism, Substance Abuse, and Addictions, The University of New Mexico
  11. Forsberg L, Kallmen H, Hermansson U, Berman AH, Helgason AR: Coding counselor behavior in motivational sessions: Inter-rater reliability for the Swedish Motivational Interviewing Treatment Integrity Code (MITI). Cognit Behav Ther. 2007, 36 (3): 162-169. 10.1080/16506070701339887. · DOI
  12. Kirschenbaum H: Carl Rogers’s life and work: an assessment on the 100th anniversary of his birth. J Counsel Dev. 2004, 82 (1): 116-124. 10.1002/j.1556-6678.2004.tb00293.x. · DOI
  13. Rogers CR: Counseling and Psychotherapy: Newer Concepts in Practice. 1942, Boston: Houghton Mifflin
  14. Miller W, Sorensen J, Selzer J: Disseminating evidence-based practices in substance abuse treatment: a review with suggestions. J Subst Abuse Treat. 2006, 31: 25-39. 10.1016/j.jsat.2006.03.005. · DOI · Paper page
  15. Can D, Gibson J, Vaz C, Georgiou PG, Narayanan SS: Barista: A Framework for Concurrent Speech Processing from USC-SAIL. Proceedings of IEEE International Conference on Audio, Speech, and Signal Processing (ICASSP): 4-9. 2014, Florence, Italy: Institute of Electrical and Electronics Engineers, http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000002 , May ,
  16. Georgiou PG, Black M, Lambert A, Baucom B, Narayanan SS: That’s Aggravating, Very Aggravating: Is it Possible to Classify Behaviors in Couple Interactions Using Automatically Derived Lexical Features?. Proceedings of Affective Computing and Intelligent Interaction: 9-12 October 2011; Memphis, TN. 2011, Berlin, Heidelberg: Springer-Verlag, 87-96. · DOI
  17. Apodaca TR, Longabaugh R: Mechanisms of change in MI: a review and preliminary evaluation of the evidence. Addiction. 2009, 104: 705-715. 10.1111/j.1360-0443.2009.02527.x. · DOI · Paper page
  18. Moyers TB, Martin T, Christopher PJ, Houck JM, Tonigan JS, Amrhein PC: Client language as a mediator of motivational interviewing efficacy: Where is the evidence?. Alcohol Clin Exp Res. 2007, 31 (Suppl 3): 40S-47S. · DOI · Paper page
  19. Can D, Georgiou PG, Atkins DC, Narayanan SS: A Case Study: Detecting Counselor Reflections in Psychotherapy for Addictions Using Linguistic Features. 2012, Paper presented at the Annual Conference of the International Speech Communication Association: 9-13 September 2012; Portland, OR
  20. Black MP, Katsamanis A, Baucom BR, Lee CC, Lammert AC, Christensen A, Georgiou PG, Narayanan SS: Toward automating a human behavioral coding system for married couples’ interactions using speech acoustic features. Speech Comm. 2013, 55: 1-21. 10.1016/j.specom.2011.12.003. · DOI

Cited by

See all 127

Metadata

Authors
Atkins, D. C., Steyvers, M., Imel, Z. E., & Smyth, P.
Venue
Implementation Science
Published
2014-04-24
Volume
9
Issue
1
Pages
49-49
ISSN
1748-5908 (ISSN-L)
OpenAlex
W2154608152
Cited by
127
References
8
FWCI
10.13
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.