"It's hard to argue with a computer": Investigating Psychotherapists' Attitudes towards Automated Evaluation
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
We present CORE-MI, an automated evaluation and assessment system that provides feedback to mental health counselors on the quality of their care. CORE-MI is the first system of its kind for psychotherapy, and an early example of applied machine-learning in a human service context. In this paper, we describe the CORE-MI system and report on a qualitative evaluation with 21 counselors and trainees. We discuss the applicability of CORE-MI to clinical practice and explore user perceptions of surveillance, workplace misuse, and notions of objectivity, and system reliability that may apply to automated evaluation systems generally.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lundahl, B., & Burke, B. L. (2009) · Journal of Clinical Psychology · doi:10.1002/jclp.20638 · Cited by 573 · cited 1×
“Although MI has demonstrated efficacy in many research-based clinical trials [32], the quality of MI delivered in real-world settings can be highly variable, due to heterogeneity in the quality of training and supervision [40].”
MethodInternet-based psychological treatments for depression
Johansson, R., & Andersson, G. (2012) · Expert Review of Neurotherapeutics · doi:10.1586/ern.12.63 · Cited by 509
Sweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
Sustaining motivational interviewing: A meta-analysis of training studies
Schwalbe, C. S., Oh, H. Y., & Zweben, A. (2014) · Addiction · doi:10.1111/add.12558 · Cited by 343
Christensen, H., Griffiths, K., Groves, C., & Korten, A. (2006) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2006.01743.x · Cited by 201 · cited 1×
“There are also established services that provide mediated counseling via the Internet and telephony networks (e.g., [29]), although prior work has found high attrition rates among users of these services [12].”
BackgroundModerated online social therapy: Designing and evaluating technology for mental health
Lederman, R., Wadley, G., Gleeson, J., Bendall, S., & Álvarez-Jiménez, M. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2513179 · Cited by 151 · cited 1×
“There are also established services that provide mediated counseling via the Internet and telephony networks (e.g., [29]), although prior work has found high attrition rates among users of these services [12].”
BackgroundComputational psychotherapy research: scaling up the evaluation of patient–provider interactions
Imel, Z. E., Steyvers, M., & Atkins, D. C. (2015) · Psychotherapy · doi:10.1037/a0036841 · Cited by 146 · cited 1×
“Unfortunately, this approach is time consuming, expensive, and does not scale up to real-world settings [1, 23].”
BackgroundSelf-citationModeling and Understanding Visual Attributes of Mental Health Disclosures in Social Media
Manikonda, L., & De Choudhury, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025932 · Cited by 116 · cited 1×
“For instance, there have been recent efforts to automatically identify and diagnose suicidal ideation [15], insomnia [25], and other mental health disclosures [33] by analyzing social media [3] and mediated speech [10].”
BackgroundA Social Media Based Index of Mental Well-Being in College Campuses
Bagroy, S., Kumaraguru, P., & De Choudhury, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025909 · Cited by 115
Designing Contestability: Interaction Design, Machine Learning, and Mental Health
Hirsch, T., Merced, K., Narayanan, S., Imel, Z. E., & Atkins, D. C. (2017) · Proceedings of the 2017 Conference on Designing Interactive Systems · doi:10.1145/3064663.3064703 · Cited by 110 · cited 2×
“While it may be “hard to argue with a computer,” as designers it is our responsibility to create mechanisms that enable users to do precisely that – particularly when the predictions that our systems make have the potential to adversely impact human welfare [19].”
BackgroundSelf-citationYang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 1×
“At the same time, they are finding their way into human service domains, including healthcare [52], public safety [16], and transportation planning.”
Background
Cited by
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
“[78] Audio (counselling session) Improving treatment User study of a ML system to automatically assess the motivational Interviewing (MI) skills of psychotherapists directly from the audio recording of a counselling session.”
BackgroundInfluentialA Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179
Tanana, M. J., Soma, C. S., Srikumar, V., Atkins, D. C., & Imel, Z. E. (2019) · Journal of Medical Internet Research · doi:10.2196/12529 · Cited by 100
Self-citationGooding, P., & Kariotis, T. (2021) · JMIR Mental Health · doi:10.2196/24668 · Cited by 100
Yildirim, N., Richardson, H., Wetscherek, M. T., Bajwa, J., Jacob, J., Pinnock, M. A., Harris, S., Coelho De Castro, D., Bannur, S., Hyland, S., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642013 · Cited by 82
Metadata
- Authors
- Hirsch, T., Soma, C., Merced, K., Kuo, P., Dembe, A., Caperton, D. D., Atkins, D. C., & Imel, Z. E.
- Venue
- Proceedings of the 2018 Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2018-06-08
- Volume
- 2018
- Pages
- 559-571
- OpenAlex
- W2808705134
- Cited by
- 42
- References
- 11
- FWCI
- 9.05
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.