Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Mental illness is one of the most pressing public health issues of our time. While counseling and psychotherapy can be effective treatments, our knowledge about how to conduct successful counseling conversations has been limited due to lack of large-scale data with labeled outcomes of the conversations. In this paper, we present a large-scale, quantitative study on the discourse of text-message-based counseling conversations. We develop a set of novel computational discourse analysis methods to measure how various linguistic aspects of conversations are correlated with conversation outcomes. Applying techniques such as sequence-based conversation models, language model comparisons, message clustering, and psycholinguistics-inspired word frequency analyses, we discover actionable conversation strategies that are associated with better conversation outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, innovations in computer science (e.g., natural language processing and machine learning) are being used to assess language patterns of successful crisis interventions in text-based mental health conversations (18, 35).”
From Key Considerations for Incorporating Conversational AI in Psychotherapy (2019) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“We also used distributed word embeddings (e.g., (Mikolov et al., 2013)) instead of TF-IDF vectors but found the latter to produce better clusters.”
MethodVADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · cited 1×
“Texters start explaining their issue largely in terms of the past and present but over time talk more about the future (see Figure 9A; each plot shows the relative amount of words in the LIWC past, present, and future categories (Tausczik and Pennebaker, 2010)).”
Psychological Aspects of Natural Language Use: Our Words, Our Selves
Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145041 · Cited by 2,624
Concreteness ratings for 40 thousand generally known English word lemmas
Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0403-5 · Cited by 1,821
Lenggenhager, J. (1947) · PubMed · Cited by 1,772
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,108
Therapeutic Discourse: Psychotherapy as Conversation
Tannen, D., Labov, W., & Fanshel, D. (1981) · Language · doi:10.2307/413709 · Cited by 938
Pyszczynski, T., & Greenberg, J. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.102.1.122 · Cited by 825 · cited 4×
“Figure 9B shows that a smaller amount of self-focus is associated with more successful conversations (providing support for the self-focus model of depression (Pyszczynski and Greenberg, 1987)).”
BackgroundResultInfluentialLinguistic Style Matching in Social Interaction
Niederhoffer, K. G., & Pennebaker, J. W. (2002) · Journal of Language and Social Psychology · doi:10.1177/026192702237953 · Cited by 562
Unsupervised Modeling of Twitter Conversations
Ritter, A., Cherry, C., & Dolan, B. (2010) · NPARC · Cited by 528
Assessing the economic costs of serious mental illness
Insel, T. R. (2008) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.08030366 · Cited by 487 · cited 1×
“In addition to the person directly experiencing a mental illness, family, friends, and communities are also affected (Insel, 2008).”
BackgroundThe Secret Life of Pronouns: Flexibility in Writing Style and Physical Health
Campbell, R. S., & Pennebaker, J. W. (2003) · Psychological Science · doi:10.1111/1467-9280.01419 · Cited by 430
Pyszczynski, T., Holt, K., & Greenberg, J. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.52.5.994 · Cited by 303 · cited 3×
“Prior work has shown that day-to-day changes in writing style are associated with positive health outcomes (Campbell and Pennebaker, 2003), and existing theories link depression to a negative view of the future (Pyszczynski et al., 1987) and a selffocusing style (Pyszczynski and Greenberg, 1987).”
BackgroundResultCatching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization
Barzilay, R., & Lee, L. (2004) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0405039 · Cited by 295 · cited 1×
“Most approaches employ Hidden Markov Model-like models (Barzilay and Lee, 2004; Ritter et al., 2010; Paul, 2012; Yang et al., 2014) which are also used in this work to model progression through conversation stages.”
MethodEchoes of power: language effects and power differences in social interaction
Danescu-Niculescu-Mizil, C., Lee, L., Pang, B., & Kleinberg, J. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187931 · Cited by 285 · cited 3×
“Large-scale studies have revealed subtle dynamics in conversations such as co-ordination or style matching effects (Niederhoffer and Pennebaker, 2002; Danescu-Niculescu-Mizil, 2012) as well as expressions of social power and status (Bramsen et al., 2011; Danescu-Niculescu-Mizil et al., 2012).”
BackgroundSentiment Analysis of Suicide Notes: A Shared Task
Pestian, J. P., Matykiewicz, P., Linn-Gust, M., South, B., Uzuner, O., Wiebe, J., Cohen, K. B., Hurdle, J., & Brew, C. (2012) · Biomedical Informatics Insights · doi:10.4137/bii.s9042 · Cited by 210 · cited 3×
“Prior work has shown that day-to-day changes in writing style are associated with positive health outcomes (Campbell and Pennebaker, 2003), and existing theories link depression to a negative view of the future (Pyszczynski et al., 1987) and a selffocusing style (Pyszczynski and Greenberg, 1987).”
BackgroundAn Evaluation of Crisis Hotline Outcomes Part 1: Nonsuicidal Crisis Callers
Kalafat, J., Gould, M. S., Munfakh, J. L. H., & Kleinman, M. (2007) · Suicide and Life-Threatening Behavior · doi:10.1521/suli.2007.37.3.322 · Cited by 162 · cited 2×
“We capture the time dimension by breaking up each conversation into five even chunks of messages.”
BackgroundResultNonparametric bootstrapping for hierarchical data
Ren, S., Lai, H., Tong, W., Aminzadeh, M., Hou, X., & Lai, S. (2010) · Journal of Applied Statistics · doi:10.1080/02664760903046102 · Cited by 148 · cited 1×
“Error bars in all plots correspond to bootstrapped 95% confidence intervals using the member bootstrapping technique from (Ren et al., 2010).”
MethodHow to Ask for a Favor: A Case Study on the Success of Altruistic Requests
Althoff, T., Danescu-Niculescu-Mizil, C., & Jurafsky, D. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14547 · Cited by 144
Self-citationRamirez-Esparza, N., Chung, C., Kacewic, E., & Pennebaker, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v2i1.18623 · Cited by 132 · cited 2×
“Previous work also found that there are linguistic cues associated with depression (Ramirez-Esparza et al., 2008; Campbell and Pennebaker, 2003) as well as with suicude (Pestian et al.”
BackgroundExtracting Social Power Relationships from Natural Language
Bramsen, P., Escobar-Molano, M. L., Patel, A., & Alonso, R. D. H. (2011) · Cited by 100
Success with Style: Using Writing Style to Predict the Success of Novels
Ganjigunte Ashok, V., Feng, S., & Choi, Y. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1181 · Cited by 84 · cited 2×
“Other studies have connected writing to measures of success in the context of requests (Althoff et al., 2014), user retention (Althoff and Leskovec, 2015), novels (Ashok et al., 2013), and scientific abstracts (Guerini et al., 2012).”
BackgroundThe Experience of Online Counseling
Haberstroh, S., Duffey, T., Evans, M., Gee, R., & Trepal, H. (2007) · Journal of Mental Health Counseling · doi:10.17744/mehc.29.3.j344651261w357v2 · Cited by 83 · cited 1×
“Thus far, quantitative evidence for effective conversation strategies has been scarce, since most studies on counseling have been limited to very small sample sizes and qualitative observations ( e.g. , Labov and Fanshel, (1977); Haberstroh et al., (2007)).”
BackgroundFinding progression stages in time-evolving event sequences
Yang, J., McAuley, J., Leskovec, J., LePendu, P., & Shah, N. H. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568044 · Cited by 59 · cited 1×
“Most approaches employ Hidden Markov Model-like models (Barzilay and Lee, 2004; Ritter et al., 2010; Paul, 2012; Yang et al., 2014) which are also used in this work to model progression through conversation stages.”
MethodSelf-citationDo Linguistic Style and Readability of Scientific Abstracts Affect their Virality?
Guerini, M., Pepe, A., & Lepri, B. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v6i1.14305 · Cited by 46
Promoting Tolerance for Ambiguity in Counselor Training Programs
Levitt, D. H., & JACQUES, J. D. (2005) · The Journal of Humanistic Counseling Education and Development · doi:10.1002/j.2164-490x.2005.tb00055.x · Cited by 45 · cited 1×
“We then compute the average duration in messages of each stage for both more and less successful counselors.”
MethodMixed Membership Markov Models for Unsupervised Conversation Modeling
Paul, M. J. (2012) · Cited by 36
Stacked Generalization Learning to Analyze Teenage Distress
Dinakar, K., Weinstein, E., Lieberman, H. A., & Selman, R. L. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14527 · Cited by 28 · cited 2×
“Other work has explored how novel interfaces based on topic models can support counselors during conversations (Dinakar et al., 2014a; 2014b; 2015; Chen, 2014).”
BackgroundReal-time Topic Models for Crisis Counseling
Dinakar, K., Chaney, A. J. B., Lieberman, H. A., & Blei, D. M. (2014) · Cited by 12
Visualizations for mental health topic models
Chen, G. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 12
Research works printed but not linked: 5
- Levitt Dana Heller, 2005, The Journal of Humanistic Counseling, Education and Development, 44(1).
- Niederhoffer Kate G., 2002, Journal of Language and Social Psychology, 21(4).
- Pyszczynski Tom, 1987, Journal of Personality and Social Psychology, 52(5).
- Ren Shiquan, 2010, Journal of Applied Statistics, 37(9).
- Tausczik Yla R., 2010, Journal of Language and Social Psychology, 29(1).
Cited by
Artificial intelligence for mental health and mental illnesses: an overview
Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1094-0 · Cited by 970
“DL is ideal for discovering intricate structures in high-dimensional data like those contained in clinician notes in EHRs [45], or clinical and non-clinical data provided by patients [47, 48].”
BackgroundHo, A., Hancock, J., & Miner, A. S. (2018) · Journal of Communication · doi:10.1093/joc/jqy026 · Cited by 629
Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409
COSMIC: COmmonSense knowledge for eMotion Identification in Conversations
Ghosal, D., Majumder, N., Gelbukh, A., Mihalcea, R., & Poria, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.224 · Cited by 361
Depression and Self-Harm Risk Assessment in Online Forums
Yates, A., Cohan, A., & Goharian, N. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1322 · Cited by 323
Metadata
- Authors
- Althoff, T., Clark, K., & Leskovec, J.
- Affiliations
- Stanford University (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2016-12
- Volume
- 4
- Pages
- 463-476
- ISSN
- 2307-387X (ISSN-L)
- PMC
- PMC5361062
- OpenAlex
- W2484061479
- Cited by
- 322
- References
- 31
- Citations by year
- 2026: 33 · 2025: 38 · 2024: 25 · 2023: 39 · 2022: 44 · 2021: 37 · 2020: 40 · 2019: 28 · 2018: 18 · 2017: 18 · 2016: 1
- FWCI
- 34.02
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1010921, 1149837)
- National Institutes of Health
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative
- National Institute of Biomedical Imaging and Bioengineering (U54EB020405)
- Army Research Office
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- counseling conversations, psychotherapy, computational discourse analysis, conversational strategies, message clustering, psycholinguistics, word frequency analysis, language model comparison, mental health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.