Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Mental illness is one of the most pressing public health issues of our time. While counseling and psychotherapy can be effective treatments, our knowledge about how to conduct successful counseling conversations has been limited due to lack of large-scale data with labeled outcomes of the conversations. In this paper, we present a large-scale, quantitative study on the discourse of text-message-based counseling conversations. We develop a set of novel computational discourse analysis methods to measure how various linguistic aspects of conversations are correlated with conversation outcomes. Applying techniques such as sequence-based conversation models, language model comparisons, message clustering, and psycholinguistics-inspired word frequency analyses, we discover actionable conversation strategies that are associated with better conversation outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, innovations in computer science (e.g., natural language processing and machine learning) are being used to assess language patterns of successful crisis interventions in text-based mental health conversations (18, 35).”
From Key Considerations for Incorporating Conversational AI in Psychotherapy (2019) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“We also used distributed word embeddings (e.g., (Mikolov et al., 2013)) instead of TF-IDF vectors but found the latter to produce better clusters.”
MethodVADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · cited 1×
“Texters start explaining their issue largely in terms of the past and present but over time talk more about the future (see Figure 9A; each plot shows the relative amount of words in the LIWC past, present, and future categories (Tausczik and Pennebaker, 2010)).”
Psychological Aspects of Natural Language Use: Our Words, Our Selves
Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145041 · Cited by 2,624
Concreteness ratings for 40 thousand generally known English word lemmas
Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0403-5 · Cited by 1,821
Lenggenhager, J. (1947) · PubMed · Cited by 1,772
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,108
Therapeutic Discourse: Psychotherapy as Conversation
Tannen, D., Labov, W., & Fanshel, D. (1981) · Language · doi:10.2307/413709 · Cited by 938
Pyszczynski, T., & Greenberg, J. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.102.1.122 · Cited by 825 · cited 4×
“Figure 9B shows that a smaller amount of self-focus is associated with more successful conversations (providing support for the self-focus model of depression (Pyszczynski and Greenberg, 1987)).”
BackgroundResultInfluentialLinguistic Style Matching in Social Interaction
Niederhoffer, K. G., & Pennebaker, J. W. (2002) · Journal of Language and Social Psychology · doi:10.1177/026192702237953 · Cited by 562
Research works printed but not linked: 5
- Levitt Dana Heller, 2005, The Journal of Humanistic Counseling, Education and Development, 44(1).
- Niederhoffer Kate G., 2002, Journal of Language and Social Psychology, 21(4).
- Pyszczynski Tom, 1987, Journal of Personality and Social Psychology, 52(5).
- Ren Shiquan, 2010, Journal of Applied Statistics, 37(9).
- Tausczik Yla R., 2010, Journal of Language and Social Psychology, 29(1).
Cited by
Artificial intelligence for mental health and mental illnesses: an overview
Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1094-0 · Cited by 970
“DL is ideal for discovering intricate structures in high-dimensional data like those contained in clinician notes in EHRs [45], or clinical and non-clinical data provided by patients [47, 48].”
BackgroundHo, A., Hancock, J., & Miner, A. S. (2018) · Journal of Communication · doi:10.1093/joc/jqy026 · Cited by 629
Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409
COSMIC: COmmonSense knowledge for eMotion Identification in Conversations
Ghosal, D., Majumder, N., Gelbukh, A., Mihalcea, R., & Poria, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.224 · Cited by 361
Depression and Self-Harm Risk Assessment in Online Forums
Yates, A., Cohan, A., & Goharian, N. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1322 · Cited by 323
Challenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288
“This issue is mainly about how to optimize the behaviors and strategies of a dialog system to maximize long-term user engagement and accomplish long-term, complex goals such as providing emotional comfort or even psychological counseling [3, 12, 101, 171].”
BackgroundMorris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018) · Journal of Medical Internet Research · doi:10.2196/10148 · Cited by 273
“Furthermore, other uses of large corpora of empathic chats have demonstrated that such data can provide important contributions to the understanding of actionable strategies to enhance crisis counseling [26] and therapist-client interactions [27].”
BackgroundA Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Sharma, A., Miner, A., Atkins, D., & Althoff, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.425 · Cited by 268
“Previous work in NLP for mental health has focused on analysis of effective conversation strategies (Althoff et al., 2016; Pérez-Rosas et al., 2019; Zhang and Danescu-Niculescu-Mizil, 2020), identification of therapeutic actions (Lee et al.”
BackgroundSelf-citationVelupillai, S., Suominen, H., Liakata, M., Roberts, A., Shah, A. D., Morley, K., Osborn, D., Hayes, J., Stewart, R., Downs, J., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.10.005 · Cited by 247
Natural Language Processing for Smart Healthcare
Zhou, B., Yang, G., Shi, Z., & Ma, S. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3210270 · Cited by 230
“For example, the insights of [223] could help improve counselor training and generate real-time counseling quality monitoring and answer suggestion support tools.”
BackgroundMalgaroli, M., Hull, T. D., Zech, J. M., & Althoff, T. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02592-2 · Cited by 201
“Search string queries are detailed in the supplementary materials.”
BackgroundMethodSelf-citationA Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450097 · Cited by 163
“These efforts have predominantly focused on analyzing techniques that are effective for seeking and providing conversational support such as adaptability to various contexts and diversity of responses [1, 49, 60, 71, 72].”
BackgroundSelf-citationKey Considerations for Incorporating Conversational AI in Psychotherapy
Miner, A. S., Shah, N., Bullock, K. D., Arnow, B. A., Bailenson, J., & Hancock, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00746 · Cited by 166
“, natural language processing and machine learning) are being used to assess language patterns of successful crisis interventions in text-based mental health conversations (18, 35).”
BackgroundThe Effectiveness of Crisis Line Services: A Systematic Review
Hoffberg, A. S., Stearns-Yoder, K. A., & Brenner, L. A. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00399 · Cited by 150
Using Social Media for Mental Health Surveillance: A Review
Skaik, R., & Inkpen, D. (2021) · ACM Computing Surveys · doi:10.1145/3422824 · Cited by 143
“Althoff et al. [2016] experimented with roughly 15K counselor messages to evaluate the linguistic aspects of effective therapy.”
BackgroundAlthoff, T., Jindal, P., & Leskovec, J. (2017) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3018661.3018672 · Cited by 177
“However, many important behaviors and outcomes pertain to the offline world including political mobilization [16], physical activity [5, 6, 25, 34], food intake [24], mental health [1, 23], obesity [20], and smoking [21].”
BackgroundSelf-citationGoldberg, S. B., Flemotomos, N., Martinez, V. R., Tanana, M. J., Kuo, P. B., Pace, B. T., Villatte, J. L., Georgiou, P. G., Van Epps, J., Imel, Z. E., et al. (2020) · Journal of Counseling Psychology · doi:10.1037/cou0000382 · Cited by 141
O'Leary, K., Schueller, S. M., Wobbrock, J. O., & Pratt, W. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173905 · Cited by 127
“Crisis Textline trains its volunteers online to provide effective text-messaging crisis interventions [1].”
BackgroundMeasuring Depression Symptom Severity from Spoken Language and 3D Facial Expressions
Haque, A., Guo, M., Miner, A. S., & Fei-Fei, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08592 · Cited by 98
“Such a solution could leverage multi-modal sensors or text messages, as is common on modern smartphones, to increase timely and cost-effective symptom screening [3].”
BackgroundChen, J., & Yang, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.336 · Cited by 186
Wang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445645 · Cited by 120
“We draw on prior work [2, 84] to obtain linguistic diversity, and use word embeddings for this purpose.”
MethodSharma, A., Rushton, K., Lin, I. W., Nguyen, T., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642761 · Cited by 102
“…including adaptability and efficacy of counselors [66], personalized vs templated counseling language [2], psychological perspective change [2, 95], staying on topic [95], therapeutic actions [50], empathy [80], moments of change [68], counseling strategies [65, 76], and conversational…”
Self-citationMental distress and language use: Linguistic analysis of discussion forum posts
Lyons, M. T., Aksayli, N. D., & Brewer, G. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.05.035 · Cited by 126
Topooco, N., Byléhn, S., Dahlström Nysäter, E., Holmlund, J., Lindegaard, J., Johansson, S., Åberg, L., Bergman Nordgren, L., Zetterqvist, M., & Andersson, G. (2019) · Journal of Medical Internet Research · doi:10.2196/13393 · Cited by 90
“Technology-based treatment enables new evaluation strategies and the current approach in particular produces vast amounts of data from patient-therapist correspondences that benefit from natural language processing to help understand and refine conversation [98,99].”
BackgroundCASS: Towards Building a Social-Support Chatbot for Online Health Community
Wang, L., Wang, D., Tian, F., Peng, Z., Fan, X., Zhang, Z., Yu, M., Ma, X., & Wang, H. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449083 · Cited by 65
“Based on previous literature [5], and for the research interest of this study, we introduce four dimensions for evaluating AI-generated responses:”
MethodMediated Social Support for Distress Reduction: AI Chatbots vs. Human
Meng, J., Rheu, M. (., Zhang, Y., Dai, Y., & Peng, W. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579505 · Cited by 74
“Natural language processing models could be particularly helpful for generating high quality support dialogues [2][51].”
BackgroundPeng, Z., Guo, Q., Tsang, K. W., & Ma, X. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376695 · Cited by 81
“We do not suggest comments based on the original support-seeking post to ensure that the suggested examples are more coherent to the provider’s thought and are diverse in expressions [1, 32].”
BackgroundBenefits and Harms of Large Language Models in Digital Mental Health
De Choudhury, M., Pendse, S. R., & Kumar, N. (2023) · doi:10.31234/osf.io/y8ax9 · Cited by 78
PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support
Sun, H., Lin, Z., Zheng, C., Liu, S., & Huang, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.130 · Cited by 64
Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction
Sharma, A., Rushton, K., Lin, I., Wadden, D., Lucas, K., Miner, A., Nguyen, T., & Althoff, T. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.555 · Cited by 47
Self-citationShameli, A., Althoff, T., Saberi, A., & Leskovec, J. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3054172 · Cited by 74
“Furthermore, researchers have studied social interactions to better understand how people could be most supportive of others [1, 2].”
BackgroundSelf-citationCausal Factors of Effective Psychosocial Outcomes in Online Mental Health Communities
Saha, K., & Sharma, A. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7326 · Cited by 72
Moments of Change: Analyzing Peer-Based Cognitive Support in Online Mental Health Forums
Pruksachatkun, Y., Pendse, S. R., & Sharma, A. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300294 · Cited by 71
“[6] use computational linguistics to study conversations between people and designated counsellors.”
A Cross-modal Review of Indicators for Depression Detection Systems
Morales, M. R., Scherer, S. J., & Levitan, R. (2017) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w17-3101 · Cited by 83
“Althoff et al. (2016) presented a large-scale quantitative study on the discourse of counseling conversations.”
BackgroundEthical by Design: Ethics Best Practices for Natural Language Processing
Leidner, J. L., & Plachouras, V. (2017) · doi:10.18653/v1/w17-1604 · Cited by 105
“Althoff, Clark and Leskovec (2016) describe an analysis of counseling conversations using NLP meth-ods; here, perhaps because the patients are still alive, even stronger privacy protection is indicated.”
BackgroundPérez‐Rosas, V., Wu, X., Resnicow, K., & Mihalcea, R. F. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1088 · Cited by 118
“Interestingly, this analysis shows the opposite behavior from the study by (Althoff et al., 2016), where successful counselors acted more in control over the conversation.”
BackgroundResultHelping the Helper : Supporting Peer Counselors via AI-Empowered Practice and Feedback
Hsu, S.-L., Shah, R. S., Senthil, P., Ashktorab, Z., Dugan, C., Geyer, W., & Yang, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710993 · Cited by 112
“Most studies on online text-based peer counseling context have focused on the automatic modeling of behaviors from peer counselors and seekers and the correlations between the behaviors and successful outcomes [4, 71, 80, 144, 147], user engagement [5, 125], or support provision [70, 126].”
Li, I., Li, Y., Li, T., Álvarez-Napagao, S., García-Gasulla, D., & Suzumura, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63799-6_27 · Cited by 63
Chikersal, P., Belgrave, D., Doherty, G., Enrique, A., Palacios, J. E., Richards, D., & Thieme, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376341 · Cited by 63
“This surprisingly contradicts previous findings by [3], and thus, requires further research.”
BackgroundResultInfluentialAn Introduction Natural Language Processing
(2019) · doi:10.46632/rmc/1/1/014 · Cited by 63
Engagement Patterns of Peer-to-Peer Interactions on Mental Health Platforms
Sharma, A. K., Choudhury, M., Althoff, T., & Sharma, A. K. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7328 · Cited by 94
Self-citationThe Effect of Moderation on Online Mental Health Conversations
Wadden, D., August, T., Li, Q., & Althoff, T. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18100 · Cited by 80
Self-citationNook, E. C., Hull, T. D., Nock, M. K., & Somerville, L. H. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2114737119 · Cited by 85
Cross-Cultural Differences in the Use of Online Mental Health Support Forums
Pendse, S. R., Niederhoffer, K., & Sharma, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359169 · Cited by 59
It’s going to be okay: Measuring Access to Support in Online Communities
Wang, Z., & Jurgens, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1004 · Cited by 82
“Prior works have also examined the language of support in online support forums for health-related issues (Biyani et al., 2014; Wang et al., 2012; De Choudhury and De, 2014; Althoff et al., 2016; De Choudhury and Kiciman, 2017), often with the aim of improving people’s access.”
The potential of conversational agents to provide a rapid HIV counseling and testing services
van Heerden, A., Ntinga, X., & Vilakazi, K. (2017) · 2017 International Conference on the Frontiers and Advances in Data Science (FADS) · doi:10.1109/fads.2017.8253198 · Cited by 57
“2 million messages sent by 408 counselors, [11] were able to construct a model able to predict, with an accuracy of 59%, the likelihood of a counseling session ending positively, after reviewing the first 20% of a conversation between a client and counselor.”
BackgroundThe Effects of Immersion and Real-World Distractions on Virtual Social Interactions
Oh, C. S., Herrera, F., & Bailenson, J. N. (2019) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2018.0404 · Cited by 67
“13 Finally, telepresence describes the psychological state of feeling physically present in a virtual environment, 14 and social presence refers to the sense of being together in the virtual environment with a real person 15 ; we address both dimensions of presence, in line with extant research.”
Conversational Agents and Mental Health: Theory-Informed Assessment of Language and Affect
Miner, A., Chow, A., Adler, S., Zaitsev, I., Tero, P., Darcy, A., & Paepcke, A. (2016) · Proceedings of the Fourth International Conference on Human Agent Interaction · doi:10.1145/2974804.2974820 · Cited by 55
Modeling Motivational Interviewing Strategies on an Online Peer-to-Peer Counseling Platform
Shah, R. S., Holt, F., Hayati, S. A., Agarwal, A., Wang, Y.-C., Kraut, R. E., & Yang, D. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555640 · Cited by 49
“We utilize client satisfaction as an outcome metric to explore the value of individual MI techniques, following prior work [1, 5, 25].”
BackgroundMethodInfluentialYoung Oh, E., Song, D., & Hong, H. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633119839177 · Cited by 52
Gu, D., Li, M., Yang, X., Gu, Y., Zhao, Y., Liang, C., & Hu, L. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103192 · Cited by 59
Harnessing the Web for Population-Scale Physiological Sensing: A Case Study of Sleep and Performance
Althoff, T., Horvitz, E., White, R. W., & Zeitzer, J. M. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052637 · Cited by 61
“Large-scale interaction data have been used to gain insights into human behavior in the areas of mood rhythms [23], diet [45], conversation strategies [5], social networks and mobile games encouraging health behaviors [7, 8], and health and disease-related search behaviors [36, 46].”
BackgroundSelf-citationPredicting Counselor Behaviors in Motivational Interviewing Encounters
Pérez‐Rosas, V., Mihalcea, R. F., Resnicow, K., Singh, S. P., Ann, L., Goggin, K., & Catley, D. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1106 · Cited by 72
“More recently, (Althoff et al., 2016) explored these phenomena in the mental health domain by analyzing text-message-based counseling”
BackgroundBalancing Objectives in Counseling Conversations: Advancing Forwards or Looking Backwards
Zhang, J., & Danescu-Niculescu-Mizil, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.470 · Cited by 47
Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations
Deng, Y., Zhang, W., Yuan, Y., & Lam, W. P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.225 · Cited by 114
Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop
Chang, J. P., & Danescu-Niculescu-Mizil, C. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1481 · Cited by 91
“terances (Althoff et al., 2016), sentiment imbalance (Niculae et al.”
BackgroundMethodInfluentialA Framework for Applying Natural Language Processing in Digital Health Interventions
Funk, B., Sadeh‐Sharvit, S., Fitzsimmons‐Craft, E. E., Trockel, M., Monterubio, G. E., Goel, N. J., Balantekin, K. N., Eichen, D. M., Flatt, R. E., Firebaugh, M., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/13855 · Cited by 55
“Computational discourse analysis methods have been employed to develop insights on what constitutes effective counseling text conversations as well [19].”
MethodTopaz, M., Koleck, T. A., Onorato, N., Smaldone, A. M., & Henry, S. B. (2020) · Nursing Outlook · doi:10.1016/j.outlook.2020.12.007 · Cited by 49
Concreteness ratings for 62,000 English multiword expressions
Muraki, E. J., Abdalla, S., Brysbaert, M., & Pexman, P. M. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01912-6 · Cited by 64
Althoff, T., Jindal, P., & Leskovec, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03053 · Cited by 42
“However, many important behaviors and outcomes pertain to the offline world including political mobilization [16], physical activity [5, 6, 25, 34], food intake [24], mental health [1, 23], obesity [20], and smoking [21].”
BackgroundSelf-citationFinding Your Voice: The Linguistic Development of Mental Health Counselors
Zhang, J., Filbin, R., Morrison, C., Weiser, J., & Danescu-Niculescu-Mizil, C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1089 · Cited by 49
“…Fanshel, 1977; Miller and Mount, 2001; Catley et al., 2006; Gaume et al., 2010) and computational (Atkins et al., 2014; Tanana et al., 2015, 2016; Althoff et al., 2016; Pérez-Rosas et al., 2018) studies of dialogic interactions in counseling and psychotherapy have highlighted the potential…”
BackgroundMethodResultInfluentialKaya, H., Fedotov, D., Dresvyanskiy, D., Doyran, M., Mamontov, D., Markitantov, M., Salah, A. A. A., Kavcar, E., Karpov, A. A., & Salah, A. A. (2019) · doi:10.1145/3347320.3357691 · Cited by 53
“Language usage [3], facial actions [10], vocal prosody [10], and speech [11] have been known to be associated with mental health, and were used as indicators for automatic analysis.”
MethodHollingshead, K., Ireland, M. E., & Loveys, K. (2017) · doi:10.18653/v1/w17-31 · Cited by 44
Abubakar, A. M., Gupta, D., & Parida, S. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.087 · Cited by 44
Online Social Networks in Health Care: A Study of Mental Disorders on Reddit
Fraga, B. S., da Silva, A. P. C., & Murai, F. (2018) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1109/wi.2018.00-36 · Cited by 42
“This finding corroborates the analysis in [6], where the authors conclude that people who go through periods of great hardship are more likely to think about the future and to be positive, when aided by others who offer help.”
BackgroundResultInfluentialTowards Motivational and Empathetic Response Generation in Online Mental Health Support
Saha, T., Gakhreja, V., Das, A. S., Chakraborty, S., & Saha, S. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531912 · Cited by 59
Goldberg, S. B., Tanana, M., Imel, Z. E., Atkins, D. C., Hill, C. E., & Anderson, T. (2020) · Psychotherapy Research · doi:10.1080/10503307.2020.1741047 · Cited by 45
Dwyer, A., Neto, A. D. A., Estival, D., Li, W., Lam‐Cassettari, C., & Antoniou, M. (2021) · JMIR Mental Health · doi:10.2196/19478 · Cited by 50
“To determine whether discourse features of text message counseling conversations could predict clinical outcomes, Althoff et al [88] examined a large data set (approximately 80,000 conversations of approximately 40 messages each).”
BackgroundSarkar, S., Gaur, M., Chen, L. K., Garg, M., & Srivastava, B. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1229805 · Cited by 38
Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study
Ziru, F. U., Hsu, Y., Chan, C. S., Lau, C., Liu, J., & YIP, P. S. F. (2023) · Journal of Medical Internet Research · doi:10.2196/51069 · Cited by 50
“Althoff et al [5] used text features to understand and evaluate counselor performance and counseling outcomes.”
Chen, Z., Lu, Y., & Wang, W. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.284 · Cited by 92
Mapping the market for remanufacturing: An application of “Big Data” analytics
Neto, J. Q. F., & Dutordoir, M. (2020) · International Journal of Production Economics · doi:10.1016/j.ijpe.2020.107807 · Cited by 39
nBERT: Harnessing NLP for Emotion Recognition in Psychotherapy to Transform Mental Health Care
Rasool, A., Aslam, S., Hussain, N., Imtiaz, S., & Riaz, W. (2025) · Information · doi:10.3390/info16040301 · Cited by 46
Measuring Teaching Practices at Scale: A Novel Application of Text-as-Data Methods
Liu, J., & Cohen, J. (2021) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211009267 · Cited by 43
“…constructive online discussions (Niculae & Danescu-Niculescu-Mizil, 2016), improve the success rate of job interviews (Naim et al., 2015), change someone’s opinion by forming persuasive arguments (Tan et al., 2016), or productively address issues related to mental illness (Althoff et al., 2016).”
BackgroundGupta, M., Malik, T., & Sinha, C. (2022) · JMIR Research Protocols · doi:10.2196/36910 · Cited by 43
“All these are important elements of a therapeutic conversation [68].”
BackgroundStupinski, A. M., Alshaabi, T., Arnold, M., Adams, J. L., Minot, J. R., Price, M., Dodds, P. S., & Danforth, C. M. (2022) · JMIR Mental Health · doi:10.2196/33685 · Cited by 45
“Analysis of text-based crisis-counseling conversations found actionable strategies associated with more effective counseling [33].”
BackgroundSwain, V. D., Zhong, Q. J., Parekh, J. R., Jeon, Y., Zimmermann, R., Czerwinski, M. P., Suh, J., Mishra, V., Saha, K., & Hernandez, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713705 · Cited by 58
On the review of image and video-based depression detection using machine learning
Ashraf, A., Gunawan, T. S., Riza, B. S., Haryanto, E. V., & Janin, Z. (2020) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v19.i3.pp1677-1684 · Cited by 40
Lin, Y., Wood, S. A., & Lawler, J. J. (2022) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2022.104539 · Cited by 32
“Performance of Vader .................................................................................................. 46 Table A2.”
InfluentialShen, X., Huang, X., Zou, S., & Gan, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127550 · Cited by 32
Identifying Moments of Change from Longitudinal User Text
Tsakalidis, A., Nanni, F., Hills, A., Chim, J., Song, J., & Liakata, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.318 · Cited by 55
Priya, P., Firdaus, M., & Ekbal, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120025 · Cited by 37
Don’t Let Me Be Misunderstood:Comparing Intentions and Perceptions in Online Discussions
Chang, J. P., Cheng, J., & Danescu-Niculescu-Mizil, C. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380273 · Cited by 41
“his work has shown that intentions and perceptions relate to future conversational outcomes. Prior work has studied other signals of outcomes, such as pragmatic cues [69], similarity between comments [1], and conversation structure [22]. These have been used to forecast outcomes such as success in negotiation [11, 18]…”
Zhang, J. S., Keegan, B., Lv, Q., & Tan, C. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18112 · Cited by 43
“Similar to prior work (Tan, Friggeri, and Adamic 2016; Atkinson, Srinivasan, and Tan 2019; Althoff, Clark, and Leskovec 2016), we adopt the Jensen-Shannon Divergence (Manning and Sch¨utze 1999) to measure the monthly unigram usage difference in comments as the language distance.”
MethodGupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanan, V., Kumaraguru, P., & Sheth, A. (2022) · Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology · doi:10.18653/v1/2022.clpsych-1.12 · Cited by 27
“…are potential FQs. Datasets: Prior datasets such as Counsel Chat (CounselChat), Counseling Conversations (Huang, 2015), Role Play (Demasi et al., 2019), Crisis Text Line (Althoff et al., 2016) and Reddit C-SSRS (Gaur et al., 2019) have been created to train CA for mental health counseling.”
Exploring Self-Identified Counseling Expertise in Online Support Forums
Lahnala, A., Zhao, Y., WELCH, C. A., Kummerfeld, J. K., An, L. C., Resnicow, K., Mihalcea, R. F., & Pérez‐Rosas, V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.392 · Cited by 39
The reflection of offline activities on users’ online social behavior: An observational study
Falavarjani, S. A. M., Zarrinkalam, F., Jovanović, J., Bagheri, E., & Ghorbani, A. A. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102070 · Cited by 28
“Interesting findings have already been reported based on such study designs that include user dietary behavior (Mejova, Haddadi, Noulas & Weber, 2015), psychological states (Althoff, Clark & Leskovec, 2016), and suicide ideation (Choudhury, Kiciman, Dredze, Coppersmith & Kumar, 2016).”
BackgroundA Shoulder to Cry on: Towards A Motivational Virtual Assistant for Assuaging Mental Agony
Saha, T., Reddy, S. M., Das, A., Saha, S., & Bhattacharyya, P. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.174 · Cited by 37
“(Althoff et al., 2016) presents an investigation on a large-scale counselling dialogue gathered from an SMS text-based counselling service.”
BackgroundInfluentialSchroeder, J., Suh, J., Wilks, C. R., Czerwinski, M. P., Munson, S. A., Fogarty, J. A., & Althoff, T. (2020) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.1145/3421937.3421975 · Cited by 30
Self-citationA Machine Learning Approach for Predicting Non-Suicidal Self-Injury in Young Adults
Martí-Puig, P., Capra, C., Vega, D., Llunas, L., & Solé‐Casals, J. (2022) · Sensors · doi:10.3390/s22134790 · Cited by 32
“A similar work applied NLP to a clinical dataset of 80,885 counselling conversations to predict ultimate therapeutic outcomes [31].”
MethodSalmi, S., Mérelle, S. Y. M., Gilissen, R., van der Mei, R., & Bhulai, S. (2022) · BMC Public Health · doi:10.1186/s12889-022-12926-2 · Cited by 30
“Examples of such studies dealt with real-time topic modeling of crisis chat helplines with LDA that leveraged experts during the training to improve the model [27], and modeling of a set amount of conversation stages [28].”
BackgroundMental Health Disorder Identification From Motivational Conversations
Saha, T., Reddy, S. M., Saha, S., & Bhattacharyya, P. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3143763 · Cited by 35
“An investigative study on a large-scale counseling conversation is presented in [27].”
BackgroundTseng, E., Bellini, R. F., Lee, Y., Ramjit, A., Ristenpart, T., & Dell, N. L. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637316 · Cited by 33
“In mental health, analyses of therapist-patient exchanges on digital platforms have explored hard-to-reach topics like linguistic markers of suicidality [4, 31, 61] and psychological responses to COVID-19 [51].”
BackgroundEngagement With Crisis Text Line Among Subgroups of Users Who Reported Suicidality
Szlyk, H. S., Roth, K. B., & García‐Perdomo, V. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201900149 · Cited by 27
A Summarization of the Visual Depression Databases for Depression Detection
Ashraf, A., Gunawan, T. S., Rahman, F. D. A., Kartiwi, M., Ismail, N., & Ulfiah (2020) · International Conference on Wireless and Telematics (ICWT) · doi:10.1109/icwt50448.2020.9243625 · Cited by 25
“Tim Althoff et al. [15] indicated a gigantic extent of quantitative evaluation over planning discussions.”
BackgroundA Multi-Persona Chatbot for Hotline Counselor Training
DeMasi, O., Li, Y., & Yu, Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.324 · Cited by 37
Quantifying the Causal Effects of Conversational Tendencies
Zhang, J., Mullainathan, S., & Danescu-Niculescu-Mizil, C. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415202 · Cited by 41
“rich signals of the roles they play in their interactions [67, 68] or their effectiveness in attaining conversational outcomes [1, 5, 9].”
BackgroundInfluentialLiu, Z., Peach, R. L., Lawrance, E. L., Noble, A., Ungless, M. A., & Barahona, M. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.779091 · Cited by 25
Metadata
- Authors
- Althoff, T., Clark, K., & Leskovec, J.
- Affiliations
- Stanford University (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2016-12
- Volume
- 4
- Pages
- 463-476
- ISSN
- 2307-387X (ISSN-L)
- PMC
- PMC5361062
- OpenAlex
- W2484061479
- Cited by
- 322
- References
- 31
- Citations by year
- 2026: 33 · 2025: 38 · 2024: 25 · 2023: 39 · 2022: 44 · 2021: 37 · 2020: 40 · 2019: 28 · 2018: 18 · 2017: 18 · 2016: 1
- FWCI
- 34.02
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1010921, 1149837)
- National Institutes of Health
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative
- National Institute of Biomedical Imaging and Bioengineering (U54EB020405)
- Army Research Office
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- counseling conversations, psychotherapy, computational discourse analysis, conversational strategies, message clustering, psycholinguistics, word frequency analysis, language model comparison, mental health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.