CoChat
Sign Up
PreprintCited by 16Mental Health via WritingTopic ModelingSentiment Analysis and Opinion Mining

Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health

Open access
Ji, Shaoxiong, Zhang, Tianlin, Yang, Kailai, Ananiadou, Sophia, Cambria, Erik, Tiedemann, Jörg·2023·arXiv
DOI 10.48550/arxiv.2304.10447arXiv 2304.10447
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretrained language models have been used in various natural language processing applications. In the mental health domain, domain-specific language models are pretrained and released, which facilitates the early detection of mental health conditions. Social posts, e.g., on Reddit, are usually long documents. However, there are no domain-specific pretrained models for long-sequence modeling in the mental health domain. This paper conducts domain-specific continued pretraining to capture the long context for mental health. Specifically, we train and release MentalXLNet and MentalLongformer based on XLNet and Longformer. We evaluate the mental health classification performance and the long-range ability of these two domain-specific pretrained models. Our models are released in HuggingFace.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.10447

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. Mostafa Amin, Erik Cambria, and Björn Schuller. 2023. Will affective computing emerge from foundation models and General AI? A first evaluation on Chat- GPT. IEEE Intelligent Systems, 38(2).
  2. Luna Ansari, Shaoxiong Ji, Qian Chen, and Erik Cam- bria. 2023. Ensemble hybrid learning methods for automated depression detection. IEEE Transactions on Computational Social Systems, 10:211-219.
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. · arXiv
  4. Erik Cambria, Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022. SenticNet 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis. In LREC, pages 3829-3839.
  5. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015. Clpsych 2015 shared task: Depression and ptsd on twitter. In Proceedings of the 2nd Workshop on CLPsych, pages 31-39.
  6. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Car- bonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models be- yond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 2978-2988.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  8. Muskan Garg, Chandni Saxena, Sriparna Saha, Veena Krishnan, Ruchi Joshi, and Vijay Mago. 2022. CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Posts. In LREC, pages 6387-6396.
  9. Daisy Harvey, Fiona Lobban, Paul Rayson, Aaron Warner, Steven Jones, et al. 2022. Natural language processing methods and bipolar disorder: scoping review. JMIR Mental Health, 9(4):e35928.
  10. Kexin Huang, Abhishek Singh, Sitong Chen, Edward Moseley, Chih-Ying Deng, Naomi George, and Charolotta Lindvall. 2020. Clinical XLNet: Mod- eling Sequential Clinical Notes and Predicting Pro- longed Mechanical Ventilation. In Proceedings of the 3rd Clinical Natural Language Processing Work- shop, pages 94-100.
  11. Clayton Hutto and Eric Gilbert. 2014. Vader: A par- simonious rule-based model for sentiment analysis of social media text. In Proceedings of the inter- national AAAI conference on web and social media, volume 8, pages 216-225.
  12. Julia Ive, Natalia Viani, Joyce Kam, Lucia Yin, Somain Verma, Stephen Puntis, Rudolf N Cardinal, Angus Roberts, Robert Stewart, and Sumithra Velupillai. 2020. Generation and evaluation of artificial mental health records for natural language processing. NPJ digital medicine, 3(1):69.
  13. Shaoxiong Ji. 2022. Towards intention understanding in suicidal risk assessment with natural language processing. In Findings of EMNLP, pages 4028- 4038. Association for Computational Linguistics.
  14. Shaoxiong Ji, Shirui Pan, Xue Li, Erik Cambria, Guodong Long, and Zi Huang. 2021. Suicidal ideation detection: A review of machine learning methods and applications. IEEE Transactions on Computational Social Systems, 8:214-226.
  15. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022b. Men- talBERT: Publicly Available Pretrained Language Models for Mental Healthcare. In Proceedings of LREC, pages 7184-7190, Marseille, France. Euro- pean Language Resources Association.
  16. Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. · arXiv
  17. Madan Krishnamurthy, Khalid Mahmood, and Pawel Marcinek. 2016. A hybrid statistical and seman- tic model for identification of mental health and behavioral disorders using social network analysis. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 1019-1026. IEEE.
  18. Aziliz Le Glaz, Yannis Haralambous, Deok-Hee Kim- Dufor, Philippe Lenca, Romain Billot, Taylor C Ryan, Jonathan Marsh, Jordan Devylder, Michel Walter, Sofian Berrouiguet, et al. 2021. Machine learning and natural language processing in mental health: systematic review. Journal of Medical Inter- net Research, 23(5):e15708.
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  20. Matthew Louis Mauriello, Thierry Lincoln, Grace Hon, Dorien Simon, Dan Jurafsky, and Pablo Paredes. 2021. Sad: A stress annotated dataset for recog- nizing everyday stressors in sms-like conversational systems. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Sys- tems, pages 1-7.
  21. Saif M. Mohammad and Peter D. Turney. 2013. Crowd- sourcing a word-emotion association lexicon. Com- putational Intelligence, 29(3):436-465.
  22. Tanya Nijhawan, Girija Attigeri, and T Ananthakrishna. 2022. Stress detection using natural language pro- cessing and machine learning over social interac- tions. Journal of Big Data, 9(1):1-24.
  23. Inna Pirina and Çagrı Çöltekin. 2018. Identifying depression on reddit: The effect of training data. In Proceedings of the 2018 EMNLP Workshop SMM4H: The 3rd Social Media Mining for Health Applications Workshop & Shared Task, pages 9-12.
  24. Guanghui Qin, Yukun Feng, and Benjamin Van Durme. 2023. The NLP task effectiveness of long-range transformers. In Proceedings of EACL.
  25. Han-Chin Shing, Suraj Nair, Ayah Zirikly, Meir Friedenberg, Hal Daumé III, and Philip Resnik. 2018. Expert, crowdsourced, and machine assess- ment of suicide risk via online postings. In Proceed- ings of the Fifth Workshop on CLPsych, pages 25- 36.
  26. Marco Spruit, Stephanie Verkleij, Kees de Schepper, and Floortje Scheepers. 2022. Exploring language markers of mental health in psychiatric stories. Ap- plied Sciences, 12(4):2179.
  27. Elsbeth Turcan and Kathleen McKeown. 2019. Dread- dit: A Reddit Dataset for Stress Analysis in Social Media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97-107.
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
  29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv
  30. Thomas Wolf, Julien Chaumond, Lysandre Debut, Vic- tor Sanh, Clement Delangue, Anthony Moi, Pier- ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 38-45.
  31. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, and Sophia Ananiadou. 2023. On the evalu- ations of ChatGPT and emotion-enhanced prompt- ing for mental health analysis. arXiv preprint arXiv:2304.03347. · arXiv
  32. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. Advances in neural infor- mation processing systems, 32.
  33. Tianlin Zhang, Annika Schoene, Shaoxiong Ji, and Sophia Ananiadou. 2022. Natural language process- ing applied to mental illness detection: A narrative review. npj Digital Medicine, 5.
  34. Tianlin Zhang, Kailai Yang, Shaoxiong Ji, and Sophia Ananiadou. 2023. Emotion fusion for mental illness detection from social media: A survey. Information Fusion, 92:231-246. · DOI

Cited by

Sorted by citations

Authors
Ji, S., Zhang, T., Yang, K., Ananiadou, S., Cambria, E., & Tiedemann, J.
Affiliations
  • University of Helsinki (FI)
  • University of Manchester (GB)
  • Nanyang Technological University (SG)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.10447
arXiv
2304.10447
OpenAlex
W4366735820
Cited by
16
References
0
Citations by year
2025: 3 · 2024: 11 · 2023: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
China Scholarship Council
Topics
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
continual pre-training, long-context modeling, mental health, psychiatric classification, social media posts, Reddit, XLNet, Longformer, long-range dependencies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review