The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare
Open accessAbstract
Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment.Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention.Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications.However, there are no existing pretrained language models for mental healthcare.This paper trains and releases two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community.Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736
Machine learning in mental health: a scoping review of methods and applications
Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019) · Psychological Medicine · doi:10.1017/s0033291719000151 · Cited by 1,047
Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media
Choudhury, M. D., Kıcıman, E., Dredze, M. H., Coppersmith, G., & Kumar, M. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858207 · Cited by 706
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618
Detection of Depression-Related Posts in Reddit Social Media Forum
Tadesse, M. M., Lin, H., Xu, B., & Yang, L. (2019) · IEEE Access · doi:10.1109/access.2019.2909180 · Cited by 502
Natural language processing in mental health applications using non-clinical texts
Calvo, R. A., Milne, D., Hussain, M. S., & Christensen, H. M. (2017) · Natural Language Engineering · doi:10.1017/s1351324916000383 · Cited by 405
Deep learning in mental health outcome research: a scoping review
Su, C., Xu, Z., Pathak, J., & Wang, F. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0780-3 · Cited by 368
CLPsych 2015 Shared Task: Depression and PTSD on Twitter
Coppersmith, G., Dredze, M. H., Harman, C., Hollingshead, K., & Mitchell, M. (2015) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.3115/v1/w15-1204 · Cited by 345
Suicidal Ideation Detection: A Review of Machine Learning Methods and Applications
Ji, S., Pan, S., Li, X., Cambria, E., Long, G., & Huang, Z. H. (2020) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2020.3021467 · Cited by 275
Self-citationA Test Collection for Research on Depression and Language Use
Losada, D. E., & Crestani, F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44564-9_3 · Cited by 247
Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings
Shing, H.-C., Nair, S., Zirikly, A., Friedenberg, M., Daumé, H., & Resnik, P. (2018) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w18-0603 · Cited by 233
Supervised Learning for Suicidal Ideation Detection in Online User Content
Ji, S., Yu, C. P., Fung, S.-F., Pan, S., & Long, G. (2018) · Complexity · doi:10.1155/2018/6157249 · Cited by 223
Self-citationApplication of Machine Learning Methods in Mental Health Detection: A Systematic Review
Rahman, R. A., Omar, K., Noah, S. A. M., Danuri, M. S. N. M., & Al-Garadi, M. A. (2020) · IEEE Access · doi:10.1109/access.2020.3029154 · Cited by 195
Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
Turcan, E., & McKeown, K. (2019) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/d19-6213 · Cited by 171
Windfuhr, K. L., & Kapur, N. (2011) · British Medical Bulletin · doi:10.1093/bmb/ldr042 · Cited by 157
Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction
Alt, C., Hübner, M., & Hennig, L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1134 · Cited by 139
Suicidal ideation and mental disorder detection with attentive relation networks
Ji, S., Li, X., Huang, Z. H., & Cambria, E. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06208-y · Cited by 136
Self-citationIdentifying Depression on Reddit: The Effect of Training Data
Pirina, I., & Çöltekin, Ç. (2018) · Social Media Mining for Health Research and Applications Workshop and Shared Tasks (SMM4H) · doi:10.18653/v1/w18-5903 · Cited by 121
CNN-based ranking for biomedical entity normalization
Li, H., Chen, Q., Tang, B., Wang, X., Xu, H., Wang, B., & Huang, D. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1805-7 · Cited by 119
Using Topic Modeling to Improve Prediction of Neuroticism and Depression in College Students
Resnik, P., Garron, A., & Resnik, R. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1133 · Cited by 93
Fine-Tuning BERT for Multi-Label Sentiment Analysis in Unbalanced Code-Switching Text
Tang, T., Tang, X., & Yuan, T. (2020) · IEEE Access · doi:10.1109/access.2020.3030468 · Cited by 90
Detection of Mental Health from Reddit via Deep Contextualized Representations
Jiang, Z., Levitan, S. I., Zomick, J., & Hirschberg, J. (2020) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2020.louhi-1.16 · Cited by 85
Song, Y., Wang, J., Liang, Z., Liu, Z., & Jiang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04815 · Cited by 85
Lin, C., Miller, T. A., Dligach, D., Bethard, S. J., & Savova, G. (2019) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/w19-1908 · Cited by 84
Yoo, M., Lee, S., & Ha, T. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.10.001 · Cited by 83
Rao, G., Zhang, Y., Zhang, L., Cong, Q., & Feng, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2973737 · Cited by 75
Classification of Mental Stress Using CNN-LSTM Algorithms with Electrocardiogram Signals
Kang, M., Shin, S., Jung, J., & Kim, Y. T. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/9951905 · Cited by 59
Recurrent Neural Network (RNN) to Analyse Mental Behaviour in Social Media
Bouarara, H. A. (2021) · International Journal of Software Science and Computational Intelligence · doi:10.4018/ijssci.2021070101 · Cited by 58
Mauriello, M. L., Lincoln, E. T., Hon, G., Simon, D., Jurafsky, D., & Paredes, P. E. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451799 · Cited by 43
Early Risk Detection of Pathological Gambling, Self-Harm and Depression Using BERTPreprint
Bucur, A.-M., Cosma, A., & Dinu, L. P. (2021) · arXiv (Cornell University) · doi:10.13140/rg.2.2.25060.50567 · Cited by 34
Surveying Machine Learning Algorithms On Eeg Signals Data For Mental Health Assessment
Gore, E., & Rathi, S. (2019) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon46936.2019.9105749 · Cited by 15
NLP-UNED at eRisk 2021: self-harm early risk detection with TF-IDF and linguistic features.
Ageitos, E. C., Fabregat, H., Araujo, L., & Martínez-Romo, J. (2021) · CLEF (Working Notes) · Cited by 5
Research works printed but not linked: 11
- Ansari, L., Ji, S., Chen, Q., and Cambria, E. (2022). Ensemble hybrid learning methods for automated depression detection. IEEE Transactions on Com- putational Social Systems.
- Cambria, E., Hussain, A., Durrani, T., Havasi, C., Eckl, C., and Munro, J. (2010). Sentic computing for patient centered applications. In IEEE ICSP, pages 1279-1282.
- Cambria, E., Benson, T., Eckl, C., and Hussain, A. (2012). Sentic PROMs: Application of sentic com- puting to the development of a novel unified frame- work for measuring health-care quality. Expert Sys- tems with Applications, 39(12):10533-10543.
- Cambria, E., Liu, Q., Decherchi, S., Xing, F., , and Kwok, K. (2022). SenticNet 7: A commonsense- based neurosymbolic ai framework for explainable sentiment analysis. In LREC.
- Grassi, M., Cambria, E., Hussain, A., and Piazza, F. (2011). Sentic web: A new paradigm for managing social media affective information. Cognitive Com- putation, 3(3):480-489.
- Lee, A., Kummerfeld, J. K., An, L. C., and Mihalcea, R. (2021). Micromodels for efficient, explainable, and reusable systems: A case study on mental health. arXiv preprint arXiv:2109.13770. · arXiv
- Lin, C., Miller, T., Dligach, D., Bethard, S., and Savova, G. (2019). A BERT-based universal model for both within-and cross-sentence clinical temporal relation extraction. In Proceedings of the 2nd Clin- ical Natural Language Processing Workshop, pages 65-71.
- Martínez-Castaño, R., Htait, A., Azzopardi, L., and Moshfeghi, Y. (2021). BERT-Based Transform- ers for Early Detection of Mental Health Illnesses. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 189-200. Springer.
- Mehta, Y., Majumder, N., Gelbukh, A., and Cambria, E. (2020). Recent trends in deep learning based personality detection. Artificial Intelligence Review, 53:2313-2339.
- Shickel, B., Siegel, S., Heesacker, M., Benton, S., and Rashidi, P. (2020). Automatic detection and clas- sification of cognitive distortions in mental health text. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 275-280. IEEE.
- Wolf, T., Chaumond, J., Debut, L., Sanh, V., Delangue, C., Moi, A., Cistac, P., Funtowicz, M., Davison, J., Shleifer, S., et al. (2020). Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 38-45.
Cited by
Sorted by citations
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K. M., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 420
The Opportunities and Risks of Large Language Models in Mental Health
Lawrence, H. R., Schneider, R. A., Rubin, S. B., Matarić, M. J., McDuff, D. J., & Jones Bell, M. (2024) · JMIR Mental Health · doi:10.2196/59479 · Cited by 235
Modern views of machine learning for precision psychiatry
Chen, Z. S., Kulkarni, P., Galatzer‐Levy, I. R., Bigio, B., Nasca, C., & Zhang, Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100602 · Cited by 247
Towards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155
Self-citationHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Self-citation
Metadata
- Authors
- Ji, S., Zhang, T., Ansari, L., Fu, J., Tiwari, P., & Cambria, E.
- Affiliations
- Aalto University (FI)
- University of Manchester (GB)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Nanyang Technological University (SG)
- Venue
- Proceedings of the Language Resources and Evaluation Conference
- Published
- 2022-06
- Pages
- 7184-7190
- ISSN
- 2522-2686
- OpenAlex
- W3209409148
- Cited by
- 233
- References
- 38
- Citations by year
- 2026: 68 · 2025: 30 · 2024: 62 · 2023: 55 · 2022: 18
- FWCI
- 79.56
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Language Resources and Evaluation Conference (LREC) (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- China Scholarship Council
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- pre-trained language models, mental disorder detection, suicidal ideation detection, masked language models, social media text analysis, mental health services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
