The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare
Open accessAbstract
Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment.Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention.Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications.However, there are no existing pretrained language models for mental healthcare.This paper trains and releases two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community.Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736
Machine learning in mental health: a scoping review of methods and applications
Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019) · Psychological Medicine · doi:10.1017/s0033291719000151 · Cited by 1,047
Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media
Choudhury, M. D., Kıcıman, E., Dredze, M. H., Coppersmith, G., & Kumar, M. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858207 · Cited by 706
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618
Research works printed but not linked: 11
- Ansari, L., Ji, S., Chen, Q., and Cambria, E. (2022). Ensemble hybrid learning methods for automated depression detection. IEEE Transactions on Com- putational Social Systems.
- Cambria, E., Hussain, A., Durrani, T., Havasi, C., Eckl, C., and Munro, J. (2010). Sentic computing for patient centered applications. In IEEE ICSP, pages 1279-1282.
- Cambria, E., Benson, T., Eckl, C., and Hussain, A. (2012). Sentic PROMs: Application of sentic com- puting to the development of a novel unified frame- work for measuring health-care quality. Expert Sys- tems with Applications, 39(12):10533-10543.
- Cambria, E., Liu, Q., Decherchi, S., Xing, F., , and Kwok, K. (2022). SenticNet 7: A commonsense- based neurosymbolic ai framework for explainable sentiment analysis. In LREC.
- Grassi, M., Cambria, E., Hussain, A., and Piazza, F. (2011). Sentic web: A new paradigm for managing social media affective information. Cognitive Com- putation, 3(3):480-489.
- Lee, A., Kummerfeld, J. K., An, L. C., and Mihalcea, R. (2021). Micromodels for efficient, explainable, and reusable systems: A case study on mental health. arXiv preprint arXiv:2109.13770. · arXiv
- Lin, C., Miller, T., Dligach, D., Bethard, S., and Savova, G. (2019). A BERT-based universal model for both within-and cross-sentence clinical temporal relation extraction. In Proceedings of the 2nd Clin- ical Natural Language Processing Workshop, pages 65-71.
- Martínez-Castaño, R., Htait, A., Azzopardi, L., and Moshfeghi, Y. (2021). BERT-Based Transform- ers for Early Detection of Mental Health Illnesses. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 189-200. Springer.
- Mehta, Y., Majumder, N., Gelbukh, A., and Cambria, E. (2020). Recent trends in deep learning based personality detection. Artificial Intelligence Review, 53:2313-2339.
- Shickel, B., Siegel, S., Heesacker, M., Benton, S., and Rashidi, P. (2020). Automatic detection and clas- sification of cognitive distortions in mental health text. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 275-280. IEEE.
- Wolf, T., Chaumond, J., Debut, L., Sanh, V., Delangue, C., Moi, A., Cistac, P., Funtowicz, M., Davison, J., Shleifer, S., et al. (2020). Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 38-45.
Cited by
Sorted by citations
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K. M., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 420
The Opportunities and Risks of Large Language Models in Mental Health
Lawrence, H. R., Schneider, R. A., Rubin, S. B., Matarić, M. J., McDuff, D. J., & Jones Bell, M. (2024) · JMIR Mental Health · doi:10.2196/59479 · Cited by 235
Modern views of machine learning for precision psychiatry
Chen, Z. S., Kulkarni, P., Galatzer‐Levy, I. R., Bigio, B., Nasca, C., & Zhang, Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100602 · Cited by 247
Towards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155
Self-citationHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Self-citation
Metadata
- Authors
- Ji, S., Zhang, T., Ansari, L., Fu, J., Tiwari, P., & Cambria, E.
- Affiliations
- Aalto University (FI)
- University of Manchester (GB)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Nanyang Technological University (SG)
- Venue
- Proceedings of the Language Resources and Evaluation Conference
- Published
- 2022-06
- Pages
- 7184-7190
- ISSN
- 2522-2686
- OpenAlex
- W3209409148
- Cited by
- 233
- References
- 38
- Citations by year
- 2026: 68 · 2025: 30 · 2024: 62 · 2023: 55 · 2022: 18
- FWCI
- 79.56
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Language Resources and Evaluation Conference (LREC) (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- China Scholarship Council
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- pre-trained language models, mental disorder detection, suicidal ideation detection, masked language models, social media text analysis, mental health services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
