Integrating Deep and Shallow Models for Multi-Modal Depression Analysis—Hybrid Architectures
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
At present, although great progress has been made in automatic depression assessment, most of the recent works only concern the audio and video paralinguistic information, rather than the linguistic information from the spoken content. In this work, we argue that beside developing good audio and video features, to build reliable depression detection systems, text-based content features are also of importance to analyse depression-related textual indicators. Furthermore, to improve the performance of automatic depression assessment systems, powerful models, capable of modelling the characteristics of depression embedded in the audio, visual and text descriptors, are also required. This paper proposes new text and video features and hybridizes deep and shallow models for depression estimation and classification from audio, video and text descriptors. The proposed hybrid framework consists of three main parts: 1) A Deep Convolutional Neural Network (DCNN) and Deep Neural Network (DNN) based audio-visual multi-modal depression recognition model for estimating the Patient Health Questionnaire depression scale (PHQ-8); 2) A Paragraph Vector (PV) and Support Vector Machine (SVM) based model for inferring the physical and mental conditions of the individual from the transcripts of the interview; 3) A Random Forest (RF) model for depression classification from the estimated PHQ-8 score and the inferred conditions of the individual. In the PV-SVM model, PV embedding is used to obtain fixed-length feature vectors from transcripts of the answers to the questions associated with psychoanalytic aspects of depression, which are subsequently fed into the SVM classifiers for detecting the presence/absence of the considered psychoanalytic symptoms. To our best knowledge, this approach is the first attempt to apply PV for depression analysis. Besides, we propose a new visual descriptor - Histogram of Displacement Range (HDR) to characterize the displacement and velocity of the facial landmarks in the video segment. Experiments have been carried out on the Audio Visual Emotion Challenge (AVEC2016) depression dataset, they demonstrate that: 1) The proposed hybrid framework effectively improves the accuracies of both depression estimation and depression classification, with an average F1 measure up to 0.746, which is higher than the best result (0.724) of the depression sub-challenge of AVEC2016. 2) HDR obtains better depression recognition performance than Bag-of-Words (BoW) and Motion History Histogram (MHH) features.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
The PHQ-8 as a measure of current depression in the general population
Kroenke, K., Strine, T. W., Spitzer, R. L., Williams, J. B. W., Berry, J. T., & Mokdad, A. H. (2009) · Journal of Affective Disorders · doi:10.1016/j.jad.2008.06.026 · Cited by 5,121 · cited 1×
“…(HRSD) [7], and selfreporting tools such as the Beck Depression Inventory [8] and the Patient Health Questionnaire depression scale (PHQ-8) [9] designed to assess the intensity of symptoms associated with psychoanalytic aspects of depression, e.g., sleep disorder, feelings and social…”
Method(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 1×
“For the audio features, considering the Geneva Minimalistic Acoustic Parameter Set (GeMAPS) [62] and the INTER-SPEECH Challenges [63] feature sets, we analysed the base-line features of the challenge over the years and create a union feature set.”
MethodRisk factors for suicide in individuals with depression: a systematic review
Hawton, K., Casañas i Comabella, C., Haw, C., & Saunders, K. (2013) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.01.004 · Cited by 1,469 · cited 1×
“Depression can increase healthcare costs [2], suicide [3] and disability [4] while it decreases quality of life and physical function.”
BackgroundThe Economic Burden of Depression in the United States: How Did It Change Between 1990 and 2000?
Greenberg, P. E., Kessler, R. C., Birnbaum, H. G., Leong, S. A., Lowe, S. W., Berglund, P. A., & Corey-Lisle, P. K. (2003) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v64n1211 · Cited by 1,280 · cited 1×
“Depression can increase healthcare costs [2], suicide [3] and disability [4] while it decreases quality of life and physical function.”
BackgroundOn the Validity of the Beck Depression Inventory
Richter, P., Werner, J., Heerlein, A., Kraus, A., & Sauer, H. (1998) · Psychopathology · doi:10.1159/000066239 · Cited by 1,113
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 1×
“On the contrary, a decision fusion approach, which fuses the outputs from the single modalities, can capture the property of each modality and achieve better performance [75].”
BackgroundEnd-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752 · cited 1×
“The reader is referred to the works of [56], [57], and the references therein for an overview of the main publications.”
BackgroundThe distress analysis interview corpus of human and computer interviews
Gratch, J., Artstein, R., Lucas, G., Stratou, G., Scherer, S., Nazarian, A., Wood, R., Boberg, J., DeVault, D., Marsella, S., et al. (2014) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/3o7bccg9xequ · Cited by 599 · cited 1×
“The AVEC2016 depression dataset [78] consists of 189 sessions of clinical interviews.”
MethodDetecting depression from facial actions and vocal prosody
Cohn, J. F., Kruez, T. S., Matthews, I., Yang, Y., Nguyen, M. H., Padilla, M. T., Zhou, F., & De la Torre, F. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349358 · Cited by 494
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · cited 1×
“On the other hand, DNNs also have the strong ability of capturing complex non-linear relationships between variables, and have performed extremely well in a variety of regression and classification problems, such as speech recognition [72], speech synthesis [73] and emotion recognition [74].”
BackgroundLearning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · cited 1×
“The reader is referred to the works of [56], [57], and the references therein for an overview of the main publications.”
BackgroundThe GRID-HAMD: standardization of the Hamilton Depression Rating Scale
Williams, J. B. W., Kobak, K. A., Bech, P., Engelhardt, N., Evans, K., Lipsitz, J., Olin, J., Pearson, J., & Kalali, A. (2008) · International Clinical Psychopharmacology · doi:10.1097/yic.0b013e3282f948f5 · Cited by 295 · cited 1×
“Commonly used diagnostic tools [6] include interview style assessment such as the Hamilton Rating Scale for Depression (HRSD) [7], and selfreporting tools such as the Beck Depression Inventory [8] and the Patient Health Questionnaire depression scale (PHQ-8) [9] designed to assess the intensity of…”
MethodNonverbal social withdrawal in depression: Evidence from manual and automatic analyses
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2013.12.007 · Cited by 255 · cited 2×
“Girard et al. [14] and Dhall et al. [43] investigated the relationship between nonverbal behavior and severity of depression using Facial Action Coding System (FACS) action units and head pose.”
BackgroundMethodMeng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., & Wang, Y. (2013) · Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge · doi:10.1145/2512530.2512532 · Cited by 225 · cited 2×
“Table 7 compares the performance of HDR (HDR-DCNN-DNN) with those of Bag-of-Words(BoW-DCNN-DNN) and Motion History Histograms(MHH-DCNN-DNN) [49], together with the parameters (threshold ( th ) and the number of frames ( m ) for MHH, the number of clusters ( K ) for BoW, and the number of ranges ( R…”
BackgroundMethodThe INTERSPEECH 2012 speaker trait challenge
Schuller, B., Steidl, S., Batliner, A., Nöth, E., Vinciarelli, A., Burkhardt, F., Son, R. V., Weninger, F., Eyben, F., Bocklet, T., et al. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-86 · Cited by 211
Vocal and Facial Biomarkers of Depression based on Motor Incoordination and Timing
Williamson, J. R., Quatieri, T. F., Helfer, B. S., Ciccarelli, G., & Mehta, D. D. (2014) · Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2661806.2661809 · Cited by 209
The nature of clinical depression: Symptoms, syndromes, and behavior analysis
Kanter, J. W., Busch, A. M., Weeks, C. E., & Landes, S. J. (2008) · The Behavior Analyst · doi:10.1007/bf03392158 · Cited by 189
Kerr, L. K. (2001) · Western Journal of Medicine · doi:10.1136/ewjm.175.5.349 · Cited by 85 · cited 1×
“Commonly used diagnostic tools [6] include interview style assessment such as the Hamilton Rating Scale for Depression (HRSD) [7], and selfreporting tools such as the Beck Depression Inventory [8] and the Patient Health Questionnaire depression scale (PHQ-8) [9] designed to assess the intensity of…”
MethodModel Fusion for Multimodal Depression Classification and Level Detection
Senoussaoui, M., Sarria-Paja, M., Santos, J. F., & Falk, T. H. (2014) · Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2661806.2661819 · Cited by 80
Depression Estimation Using Audiovisual Features and Fisher Vector Encoding
Jain, V., Crowley, J. L., Dey, A. K., & Lux, A. (2014) · Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2661806.2661817 · Cited by 78 · cited 1×
“Using the AVEC2014 [17] dataset for BDI-II score estimation, Jain et al. [53] focused on visual descriptors and Fisher Vector aggregation, along with GMMs for depression scores prediction, while Gupta et al. [54] adopted a support vector regression (SVR) to model the relationship between the audio,…”
Method
Cited by
Deep learning for depression recognition with audiovisual cues: A review
He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., Guo, C., Wang, H., Ding, S., Wang, Z., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.012 · Cited by 253
Fan, H., Zhang, X., Xu, Y., Fang, J., Zhang, S., Zhao, X., & Yu, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102161 · Cited by 137
Prediction of air pollutant concentrations based on the long short-term memory neural network
Wu, Z., Tian, Y., Li, M., Wang, B., Quan, Y., & Liu, J. (2024) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2023.133099 · Cited by 35
Wang, L., Wang, C., Li, C., Murai, T., Bai, Y., Song, Z., Zhang, S., Zhang, Q., Huang, Y., Bi, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01933-3 · Cited by 33
Metadata
- Authors
- Yang, L., Jiang, D., & Sahli, H.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2021-01-01
- Volume
- 12
- Issue
- 1
- Pages
- 239-253
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2891155597
- Cited by
- 118
- References
- 23
- FWCI
- 6.48
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.