The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Depression recognition based on dynamic facial and vocal expression features using partial least square regression
Abstract
Depression is a typical mood disorder, and the persons who are often in this state face the risk in mental and even physical problems. In recent years, there has therefore been increasing attention in machine based depression analysis. In such a low mood, both the facial expression and voice of human beings appear different from the ones in normal states. This paper presents a novel method, which comprehensively models visual and vocal modalities, and automatically predicts the scale of depression. On one hand, Motion History Histogram (MHH) extracts the dynamics from corresponding video and audio data to represent characteristics of subtle changes in facial and vocal expression of depression. On the other hand, for each modality, the Partial Least Square (PLS) regression algorithm is applied to learn the relationship between the dynamic features and depression scales using training data, and then predict the depression scale for an unseen one. Predicted values of visual and vocal clues are further combined at decision level for final decision. The proposed approach is evaluated on the AVEC2013 dataset and experimental results clearly highlight its effectiveness and better performance than baseline results provided by the AVEC2013 challenge organiser.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592
Computer Vision: A Modern Approach
Forsyth, D. A., & Ponce, J. (2002) · Cited by 3,728
The recognition of human movement using temporal templates
Bobick, A., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,837
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Survey on speech emotion recognition: Features, classification schemes, and databases
Ayadi, M. M. H. E., Kamel, M. S., & Karray, F. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,139
SIMPLS: An alternative approach to partial least squares regression
de Jong, S. (1993) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(93)85002-x · Cited by 1,905
Scale for the Assessment of Negative Symptoms
Andreasen, N. C. (1982) · PsycTESTS Dataset · doi:10.1037/t12696-000 · Cited by 1,888
Research works printed but not linked: 6
- 10.5555/645312.648916 · DOI
- W. T. Freeman , W. T. Freeman , M. Roth , and M. Roth . Orientation histograms for hand gesture recognition . In IEEE International Workshop on Automatic Face and Gesture Recognition , pages 296 -- 301 , 1994 . W. T. Freeman, W. T. Freeman, M. Roth, and M. Roth. Orientation histograms for hand gesture recognition. In IEEE International Workshop on Automatic Face and Gesture Recognition, pages 296--301, 1994.
- Health & Consumer Protection Directorate General . Improving the mental health of the population: Towards a strategy on mental health for the european union , 2005 . Health & Consumer Protection Directorate General. Improving the mental health of the population: Towards a strategy on mental health for the european union, 2005.
- Health & Consumer Protection Directorate General . Mental health in the EU , 2008 . Health & Consumer Protection Directorate General. Mental health in the EU, 2008.
- S. Salmans . Depression: questions you have - answers you need . People's Medical Society , 1995 . S. Salmans. Depression: questions you have - answers you need. People's Medical Society, 1995.
- 10.5555/520809.796139 · DOI
Cited by
Sorted by citations
Valstar, M., Schuller, B. W., Smith, K., Almaev, T. R., Eyben, F., Krajewski, J., Cowie, R., & Pantić, M. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661807 · Cited by 476
“127 for Valence and Arousal, respectively [20].”
Ma, X., Yang, H., Chen, Q., Huang, D., & Wang, Y. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988267 · Cited by 310
Self-citationVisually Interpretable Representation Learning for Depression Recognition from Facial Images
Zhou, X., Jin, K., Shang, Y., & Guo, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2828819 · Cited by 278
“[8] adopted motion history histogram (MHH) [25] based feature to encode motion cue from videos, and the partial least squares (PLS) [30] was”
BackgroundMethodInfluentialAutomated Depression Diagnosis Based on Deep Networks to Encode Facial Appearance and Dynamics
Zhu, Y. M., Shang, Y., Shao, Z., & Guo, G. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2650899 · Cited by 258
“[7] utilized Motion History Histogram (MHH) [25] to characterize motion information of each pixel in the video.”
MethodDeep learning for depression recognition with audiovisual cues: A review
He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., Guo, C., Wang, H., Ding, S., Wang, Z., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.10.012 · Cited by 273
“Regarding modality, speech and video samples [14, 94, 113, 17, 96, 97, 99, 100, 101, 104, 121, 122, 123, 149, 150, 16, 151, 152, 153], physiological signals [100, 154, 154, 155, 156, 157, 158, 159, 160], and text [100, 108] have been employed to improve the performance of depression assessment.”
MethodSelf-citationAutomated depression analysis using convolutional neural networks from speech
He, L., & Cui, C. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.05.007 · Cited by 250
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236
“There are several approaches to valence and arousal estimation, the most obvious of which is static regression, such as linear regression [25,37], partial least squares regression [22], or Support Vector Machine for Regression (SVR) [12,21,38], which is also frequently used as a baseline method [11,41-43].”
MethodDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Automatic Assessment of Depression Based on Visual Cues: A Systematic Review
Pampouchidou, A., Simos, P. G., Kostas Marias, A., Mériaudeau, F., Yang, F., Pediaditis, M., & Tsiknakis, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2724035 · Cited by 235
“In [121] decision fusion was implemented linearly through a weighted sum according to the formula”
MethodJan, A., Meng, H., Gaus, Y. F. B. A., & Zhang, F. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2017.2721552 · Cited by 219
“[11] modeled the visual and vocal cues for depression analysis.”
BackgroundMethodResultInfluentialSelf-citationMultimodal Depression Detection: Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors
Alghowinem, S., Goecke, R., Wagner, M., Epps, J., Hyett, M. P., Parker, G. B., & Breakspear, M. J. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2634527 · Cited by 204
“To the best of our knowledge, only a few studies have investigated multiple channels for this task [18], [19], [20], [21], [22], [23].”
BackgroundMethodResultDynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding
Dibeklioğlu, H., Hammal, Z., & Cohn, J. F. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2676878 · Cited by 201
“Using AVEC and a few non-publicly available resources [25], audiovisual detection of depression has been proposed [26]– [33].”
Facial Expression Recognition Using Computer Vision: A Systematic Review
Canedo, D., & Neves, A. J. R. (2019) · Applied Sciences · doi:10.3390/app9214678 · Cited by 200
Kollias, D., & Zafeiriou, S. P. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3014171 · Cited by 206
“In the beginning, data and annotations were scarce, hence research relied on extracting highly engineered handcrafted features and designing ad-hoc learning strategies [40], [41], [48], [52].”
MethodAutomatic depression recognition using CNN with attention mechanism from videos
He, L., Chan, J. C.-W., & Wang, Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.015 · Cited by 179
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Multimodal Spatiotemporal Representation for Automatic Depression Level Detection
Niu, M., Tao, J., Liu, B., Huang, J., & Lian, Z. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3031345 · Cited by 171
“In the works of [7], [15], [23], [24], [25], video dynamic features are generated by calculating the statistical histogram, but the histogram can not reflect the uneven distribution of features in temporal space [48].”
BackgroundMethodInfluentialA hybrid model for depression detection using deep learning
Vandana, V., Marriwala, N. K., & Chaudhary, D. (2022) · Measurement Sensors · doi:10.1016/j.measen.2022.100587 · Cited by 164
Video-Based Depression Level Analysis by Encoding Deep Spatiotemporal Features
Jazaery, M. A., & Guo, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2870884 · Cited by 162
“The speech characteristics can be useful for depression analysis [5], [6], some depression analysis methods are developed using the audio data [7], [8], [9], [10], [11], [12].”
MethodKhoo, L. S., McNaney, R., Lim, M. K., & McNaney, R. (2024) · Sensors · doi:10.3390/s24020348 · Cited by 194
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Hichem Sahli, B. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811641 · Cited by 149
“Among other features, Meng et al. in [17] performed Motion History Histograms (MHH) on the LLDs to extract change information of the vocal expressions.”
BackgroundSpectral Representation of Behaviour Primitives for Depression Analysis
Song, S., Jaiswal, S., Shen, L., & Valstar, M. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2970712 · Cited by 151
“For example, Meng et al. [29] extracted LBP and EOH as visual features and LLD as audio features, and applied Motion History Histogram (MHH) to extract dynamics from short video segments.”
BackgroundMethodMultimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features
Nasir, M., Jati, A., Shivakumar, P. G., Chakravarthula, S. N., & Georgiou, P. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988261 · Cited by 150
“[28] used Motion History Histogram (MHH) dynamical features [29] to recognize depression from both audio and video.”
MethodAutomated Depression Diagnosis Based on Facial Dynamic Analysis and Sparse Coding
Wen, L., Li, X., Guo, G., & Zhu, Y. M. (2015) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2015.2414392 · Cited by 141
“Meng et al. utilized Motion History Histogram (MHH) to capture temporal information of each pixel in the video [6] .”
Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech
Karam, Z. N., Provost, E. M., Singh, S. P., Montgomery, J., Archer, C., Harrington, G. J., & McInnis, M. G. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854525 · Cited by 140
“The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. major depression (MD) [5] [6] [7] [8], autism [9] [10] [11] [12], and post-traumatic stress disorder (PTSD) [13] [14] [15] [16].”
BackgroundFan, H., Zhang, X., Xu, Y., Fang, J., Zhang, S., Zhao, X., & Yu, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102161 · Cited by 137
MDN: A Deep Maximization-Differentiation Network for Spatio-Temporal Depression Detection
de Melo, W. C., Granger, É., & López, M. B. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3072579 · Cited by 137
“The following researches on AVEC2013 dataset are relied on LPQ [34], LBP [35], Pyramid of Histogram of Gra-dients (PHOG) [36], Local Phase Quantization from Three Orthogonal Planes (LPQ-TOP) [37], and Canonical Correlation Analysis (CCA) [38].”
MethodIntegrating Deep and Shallow Models for Multi-Modal Depression Analysis—Hybrid Architectures
Yang, L., Jiang, D., & Hichem Sahli, B. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2870398 · Cited by 123
“Table 7 compares the performance of HDR (HDR-DCNN-DNN) with those of Bag-of-Words(BoW-DCNN-DNN) and Motion History Histograms(MHH-DCNN-DNN) [49], together with the parameters (threshold ( th ) and the number of frames ( m ) for MHH, the number of clusters ( K ) for BoW, and the number of ranges ( R…”
BackgroundMethodMulti-modal emotion analysis from facial expressions and electroencephalogram
Huang, X., Kortelainen, J., Zhao, G., Li, X., Moilanen, A., Seppänen, T., & Pietikäinen, M. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.09.015 · Cited by 117
A Deep Multiscale Spatiotemporal Network for Assessing Depression From Facial Dynamics
de Melo, W. C., Granger, É., & Hadid, A. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3021755 · Cited by 116
“The models in [20], [24], [28], [30], [33], [56], [58] are based on hand-engineered representations.”
BackgroundAutomatic Depression Scale Prediction using Facial Expression Dynamics and Regression
Jan, A., Meng, H., Gaus, Y. F. A., Zhang, F., & Turabzadeh, S. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661812 · Cited by 114
“[14] presents a novel method, which comprehensively models visual and vocal modalities with dynamic features to automatically predicts the scale of depression.”
BackgroundMethodResultInfluentialSelf-citationHe, L., Jiang, D., & Hichem Sahli, B. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2877129 · Cited by 113
“In [48], the authors adopted Motion History Histogram (MHH) to extract dynamic features from videos.”
MethodDepression Level Prediction Using Deep Spatiotemporal Features and Multilayer Bi-LTSM
Uddin, M. A., Joolee, J. B., & Lee, Y.-K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2970418 · Cited by 109
Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text
Pampouchidou, A., Simantiraki, O., Fazlollahi, A., Pediaditis, M., Manousos, D., Roniotis, A., Giannakakis, G. A., Mériaudeau, F., Simos, P. G., Kostas Marias, A., et al. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988266 · Cited by 108
“Meng et al. [17] in their participation to AVEC 2013 [30], used MHIs for depression recognition, by extracting Local Binary Patterns (LBP) and Edge of Oriented Histograms from them.”
Methodde Melo, W. C., Granger, É., & Hadid, A. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756568 · Cited by 104
“[25] proposed a conventional method based on Motion History Histogram [26] to capture motion information of facial expressions.”
MethodDepression Detection Based on Deep Distribution Learning
de Melo, W. C., Granger, É., & Hadid, A. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803467 · Cited by 103
“The methods in [17, 18, 19, 20, 21] are based on handcrafted features.”
MethodSpatial–Temporal Attention Network for Depression Recognition from facial videos
Pan, Y., Shang, Y., Liu, T., Shao, Z., Guo, G., Ding, H., & Hu, Q. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121410 · Cited by 102
Towards Automatic Depression Detection: A BiLSTM/1D CNN-Based Model
Lin, L., Chen, X., Shen, Y., & Zhang, L. (2020) · Applied Sciences · doi:10.3390/app10238701 · Cited by 107
Gooding, P., & Kariotis, T. (2021) · JMIR Mental Health · doi:10.2196/24668 · Cited by 100
Hybrid Depression Classification and Estimation from Audio Video and Text Information
Yang, L., Hichem Sahli, B., Xia, X., Pei, E., Oveneke, M. C., & Jiang, D. (2017) · doi:10.1145/3133944.3133950 · Cited by 98
“[12], we propose a method to extract the dynamics from the AUs.”
MethodDeep Multi-Modal Network Based Automated Depression Severity Estimation
Uddin, M. A., Joolee, J. B., & Sohn, K.-A. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3179478 · Cited by 102
“Meng et al. [44] utilized the Motion History Histogram (MHH) to obtain the variations in facial and vocal expression.”
Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors
Khalid, H., Kim, M., Tariq, S., & Woo, S. S. (2021) · Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection · doi:10.1145/3476099.3484315 · Cited by 110
“Recently, there has been some recent interest in multimodal learning because of the fact that multimodal outperforms single feature-based methods [25, 42, 45, 53].”
BackgroundMuscular Movement Model-Based Automatic 3D/4D Facial Expression Recognition
Zhen, Q., Huang, D., Wang, Y., & Chen, L. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2557063 · Cited by 97
“In recent years, as a major branch of automatic face analysis, Facial Expression Recognition (FER) has attracted increasing interests in the filed of computer vision and pattern recognition due to its scientific challenges, and has shown its potential in many applications, such as psychological analysis [1], transport…”
BackgroundSelf-citationDepression Recognition Using Remote Photoplethysmography From Facial Videos
Casado, C. A., Cañellas, M. L., & López, M. B. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3238641 · Cited by 94
Meng, H., Bianchi‐Berthouze, N., Deng, Y. S., Cheng, J., & Cosmas, J. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2418092 · Cited by 94
Self-citationFacial Depression Recognition by Deep Joint Label Distribution and Metric Learning
Zhou, X., Wei, Z., Xu, M., Qu, S., & Guo, G. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3022732 · Cited by 83
“Our DJ-LDML is first compared to 13 visual-based state of the arts [4], [7], [9], [14], [15], [17], [25], [49], [50],”
BackgroundMethodInfluentialCross-cultural detection of depression from nonverbal behaviour
Alghowinem, S., Goecke, R., Cohn, J. F., Wagner, M., Parker, G. B., & Breakspear, M. J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163113 · Cited by 77
He, L., Guo, C., Tiwari, P., Pandey, H. M., & Dang, W. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22426 · Cited by 78
“[33] Meng H, Huang D,Wang H, Yang H, AI-Shuraifi M,Wang Y.”
Zhang, S., Zhang, X., Zhao, X., Fang, J., Niu, M., Zhao, Z., Yu, J., & Tian, Q. C. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3312263 · Cited by 80
“In particular, typical (LPQ)[22],MotionHistoryHis-togram(MHH)[26],LocalGaborBinaryPatternsfromThree OrthogonalPlanes(LGBP-TOP)[23],LocalBinaryPatterns FromThreeOrthogonalPlanes(LBP-TOP)[25],LocalPhase QuantizationfromThreeOrthogonalPlanes(LPQ-TOP)[24], andsoon.”
MethodEncoding Temporal Information For Automatic Depression Recognition From Facial Analysis
de Melo, W. C., Granger, É., & López, M. B. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054375 · Cited by 72
“Handengineered representations are employed in [13, 17, 18, 19, 20].”
Towards Robust Deep Neural Networks for Affect and Depression Recognition from Speech
Othmani, A., Kadoch, D., Bentounes, K. A., Rejaibi, E., Alfred, R., & Hadid, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_1 · Cited by 79
“[18] spectral + energy PLS regression AVEC2013 RMSE 11.”
BackgroundGuo, W., Yang, H., Liu, Z., Xu, Y., & Hu, B. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.609760 · Cited by 71
“Histogram of optical flow (Gupta et al., 2014), motion history histogram (MHH) (Meng et al., 2013), and space-time interest points (STIPs) (He et al., 2015) are extracted to describe the facial motion.”
MethodSelf-citationPRA-Net: Part-and-Relation Attention Network for depression recognition from facial expression
Liu, Z., Yuan, X., Li, Y., Shangguan, Z., Zhou, L., & Hu, B. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106589 · Cited by 69
Smart Contract Vulnerability Detection Based on Deep Learning and Multimodal Decision Fusion
Deng, W., Wei, H., Huang, T., Cao, C., Peng, Y., & Hu, X. (2023) · Sensors · doi:10.3390/s23167246 · Cited by 63
“Feature-based Early Emotion recognition [41–43] Decision-based Terminal Emotion recognition [44,45] Hybrid Fusion Early and Terminal Event detection [46–48]”
MethodKaya, H., Eyben, F., Salah, A. A., & Schuller, B. W. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854298 · Cited by 63
“[18] (10.”
Multimodal depression recognition with dynamic visual and audio cues
He, L., Jiang, D., & Hichem Sahli, B. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344581 · Cited by 63
“Motion History Histogram (MHH) is a descriptive representation of temporal-series, which achieved good performance in human action recognition [22] and depression analysis [23], [24].”
BackgroundAutomated audiovisual depression analysis
Girard, J. M., & Cohn, J. F. (2015) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2014.12.010 · Cited by 63
“[32] Meng H, Hunag D, Wang H, Yang H, Al-shuraifi M, Wang Y: Depression recognition based on dynamic facial and vocal expression features using partial least square regression.”
MethodHybrid Network Feature Extraction for Depression Assessment from Speech
Zhao, Z., Li, Q., Cummins, N., Liu, B., Wang, H., Tao, J., & Schuller, B. W. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2396 · Cited by 61
“Moreover, as shown in Table 2 and Table 3, although the performance of the BLSTM, DCNN, SAN and BLSTM-DCNN models is inferior to that of the proposed hybrid model, it still surpasses that of the AVEC 2013 or AVEC 2014 audio baseline and the multi-modal work proposed in [30, 31] in terms of RMSE and MAE.”
ResultSelf-citationSequential fusion of facial appearance and dynamics for depression recognition
Chen, Q., Chaturvedi, I., Ji, S., & Cambria, E. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.07.005 · Cited by 61
“Meng et al. [9] proposed to use motion history histogram (MHH) [23] feature to model motion in videos, and then use the partial least squares (PLS) [24] for training regression model.”
BackgroundMethodPan, Y., Shang, Y., Shao, Z., Liu, T., Guo, G., & Ding, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3296318 · Cited by 58
Detecting Depression with Word-Level Multimodal Fusion
Rohanian, M., Hough, J., & Purver, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2283 · Cited by 58
“However, while this work tries to predict depression based on late or early fusion methods [10, 12] or the sequential nature of their inputs [17, 18], learning the time-dependent relationships between language, visual and audio features in detecting depression is still unexplored.”
BackgroundLQGDNet: A Local Quaternion and Global Deep Network for Facial Depression Recognition
Shang, Y., Pan, Y., Jiang, X., Shao, Z., Guo, G., Liu, T., & Ding, H. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3139651 · Cited by 58
“Most hand-crafted features [5], [8], [18], [19], [20] require a gray-”
MethodThe SRI AVEC-2014 Evaluation System
Mitra, V., Shriberg, E. E., McLaren, M., Kathol, A., Richey, C., Vergyri, D., & Graciarena, M. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661818 · Cited by 56
“Several studies [17, 18, 19, 20] have used both audio and video modalities.”
Automatic nonverbal behavior indicators of depression and PTSD: the effect of gender
Stratou, G., Scherer, S. J., Gratch, J. S., & Morency, L. (2014) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-014-0161-4 · Cited by 57
“On the side of automatic assessment of depression there has been recent work [17,18] that focuses on the automatic diagnosis of depression from multimodal features, exploring dynamics of the face.”
BackgroundMultimodal mild depression recognition based on EEG-EM synchronization acquisition network
Zhu, J., Wang, Y., La, R., Zhan, J., Niu, J., Zeng, S., & Hu, X. (2019) · IEEE Access · doi:10.1109/access.2019.2901950 · Cited by 56
“[37] investigated the fusion of facial and vocal expressions, and used a weighted sum decision fusion; their result showed a slight improvement comparedwith individual channels.”
MethodSelf-citationXu, J., Güneş, H., Kusumam, K., Valstar, M., & Song, S. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3415770 · Cited by 67
“However, handcrafted methods [23], [26], [27], [28] which are engineered to summarize the frame-level/thin slice-level features into a video-level descriptor, generally fail to learn depressionspecific features that can be learned using deep learning methods.”
BackgroundEncoding Visual Behaviors with Attentive Temporal Convolution for Depression Prediction
Du, Z., Li, W., Huang, D., & Wang, Y. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756584 · Cited by 56
“Pioneering studies on this issue mostly model depressive temporal characteristics in the face area, making use of facial appearances, head poses, gazes and so forth [7] [23] [17] [1].”
BackgroundSelf-citationHe, L., Guo, C., Tiwari, P., Su, R., Pandey, H. M., & Dang, W. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22704 · Cited by 57
“Meng et al.(30) adopted LBP and Edge Orientation Histogram (EOH) as frame‐based features and Motion History Histogram (MHH) to represent dynamic and discriminative patterns from videos.”
MethodA Multimodal Approach for Detection and Assessment of Depression Using Text, Audio and Video
Zhang, W., Mao, K., & Chen, J. (2024) · Phenomics · doi:10.1007/s43657-023-00152-8 · Cited by 61
“Meng et al. (2013) proposed using MHI along with extracting Local Binary Patterns and Edge of Oriented Histograms for depression recognition in the Audio/visual emotion challenge (AVEC) 2013 Challenge.”
Multi task sequence learning for depression scale prediction from video
Chao, L., Tao, J., Yang, M., & Li, Y. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344620 · Cited by 53
“[11] presents a novel method, which comprehensively models visual and vocal modalities with dynamic features to automatically predicts the scale of depression.”
MethodNiu, M., Liu, B., Tao, J., & Li, Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.056 · Cited by 48
Reducing noisy annotations for depression estimation from facial images
He, L., Tiwari, P., Lv, C., Wu, W., & Guo, L. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.05.025 · Cited by 48
Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion RecognitionPreprint
Ortega, J. D. S., Senoussaoui, M. E. A., Granger, É., Pedersoli, M., Cardinal, P., & Koerich, A. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03196 · Cited by 57
“Meng et al. [15] propose a multi-layer system using Motion History Histogram dynamical features to recognize depression from both audio and video.”
Automated screening for distress: A perspective for the future
Rana, R. K., Latif, S., Gururajan, R., Gray, A. D., Mackenzie, G., Humphris, G. M., & Dunn, J. C. (2019) · European Journal of Cancer Care · doi:10.1111/ecc.13033 · Cited by 46
Facial expression analysis using Decomposed Multiscale Spatiotemporal Networks
de Melo, W. C., Granger, É., & López, M. B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121276 · Cited by 43
A Linguistically-Informed Fusion Approach for Multimodal Depression Detection
Morales, M. R., Scherer, S. J., & Levitan, R. (2018) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w18-0602 · Cited by 43
“Late fusion approaches have also been investigated by some (Joshi et al., 2013; Meng et al., 2013).”
BackgroundMethodBipolar Disorder Recognition via Multi-scale Discriminative Audio Temporal Representation
Du, Z., Li, W., Huang, D., & Wang, Y. (2018) · doi:10.1145/3266302.3268997 · Cited by 41
“Recent studies [21] [19] show that automatic audio and video analysis techniques can advance detection of some mental disorders, e.”
BackgroundSelf-citationKaya, H., & Salah, A. A. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654978 · Cited by 40
“From the table we observe that features using range functional give more information about the target variable than the mode features.”
MethodDepressioner: Facial dynamic representation for automatic depression level prediction
Niu, M., He, L., Li, Y., & Liu, B. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117512 · Cited by 39
Dual Attention and Element Recalibration Networks for Automatic Depression Level Prediction
Niu, M., Zhao, Z., Tao, J., Li, Y., & Schuller, B. W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3177737 · Cited by 39
Dual-Stream Multiple Instance Learning for Depression Detection With Facial Expression Videos
Shangguan, Z., Liu, Z., Li, G., Chen, Q., Ding, Z., & Hu, B. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3204757 · Cited by 43
A novel multimodal depression diagnosis approach utilizing a new hybrid fusion method
Zhang, X., Li, B., & Qi, G. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106552 · Cited by 43
MLM-EOE: Automatic Depression Detection via Sentimental Annotation and Multi-Expert Ensemble
Lin, Z., Wang, Y., Zhou, Y., Du, F., & Yang, Y. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3585599 · Cited by 43
“Earlystudies onADD oftenutilizedtraditional machinelearn-ing techniques, such as Local Binary Pattern (LBP), Edge Orientation Histogram (EOH), and Low-Level Descriptors (LLD), to represent features of individual video frames [26], [27], [28].”
Self-citationMulti-scale Temporal Modeling for Dimensional Emotion Recognition in Video
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661811 · Cited by 39
Du, Z., Wu, S., Huang, D., Li, W., & Wang, Y. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2940224 · Cited by 39
Self-citationRetracted: Diagnostic Criteria for Depression based on Both Static and Dynamic Visual Features
Pandya, D. D., Jadeja, A., Degadwala, S., & Vyas, D. (2023) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot56793.2023.10053450 · Cited by 37
RetractedDesigning a framework for assisting depression severity assessment from facial image analysis
Pampouchidou, A., Kostas Marias, A., Tsiknakis, M., Simos, P. G., Yang, F., & Mériaudeau, F. (2015) · IEEE International Conference on Signal and Image Processing Applications (ICSIPA) · doi:10.1109/icsipa.2015.7412257 · Cited by 37
“LBP was computed for 41×windowSize images for plane 1, plus the N + M images of planes 2 and 3, for the parameter combination [8,1] and [16,2] (neighbourhood and radius respectively) [33].”
MethodShan, G., Lina, Z., & Zhang, D. (2020) · Information & Management · doi:10.1016/j.im.2020.103349 · Cited by 37
WavDepressionNet: Automatic Depression Level Prediction via Raw Speech Signals
Niu, M., Tao, J., Li, Y., Qin, Y., & Li, Y. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3272553 · Cited by 36
Automatic Depression Recognition With an Ensemble of Multimodal Spatio-Temporal Routing Features
Wang, Y., Lin, Z., Yang, C., Zhou, Y., & Yang, Y. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3543226 · Cited by 36
“Techniques like Local Phase Quantization (LPQ) [21] and Motion History His-togram (MHH) [22] extracted facial and behavioral features for analysis in machine learning models such as Support Vector Regression (SVR) [21] and Partial Least Squares (PLS) [23].”
Self-citationMeshram, P., & Rambola, R. K. (2022) · Expert Systems · doi:10.1111/exsy.12933 · Cited by 35
“These neural networks have a better application in describing the visible and facial features (Meng et al., 2014; Weber et al, 2016).”
BackgroundRetractedNiu, M., Zhao, Z., Tao, J., Li, Y., & Schuller, B. W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3182658 · Cited by 35
Parameswaran, N. S., & Venkataraman, D. (2019) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v14.i1.pp503-512 · Cited by 35
“Features of video and audio information are separated from the video utilizing a Movement History Histogram (MHH) which represents to the qualities of minute changes that occur in face and vocal appearances of the depressed [17].”
MethodTDCA-Net: Time-Domain Channel Attention Network for Depression Detection
Cai, C., Niu, M., Liu, B., Tao, J., & Liu, X. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1176 · Cited by 33
“They [7, 8, 9, 10, 17, 34, 35] use hand-crafted features as input to the model.”
MethodMachot, F. A., Ali, M., Ranasinghe, S., Mosa, A. H., & Kyamakya, K. (2018) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3197768.3201523 · Cited by 33
“[24] Visual and vocal modalities, Motion History Histogram (MHH), Partial Least Square (PLS), regression algorithm Scale of depression”
MethodNiu, M., Li, Y., Tao, J., Zhou, X., & Schuller, B. W. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3382334 · Cited by 35
“In general, it is difficult to match the experimental performance of methods [4], [5], [47], [48], [49], [50] using hand-crafted features with those using neural networks [10], [12], [13], [35], [51], [52].”
MethodEffects of feature type, learning algorithm and speaking style for depression detection from speech
Mitra, V., & Shriberg, E. E. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178877 · Cited by 29
“It was demonstrated in [20] that use of both audio and video modalities improves the accuracy of a MDD detection compared to using each modality alone.”
MethodCsenet: Complex Squeeze-and-Excitation Network for Speech Depression Level Prediction
Fan, C., Lv, Z., Pei, S., & Niu, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746011 · Cited by 29
Multi-modality Hierarchical Recall based on GBDTs for Bipolar Disorder Classification
Xing, X., Cai, B., zhao, Y., Li, S., He, Z., & Fan, W. (2018) · doi:10.1145/3266302.3266311 · Cited by 28
“Inspired by Motion History Histogram (MHH) [17, 31], we extract the 17 key AUs MHH features fau_mhh ∈ R17×5×10=850 with 5 time intervals Mk ∈ {10, 20, 30, 40, 50} and 10 equally spaced bins Rb ∈ {−5,−4,−3, .”
MethodNoise and reverberation effects on depression detection from speech
Mitra, V., Tsiartas, A., & Shriberg, E. E. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472788 · Cited by 28
“Studies have also used both audio and video modalities [24, 25] for MDD detection.”
BackgroundMethod
Metadata
- Authors
- Meng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., & Wang, Y.
- Affiliations
- Brunel University of London (GB)
- Beihang University (CN)
- Venue
- Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge
- Publisher
- ACM
- Published
- 2013-10-21
- Pages
- 21-30
- OpenAlex
- W1988088279
- Semantic Scholar
- e638499e3649fb69142208c33abf63093c96b00c
- Cited by
- 225
- References
- 47
- Citations by year
- 2026: 7 · 2025: 24 · 2024: 24 · 2023: 21 · 2022: 23 · 2021: 18 · 2020: 18 · 2019: 21 · 2018: 14 · 2017: 16 · 2016: 10 · 2015: 18 · 2014: 11
- FWCI
- 17.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- depression detection, facial expressions, vocal expression, multimodal fusion, partial least squares regression, depression severity prediction, decision fusion, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
