The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Depression recognition based on dynamic facial and vocal expression features using partial least square regression
Abstract
Depression is a typical mood disorder, and the persons who are often in this state face the risk in mental and even physical problems. In recent years, there has therefore been increasing attention in machine based depression analysis. In such a low mood, both the facial expression and voice of human beings appear different from the ones in normal states. This paper presents a novel method, which comprehensively models visual and vocal modalities, and automatically predicts the scale of depression. On one hand, Motion History Histogram (MHH) extracts the dynamics from corresponding video and audio data to represent characteristics of subtle changes in facial and vocal expression of depression. On the other hand, for each modality, the Partial Least Square (PLS) regression algorithm is applied to learn the relationship between the dynamic features and depression scales using training data, and then predict the depression scale for an unseen one. Predicted values of visual and vocal clues are further combined at decision level for final decision. The proposed approach is evaluated on the AVEC2013 dataset and experimental results clearly highlight its effectiveness and better performance than baseline results provided by the AVEC2013 challenge organiser.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592
Computer Vision: A Modern Approach
Forsyth, D. A., & Ponce, J. (2002) · Cited by 3,728
The recognition of human movement using temporal templates
Bobick, A., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,837
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Survey on speech emotion recognition: Features, classification schemes, and databases
Ayadi, M. M. H. E., Kamel, M. S., & Karray, F. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,139
SIMPLS: An alternative approach to partial least squares regression
de Jong, S. (1993) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(93)85002-x · Cited by 1,905
Scale for the Assessment of Negative Symptoms
Andreasen, N. C. (1982) · PsycTESTS Dataset · doi:10.1037/t12696-000 · Cited by 1,888
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1×
“using static features extracted from still face images [31, 37].”
MethodRecognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Speech emotion recognition using hidden Markov models
Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003) · Speech Communication · doi:10.1016/s0167-6393(03)00099-2 · Cited by 920
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 1×
“Therefore, a nat-ural trend appears to combine the vocal modality with the visual one, claiming that both the clues are complementary to each other and the joint use improves system performance [7] [49].”
BackgroundThe role of voice quality in communicating emotion, mood and attitude
Gobl, C. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00082-1 · Cited by 790
Information combination operators for data fusion: a comparative review with classification
Bloch, I. (1996) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.477860 · Cited by 640
Describing the emotional states that are expressed in speech
Cowie, R., & Cornelius, R. R. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00071-7 · Cited by 615
Zhang, Z., Lyons, M., Schuster, M., & Akamatsu, S. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670990 · Cited by 613
Valstar, M., Schuller, B. W., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., & Pantić, M. (2013) · doi:10.1145/2512530.2512533 · Cited by 581
Pantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580
Detecting depression from facial actions and vocal prosody
Cohn, J. F., Kruez, T. S., Matthews, I., Yang, Y., Nguyen, M. H., Padilla, M. T., Zhou, F., & De la Torre, F. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349358 · Cited by 494
Recognizing human facial expressions from long image sequences using optical flow
Yacoob, Y., & Davis, L. S. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.506414 · Cited by 463
Nonlinear feature based classification of speech under stress.
Zhou, G., Hansen, J. H. L., & Kaiser, J. F. (2001) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.905995 · Cited by 409
Face recognition using active appearance models
Edwards, G. J., Cootes, T. F., & Taylor, C. (1998) · Lecture notes in computer science · doi:10.1007/bfb0054766 · Cited by 408
The first facial expression recognition and analysis challenge
Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771374 · Cited by 385
Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships
Tong, Y., Liao, W., & Ji, Q. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1094 · Cited by 380
Some observations on oral air flow during phonation
Teager, H. M. (1980) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/tassp.1980.1163453 · Cited by 344
Fast multiple object tracking via a hierarchical particle filter
Yang, C., Ramani Duraiswami, B., & Davis, L. S. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.95 · Cited by 319
Hamm, J., Kohler, C., Gur, R., & Verma, R. (2011) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.06.023 · Cited by 296
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
El Kaliouby, R., & Robinson, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.427 · Cited by 282
AVEC 2011–The First International Audio/Visual Emotion Challenge
Schuller, B. W., Valstar, M., Eyben, F., McKeown, G. J., Cowie, R., & Pantić, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_53 · Cited by 270
IEEE International Conference on Automatic Face and Gesture Recognition
Thippur, A., Ek, C. H., & Kjellström, H. (2013) · Cited by 257
Schuller, B. W., Valstar, M., Eyben, F., Cowie, R., & Pantić, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388776 · Cited by 225
Optimal linear combination of neural networks for improving classification performance
Ueda, N. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.825759 · Cited by 205
Social risk and depression: Evidence from manual and automatic facial expression analysis
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., & Rosenwald, D. P. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553748 · Cited by 204
Automatic Recognition of Non-Acted Affective Postures
Kleinsmith, A., Bianchi‐Berthouze, N., & Steed, A. J. (2011) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2103557 · Cited by 174
Automatic behavior descriptors for psychological disorder analysis
Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A., & Morency, L.-P. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553789 · Cited by 147
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
el Kaliouby, R., & Robinson, P. (2005) · Springer eBooks · doi:10.1007/0-387-27890-7_11 · Cited by 140
A Human Action Recognition System for Embedded Computer Vision Application
Meng, H., Pears, N. E., & Bailey, C. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383420 · Cited by 114
Self-citationAutomated video-based facial expression analysis of neuropsychiatric disorders
Wang, P., Barrett, F. S., Martin, E. A., Milonova, M., Gur, R. E., Gur, R. C., Kohler, C. G., & Verma, R. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2007.09.030 · Cited by 90
Affective State Level Recognition in Naturalistic Facial and Vocal Expressions
Meng, H., & Bianchi‐Berthouze, N. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2253768 · Cited by 75
Self-citationAudio-Visual Affect Recognition through Multi-Stream Fused HMM for HCI
Zeng, Z., Tu, J., Pianfetti, B., Liu, M., Zhang, T., Zhang, Z., Huang, T. S., & Levinson, S. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.77 · Cited by 62
Descriptive temporal template features for visual motion recognition
Meng, H., & Pears, N. E. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2009.03.003 · Cited by 49
Self-citationEmotion recognition by two view SVM_2K classifier on dynamic facial expression features
Meng, H., Romera‐Paredes, B., & Bianchi‐Berthouze, N. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771362 · Cited by 40
Self-citationMotion History Histograms for Human Action Recognition
Meng, H., Pears, N. E., Freeman, M., & Bailey, C. (2008) · Advances in computer vision and pattern recognition · doi:10.1007/978-1-84800-304-0_7 · Cited by 36
Self-citationEstimation and Decision Fusion: A Survey
Sinha, A., Chen, H., Danu, D. G., Kirubarajan, T., & Farooq, M. (2006) · IEEE International Conference on Engineering of Intelligent Systems (ICEIS) · doi:10.1109/iceis.2006.1703130 · Cited by 9
Research works printed but not linked: 6
- 10.5555/645312.648916 · DOI
- W. T. Freeman , W. T. Freeman , M. Roth , and M. Roth . Orientation histograms for hand gesture recognition . In IEEE International Workshop on Automatic Face and Gesture Recognition , pages 296 -- 301 , 1994 . W. T. Freeman, W. T. Freeman, M. Roth, and M. Roth. Orientation histograms for hand gesture recognition. In IEEE International Workshop on Automatic Face and Gesture Recognition, pages 296--301, 1994.
- Health & Consumer Protection Directorate General . Improving the mental health of the population: Towards a strategy on mental health for the european union , 2005 . Health & Consumer Protection Directorate General. Improving the mental health of the population: Towards a strategy on mental health for the european union, 2005.
- Health & Consumer Protection Directorate General . Mental health in the EU , 2008 . Health & Consumer Protection Directorate General. Mental health in the EU, 2008.
- S. Salmans . Depression: questions you have - answers you need . People's Medical Society , 1995 . S. Salmans. Depression: questions you have - answers you need. People's Medical Society, 1995.
- 10.5555/520809.796139 · DOI
Cited by
Sorted by citations
Valstar, M., Schuller, B. W., Smith, K., Almaev, T. R., Eyben, F., Krajewski, J., Cowie, R., & Pantić, M. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661807 · Cited by 476
“127 for Valence and Arousal, respectively [20].”
Ma, X., Yang, H., Chen, Q., Huang, D., & Wang, Y. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988267 · Cited by 310
Self-citationVisually Interpretable Representation Learning for Depression Recognition from Facial Images
Zhou, X., Jin, K., Shang, Y., & Guo, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2828819 · Cited by 278
“[8] adopted motion history histogram (MHH) [25] based feature to encode motion cue from videos, and the partial least squares (PLS) [30] was”
BackgroundMethodInfluentialAutomated Depression Diagnosis Based on Deep Networks to Encode Facial Appearance and Dynamics
Zhu, Y. M., Shang, Y., Shao, Z., & Guo, G. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2650899 · Cited by 258
“[7] utilized Motion History Histogram (MHH) [25] to characterize motion information of each pixel in the video.”
MethodDeep learning for depression recognition with audiovisual cues: A review
He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., Guo, C., Wang, H., Ding, S., Wang, Z., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.10.012 · Cited by 273
“Regarding modality, speech and video samples [14, 94, 113, 17, 96, 97, 99, 100, 101, 104, 121, 122, 123, 149, 150, 16, 151, 152, 153], physiological signals [100, 154, 154, 155, 156, 157, 158, 159, 160], and text [100, 108] have been employed to improve the performance of depression assessment.”
MethodSelf-citation
Metadata
- Authors
- Meng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., & Wang, Y.
- Affiliations
- Brunel University of London (GB)
- Beihang University (CN)
- Venue
- Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge
- Publisher
- ACM
- Published
- 2013-10-21
- Pages
- 21-30
- OpenAlex
- W1988088279
- Semantic Scholar
- e638499e3649fb69142208c33abf63093c96b00c
- Cited by
- 225
- References
- 47
- Citations by year
- 2026: 7 · 2025: 24 · 2024: 24 · 2023: 21 · 2022: 23 · 2021: 18 · 2020: 18 · 2019: 21 · 2018: 14 · 2017: 16 · 2016: 10 · 2015: 18 · 2014: 11
- FWCI
- 17.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- depression detection, facial expressions, vocal expression, multimodal fusion, partial least squares regression, depression severity prediction, decision fusion, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
