Model Fusion for Multimodal Depression Classification and Level Detection
Abstract
Audio-visual emotion and mood disorder cues have been recently explored to develop tools to assist psychologists and psychiatrists in evaluating a patient's level of depression. In this paper, we present a number of different multimodal depression level predictors using a model fusion approach, in the context of the AVEC14 challenge. We show that an i-vector based representation for short term audio features contains useful information for depression classification and prediction. We also employed a classification step prior to regression to allow having different regression models depending on the presence or absence of depression. Our experiments show that a combination of our audio-based model and two other models based on the LGBP-TOP video features lead to an improvement of 4% over the baseline model proposed by the challenge organizers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multimodality is central in depression prediction approaches, either by early fusion or late fusion with audio representations.
“Multimodality is central in such approaches either by applying early fusion [255] or late fusion [256] with audio representations.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [255]
Approaches dedicated to depression prediction mainly use direct representations from video without detecting primitive FEs or AUs, based on local, dynamic representations of appearance.
“The approaches dedicated to depression prediction are mainly using direct representations from video without detecting primitive FEs or AUs [253], [254], [255], [256]. They are based on local, dynamic representations of appearance (LBP-TOP or variants) for modelling continuous classification problems.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [255]
For predicting levels of depression, local, dynamic representations of appearance were usually combined with acoustic representations.
“For predicting levels of depression, local, dynamic representations of appearance were usually combined with acoustic representations [253], [254], [255], [256].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [255]
Cites
Comparison of Beck Depression Inventories -IA and -II in psychiatric outpatients
Beck, A. T., Steer, R. A., Ball, R., & Ranieri, W. F. (1996) · Journal of Personality Assessment · doi:10.1207/s15327752jpa6703_13 · Cited by 5,965 · see citing works
Support Vector Regression Machines
Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., & Vapnik, V. N. (1996) · Cited by 4,654 · see citing works
Speaker Verification Using Adapted Gaussian Mixture Models
Reynolds, D. A., Quatieri, T. F., & Dunn, R. B. (2000) · Digital Signal Processing · doi:10.1006/dspr.1999.0361 · Cited by 4,282 · see citing works
Front-End Factor Analysis for Speaker Verification
Dehak, N., Kenny, P. J., Dehak, R., Dumouchel, P., & Ouellet, P. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2064307 · Cited by 3,549 · see citing works
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,509 · see citing works
(2000) · Applied Physics Letters · doi:10.1162/15324430152748236 · Cited by 1,819 · see citing works
Proceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003 · see citing works
An Introduction to Generalized Linear Models
Dobson, A. J., & Barnett, A. G. (2008) · doi:10.1201/9780367807849 · Cited by 948 · see citing works
Valstar, M., Schuller, B. W., Smith, K., Almaev, T. R., Eyben, F., Krajewski, J., Cowie, R., & Pantić, M. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661807 · Cited by 476 · see citing works
Language recognition via i-vectors and dimensionality reduction
Dehak, N., Torres‐Carrasquillo, P. A., Reynolds, D. A., et al. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-328 · Cited by 424 · see citing works
MSR Identity Toolbox v1.0: A MATLAB Toolbox for Speaker Recognition Research
Sadjadi, S. O., Slaney, M., & Heck, L. P. (2013) · Cited by 195 · see citing works
Vocal biomarkers of depression based on motor incoordination
Williamson, J. R., Quatieri, T. F., Helfer, B. S., Horwitz-Martin, R. L., Yu, B., & Mehta, D. D. (2013) · doi:10.1145/2512530.2512531 · Cited by 189 · see citing works
An investigation of depressed speech detection: features and normalization
Cummins, N., Epps, J., Breakspear, M. J., & Goecke, R. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-750 · Cited by 186 · see citing works
A Study of the Cosine Distance-Based Mean Shift for Telephone Speech Diarization
Senoussaoui, M. E. A., Kenny, P. J., Stafylakis, T., & Dumouchel, P. (2013) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2013.2285474 · Cited by 169 · see citing works
Self-citationDiagnosis of depression by behavioural signals
Cummins, N., Joshi, J., Dhall, A., Sethu, V., Goecke, R., & Epps, J. (2013) · doi:10.1145/2512530.2512535 · Cited by 138 · see citing works
Exploiting intra-conversation variability for speaker diarization
Shum, S. H., Dehak, N., Chuangsuwanich, E., et al. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-383 · Cited by 114 · see citing works
On the relative importance of vocal source, system, and prosody in human depression
Horwitz-Martin, R. L., Quatieri, T. F., Helfer, B. S., Yu, B., Williamson, J. R., & Mundt, J. C. (2013) · Wearable and Implantable Body Sensor Networks · doi:10.1109/bsn.2013.6575522 · Cited by 65 · see citing works
Automatic detection of depression in speech using Gaussian mixture modeling with factor analysis
Sturim, D. E., Torres‐Carrasquillo, P. A., Quatieri, T. F., Malyska, N., & McCree, A. V. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-746 · Cited by 62 · see citing works
Cummins, N., Epps, J., Sethu, V., et al. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853741 · Cited by 51 · see citing works
A study of acoustic features for the classification of depressed speech
Lopez‐Otero, P., Docío-Fernández, L., & Mateo, C. G. (2014) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro.2014.6859774 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Jan, A., Meng, H., Gaus, Y. F. B. A., & Zhang, F. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2017.2721552 · Cited by 219 · see citing works
Fan, H., Zhang, X., Xu, Y., Fang, J., Zhang, S., Zhao, X., & Yu, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102161 · Cited by 137 · see citing works
- Authors
- Senoussaoui, M., Sarria-Paja, M., Santos, J. F., & Falk, T. H.
- Affiliations
- Institut National de la Recherche Scientifique (CA)
- Venue
- Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge
- Publisher
- ACM
- Published
- 2014-11-07
- Pages
- 57-63
- OpenAlex
- W2055134133
- Cited by
- 80
- References
- 20
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 5 · 2023: 9 · 2022: 8 · 2021: 6 · 2020: 6 · 2019: 7 · 2018: 8 · 2017: 9 · 2016: 4 · 2015: 3
- FWCI
- 8.73
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Workshop on Audio/Visual Emotion Challenge (AVEC) (publishedVersion)
- Funders
- Departamento Administrativo de Ciencia, Tecnología e Innovación (COLCIENCIAS)
- Fonds Québécois de la Recherche sur la Nature et les Technologies
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- depression classification, multimodal fusion, mood disorders, i-vectors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
