Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Depression is one of the most common mood disorders. Technology has the potential to assist in screening and treating people with depression by robustly modeling and tracking the complex behavioral cues associated with the disorder (e.g., speech, language, facial expressions, head movement, body language). Similarly, robust affect recognition is another challenge which stands to benefit from modeling such cues. The Audio/Visual Emotion Challenge (AVEC) aims toward understanding the two phenomena and modeling their correlation with observable cues across several modalities. In this paper, we use multimodal signal processing methodologies to address the two problems using data from human-computer interactions. We develop separate systems for predicting depression levels and affective dimensions, experimenting with several methods for combining the multimodal information. The proposed depression prediction system uses a feature selection approach based on audio, visual, and linguistic cues to predict depression scores for each session. Similarly, we use multiple systems trained on audio and visual cues to predict the affective dimensions in continuous-time. Our affect recognition system accounts for context during the frame-wise inference and performs a linear fusion of outcomes from the audio-visual systems. For both problems, our proposed systems outperform the video-feature based baseline systems. As part of this work, we analyze the role played by each modality in predicting the target variable and provide analytical insights.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Shi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · cited 1×
“To capture motion information, optical-flow-based motion vectors are computed between a pair of consecutive frames at keypoints detected using a corner detection algorithm [37]; please see Fig.”
MethodComparison of Beck Depression Inventories -IA and -II in psychiatric outpatients
Beck, A. T., Steer, R. A., Ball, R., & Ranieri, W. F. (1996) · Journal of Personality Assessment · doi:10.1207/s15327752jpa6703_13 · Cited by 5,965
Ferrari, A. J., Charlson, F. J., Norman, R. E., Patten, S. B., Freedman, G., Murray, C. J. L., Vos, T., & Whiteford, H. A. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001547 · Cited by 3,115
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 1×
“Baseline features: The audio baseline features are adopted from the AVEC 2013 challenge, extracted using the openSMILE toolkit [11].”
MethodDeformable Model Fitting by Regularized Landmark Mean-Shift
Saragih, J. M., Lucey, S., & Cohn, J. F. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0380-4 · Cited by 787
Acoustical properties of speech as indicators of depression and suicidal risk
France, D. J., Shiavi, R. G., Silverman, S., Silverman, M., & Wilkes, M. (2000) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.846676 · Cited by 504
Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing
Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D'Errico, F., & Schroeder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.27 · Cited by 381
Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2013.12.007 · Cited by 255 · cited 1×
“, an increase in withdrawing gestures and fewer smiles [44, 15]).”
BackgroundAutomatic behavior descriptors for psychological disorder analysis
Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A., & Morency, L.-P. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553789 · Cited by 147 · cited 1×
“We fit 66 landmark points to the face for each frame, using mean-shift based deformable model fitting [44] as shown in Fig.”
Method
Cited by
A review of depression and suicide risk assessment using speech analysis
Cummins, N., Scherer, S., Krajewski, J., Schnieder, S., Epps, J., & Quatieri, T. F. (2015) · Speech Communication · doi:10.1016/j.specom.2015.03.004 · Cited by 1,003
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346
“During the last two decades, audio-visual emotion recognition integrating speech and facial expression, has attracted extensive attention owing to its promising potential applications in human-computer-interaction [1], [2].”
BackgroundResearching Mental Health Disorders in the Era of Social Media: Systematic Review
Wongkoblap, A., Vadillo, M. A., & Curcin, V. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7215 · Cited by 268
Deep learning for depression recognition with audiovisual cues: A review
He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., Guo, C., Wang, H., Ding, S., Wang, Z., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.012 · Cited by 253
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236
“averaging, as in [29], to more complex methods such as linear and multi-linear regression [15,25,26,28,30,37,38], SVR [39], random forests [32] or Kalman filters based [34–36].”
Method
Metadata
- Authors
- Gupta, R., Malandrakis, N., Xiao, B., Guha, T., Van Segbroeck, M., Black, M., Potamianos, A., & Narayanan, S.
- Venue
- Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge
- Publisher
- ACM
- Published
- 2014-11-07
- Pages
- 33-40
- OpenAlex
- W2095540482
- Cited by
- 112
- References
- 10
- FWCI
- 17.50
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions. https://app.cochat.ai/papers/10.1145/2661806.2661810
- The paper
- Gupta, R., Malandrakis, N., Xiao, B., Guha, T., Van Segbroeck, M., Black, M., Potamianos, A., & Narayanan, S. (2014). Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions. Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge. https://doi.org/10.1145/2661806.2661810
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.