Cited by 288Emotion and Mood RecognitionSpeech and Audio ProcessingSentiment Analysis and Opinion Mining

The DOI matches its Crossref record; 74 of the 81 references it cites were found in a registry or an index.

LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework

Open access

Wöllmer, Martin, Kaiser, Moritz, Eyben, Florian, Schuller, Björn, Rigoll, Gerhard·2013·Image and Vision Computing

DOI 10.1016/j.imavis.2012.03.001

Open this paper in CoChat — free account

Abstract

Automatically recognizing human emotions from spontaneous and non-prototypical real-life data is currently one of the most challenging tasks in the field of affective computing. This article presents our recent advances in assessing dimensional representations of emotion, such as arousal, expectation, power, and valence, in an audiovisual human-computer interaction scenario. Building on previous studies which demonstrate that long-range context modeling tends to increase accuracies of emotion recognition, we propose a fully automatic audiovisual recognition approach based on Long Short-Term Memory (LSTM) modeling of word-level audio and video features. LSTM networks are able to incorporate knowledge about how emotions typically evolve over time so that the inferred emotion estimates are produced under consideration of an optimal amount of context. Extensive evaluations on the Audiovisual Sub-Challenge of the 2011 Audio/Visual Emotion Challenge show how acoustic, linguistic, and visual features contribute to the recognition of different affective dimensions as annotated in the SEMAINE database. We apply the same acoustic features as used in the challenge baseline system whereas visual features are computed via a novel facial movement feature extractor. Comparing our results with the recognition scores of all Audiovisual Sub-Challenge participants, we find that the proposed LSTM-based technique leads to the best average recognition performance that has been reported for this task so far.

Integrity signals

Registry

Crossref · 10.1016/j.imavis.2012.03.001

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,543
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Data Mining
Witten, I. H. (2008) · Cited by 9,143
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 6,152
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 6,122
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120

Show all 81 references

Sorted by citations

Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,152
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D’Mello, S. K., & Kory-Westlund, J. M. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 695
A survey of multimodal sentiment analysis
Soleymani, M., García, D., Jou, B., Schuller, B. W., Chang, S., & Pantić, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 647
Self-citation
A survey of sentiment analysis in social media
Yue, L., Chen, W., Li, X., Zuo, W., & Yin, M. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-018-1236-4 · Cited by 594
Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527
Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching
Zhang, S., Zhang, S. S., Huang, T., & Gao, W. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 450
Youtube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403
Self-citation
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346
Deep Learning Techniques for Speech Emotion Recognition, from Databases to Models
Abbaschian, B., Sierra-Sosa, D., & Elmaghraby, A. S. (2021) · Sensors · doi:10.3390/s21041249 · Cited by 311
Non-ferrous metals price forecasting based on variational mode decomposition and LSTM network
Liu, Y., Yang, C., Huang, K., & Gui, W. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105006 · Cited by 272
Beyond Short Snippets: Deep Networks for Video ClassificationPreprint
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.08909 · Cited by 292
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Wearable-Based Affect Recognition—A Review
Schmidt, P., Reiss, A., Dürichen, R., & Laerhoven, K. V. (2019) · Sensors · doi:10.3390/s19194079 · Cited by 291
Facial Expression Recognition: A Survey
Huang, Y., Chen, F., Lv, S., & Wang, X. (2019) · Symmetry · doi:10.3390/sym11101189 · Cited by 195
A survey of emotion recognition methods with emphasis on E-Learning environments
Imani, M., & Montazer, G. A. (2019) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102423 · Cited by 195
Deep learning the dynamic appearance and shape of facial action units
Jaiswal, S., & Valstar, M. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477625 · Cited by 170
Audio-visual emotion fusion (AVEF): A deep efficient weighted approach
Ma, Y., Hao, Y., Chen, M., Chen, J., Lu, P., & Košir, A. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.06.003 · Cited by 155
Audio to Body Dynamics
Shlizerman, E., Dery, L., Schoen, H., & Kemelmacher-Shlizerman, I. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00790 · Cited by 154
Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Hichem Sahli, B. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811641 · Cited by 149
Deep neural architectures for prediction in healthcare
Kollias, D., Tagaris, A., Stafylopatis, A., Kollias, S., & Tagaris, G. L. (2017) · Complex & Intelligent Systems · doi:10.1007/s40747-017-0064-6 · Cited by 148
A systematic literature review of speech emotion recognition approaches
Singh, Y. B., & Goel, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.028 · Cited by 156
Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 146
Long Short Term Memory Hyperparameter Optimization for a Neural Network Based Emotion Recognition Framework
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2018) · IEEE Access · doi:10.1109/access.2018.2868361 · Cited by 141
D-vlog: Multimodal Vlog Dataset for Depression Detection
Yoon, J., Kang, C., Kim, S., & Han, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21483 · Cited by 134
Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811634 · Cited by 124
Robust continuous prediction of human emotions using multiscale dynamic cues
Nicolle, J., Rapp, V., Bailly, K., Prévost, L., & Chétouani, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388783 · Cited by 113
A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition
Praveen, R. G., de Melo, W. C., Ullah, N., Aslam, M. H., Zeeshan, O., Denorme, T., Pedersoli, M., Koerich, A. L., Bacon, S., Cardinal, P., et al. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00278 · Cited by 115
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811638 · Cited by 114
Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition
Nguyen, D. H., Nguyen, K., Sridharan, S., Dean, D., & Fookes, C. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.06.005 · Cited by 108
ShEMO: a large-scale validated database for Persian speech emotion detection
Nezami, O. M., Lou, P. J., & Karami, M. (2018) · Language Resources and Evaluation · doi:10.1007/s10579-018-9427-x · Cited by 97
Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
Chen, S., & Jin, Q. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967286 · Cited by 86
Impact of training data size on the LSTM performances for rainfall–runoff modeling
Boulmaiz, T., Guermoui, M., & Boutaghane, H. (2020) · Modeling Earth Systems and Environment · doi:10.1007/s40808-020-00830-w · Cited by 87
Cross-domain facial expression recognition via an intra-category common feature and inter-category Distinction feature fusion network
Ji, Y., Hu, Y., Yang, Y., Shen, F., & Shen, H. T. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.12.037 · Cited by 86
Information Fusion in Attention Networks Using Adaptive and Multi-Level Factorized Bilinear Pooling for Audio-Visual Emotion Recognition
Zhou, H., Du, J., Zhang, Y., Wang, Q., Liu, Q.-F., & Lee, C.-H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3096037 · Cited by 85
Adaptive confidence learning for the personalization of pain intensity estimation systems
Kächele, M., Amirian, M., Thiam, P., Werner, P., Walter, S., Palm, G., & Schwenker, F. (2016) · Evolving Systems · doi:10.1007/s12530-016-9158-4 · Cited by 82
Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 87
Fine-Grained Facial Expression Recognition in the Wild
Liang, L., Lang, C., Li, Y., Feng, S., & Zhao, J. C. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3007327 · Cited by 82
Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset
Ong, D. C., Wu, Z., Zhi‐Xuan, T., Reddan, M. C., Kahhalé, I., Mattek, A. M., & Zaki, J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2955949 · Cited by 81
Recognizing Induced Emotions of Movie Audiences from Multimodal Information
Muszyński, M., Tian, L., Lai, C., Moore, J. D., Κωστούλας, Θ., Lombardo, P., Pun, T., & Chanel, G. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2902091 · Cited by 79
Wearable Assistive Technologies for Autism: Opportunities and Challenges
Mansouri-Benssassi, E., Gómez, J. C., Boyd, L. E., Hayes, G. R., & Ye, J. (2018) · IEEE Pervasive Computing · doi:10.1109/mprv.2018.022511239 · Cited by 76
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)
(2014) · doi:10.3115/v1/w14-43 · Cited by 66
Unsupervised Adversarial Domain Adaptation for Cross-Lingual Speech Emotion Recognition
Latif, S., Qadir, J., & Bilal, M. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925513 · Cited by 64
The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition
Li, D., Wang, Z., Wang, C., Liu, S., Chi, W., Dong, E., Song, X., Gao, Q., & Song, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2949707 · Cited by 66
Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention
Praveen, R. G., Cardinal, P., & Granger, É. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2022.3233083 · Cited by 62
Learning Facial Action Units from Web Images with Scalable Weakly Supervised Clustering
Zhao, K., Chu, W.-S., & Martı́nez, A. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00223 · Cited by 62
Deep Auto-Encoders With Sequential Learning for Multimodal Dimensional Emotion Recognition
Nguyen, D., Nguyen, T. T., Zeng, R., Nguyen, T. T., Tran, S. N., Nguyen, T., Sridharan, S., & Fookes, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3063612 · Cited by 67
To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations
Ahuja, C., Ma, S., Morency, L., & Sheikh, Y. A. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353725 · Cited by 66
Mmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis
Paraskevopoulos, G., Georgiou, E., & Potamianos, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746418 · Cited by 62
Automatic Recognition of Facial Displays of Unfelt Emotions
Kulkarni, K., Corneanu, C. A., Ofodile, I., Escalera, S., Baró, X., Hyniewska, S., Allïk, J., & Anbarjafari, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2874996 · Cited by 67
Emotion recognition in never-seen languages using a novel ensemble method with emotion profiles
Albornoz, E. M., & Milone, D. H. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2503757 · Cited by 57
Improving multilingual speech emotion recognition by combining acoustic features in a three-layer model
Li, X., & Akagi, M. (2019) · Speech Communication · doi:10.1016/j.specom.2019.04.004 · Cited by 56
Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition
Praveen, R. G., Granger, É., & Cardinal, P. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667055 · Cited by 55
Strength modelling for real-worldautomatic continuous affect recognition from audiovisual signals
Han, J., Zhang, Z., Cummins, N., Ringeval, F., & Schuller, B. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2016.11.020 · Cited by 53
Self-citation
Facial emotion recognition
Ma, X., Weisi, L., Dongyan, H., Dong, M., & Li, H. (2017) · IEEE International Conference on Signal and Image Processing (ICSIP) · doi:10.1109/siprocess.2017.8124509 · Cited by 56
Continuous sign language recognition from wearable IMUs using deep capsule networks and game theory
Suri, K., & Gupta, R. (2019) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2019.08.006 · Cited by 54
Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54
Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities
Savran, A., Cao, H., Nenkova, A., & Verma, R. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2362101 · Cited by 53
Robust Audiovisual Emotion Recognition: Aligning Modalities, Capturing Temporal Information, and Handling Missing Features
Goncalves, L., & Busso, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3216993 · Cited by 55
Multi task sequence learning for depression scale prediction from video
Chao, L., Tao, J., Yang, M., & Li, Y. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344620 · Cited by 53
Expression Recognition Method Based on a Lightweight Convolutional Neural Network
Zhao, G., Yang, H., & Yu, M. (2020) · IEEE Access · doi:10.1109/access.2020.2964752 · Cited by 53
Exploring the Contextual Factors Affecting Multimodal Emotion Recognition in Videos
Bhattacharya, P., Gupta, R., & Yang, Y. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3071503 · Cited by 49
Untitled work
(2019) · doi:10.1109/smc.2019.8914655 · Cited by 52
Boosting crop classification by hierarchically fusing satellite, rotational, and contextual data
Barrière, V., Claverie, M., Schneider, M., Lemoine, G., & d’Andrimont, R. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114110 · Cited by 61
A Lightweight Residual Networks Framework for DDoS Attack Classification Based on Federated Learning
Qin, T., Guang, C., Chen, W., & Si, W. (2021) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps51825.2021.9484622 · Cited by 50
Two-Stage Fuzzy Fusion Based-Convolution Neural Network for Dynamic Emotion Recognition
Wu, M., Su, W., Chen, L., Witold Pedrycz, C., & Hirota, K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2966440 · Cited by 50
Multi-level correlation mining framework with self-supervised label generation for multimodal sentiment analysis
Li, Z., Guo, Q., Pan, Y., Ding, W., Yu, J., Zhang, Y., Liu, W., Chen, H., Wang, H., & Xie, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101891 · Cited by 49
An Intelligent Long Short-Term Memory-Based Machine Learning Model for the Potential Assessment of Global Hydropower Capacity in Sustainable Energy Transition and Security
Raza, M. A., Karim, A., Alqarni, M., Al‐Khasawneh, M. A., Jumani, T. A., Aman, M. M., & Masud, M. I. (2025) · Energies · doi:10.3390/en18133324 · Cited by 50
Survey on psychotherapy chatbots
Xu, B., & Zhuang, Z. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6170 · Cited by 51
Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
Praveen, R. G., & Alam, M. J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00483 · Cited by 54
Multimodal Content Analysis for Effective Advertisements on YouTube
Vedula, N., Sun, W., Lee, H., Gupta, H., Ogihara, M., Johnson, J., Ren, G., & Parthasarathy, S. (2017) · Proceedings · doi:10.1109/icdm.2017.149 · Cited by 49
Investigating Word Affect Features and Fusion of Probabilistic Predictions Incorporating Uncertainty in AVEC 2017
Dang, T., Stasak, B., Huang, Z., Jayawardena, S., Atcheson, M., Hayat, M., Le, P. N., Sethu, V., Goecke, R., & Epps, J. (2017) · doi:10.1145/3133944.3133952 · Cited by 48
Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition
Zhang, H., Gou, R., Shang, J., Shen, F., Wu, Y., & Dai, G. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.643202 · Cited by 45
Inferring Depression and Affect from Application Dependent Meta Knowledge
Kächele, M., Schels, M., & Schwenker, F. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661813 · Cited by 47
Early prediction for physical human robot collaboration in the operating room
Zhou, T., & Wachs, J. P. (2017) · Autonomous Robots · doi:10.1007/s10514-017-9670-9 · Cited by 45
Multi-resolution modulation-filtered cochleagram feature for LSTM-based dimensional emotion recognition from speech
Peng, Z., Dang, J., Unoki, M., & Akagi, M. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.027 · Cited by 42
Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266304 · Cited by 40
Reconstruction-error-based learning for continuous emotion recognition in speech
Han, J., Zhang, Z., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952580 · Cited by 42
Self-citation
Higher-order Multivariable Polynomial Regression to Estimate Human Affective States
Wei, J., Chen, T., Liu, G., & Yang, J. (2016) · Scientific Reports · doi:10.1038/srep23384 · Cited by 41
Forecasting Renewable Energy Consumption under Zero Assumptions
Ma, J., Oppong, A., Acheampong, K. N., & Abruquah, L. A. (2018) · Sustainability · doi:10.3390/su10030576 · Cited by 40
Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661811 · Cited by 39
Transformer-Based Spiking Neural Networks for Multimodal Audiovisual Classification
Guo, L., Gao, Z., Qu, J., Zheng, S., Jiang, R., Lu, Y., & Qiao, H. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3327081 · Cited by 44
Understanding Emotional Responses to Mobile Video Advertisements via Physiological Signal Sensing and Facial Expression Analysis
Pham, P., & Wang, J. (2017) · International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025186 · Cited by 31
Joint Multi-Scale Multimodal Transformer for Emotion Using Consumer Devices
Khan, M., Ahmad, J., Gueaieb, W., Masi, G., Karray, F. O., & Saddik, A. E. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3532322 · Cited by 37
Speech Emotion Recognition Using Deep Learning Techniques
Ganapathy, A. (2016) · ABC Journal of Advanced Research · doi:10.18034/abcjar.v5i2.550 · Cited by 34
End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks
Huang, J., Li, Y., Tao, J., Lian, Z., & Yi, J. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461963 · Cited by 32
A Systematic Review on Emotion Recognition System Using Physiological Signals: Data Acquisition and Methodology
Tawsif, K., Aziz, N. A. A., Raja, J. E., Hossen, M. J., & Jesmeen, M. Z. H. (2022) · Emerging Science Journal · doi:10.28991/esj-2022-06-05-017 · Cited by 32
Measuring two aspects of emotion recognition ability: Accuracy vs. sensitivity
Lyusin, D., & Ovsyannikova, V. (2015) · Learning and Individual Differences · doi:10.1016/j.lindif.2015.04.010 · Cited by 34
Hybrid Multi-Attention Network for Audio–Visual Emotion Recognition Through Multimodal Feature Fusion
Moorthy, S., & Moon, Y.-K. (2025) · Mathematics · doi:10.3390/math13071100 · Cited by 31
Exploring Multimodal Visual Features for Continuous Affect Recognition
Sun, B., Cao, S., Li, L., He, J. Y. D., & Yu, L. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988270 · Cited by 31
Adaptation and contextualization of deep neural network models
Kollias, D., Yu, M., Tagaris, A., Leontidis, G., Stafylopatis, A., & Kollias, S. (2017) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2017.8280975 · Cited by 31
Multi-Factor RFG-LSTM Algorithm for Stock Sequence Predicting
Su, Z., Xie, H., & Han, L. (2020) · Computational Economics · doi:10.1007/s10614-020-10008-2 · Cited by 32
Wearable affect and stress recognition: A reviewPreprint
Schmidt, P., Reiss, A., Duerichen, R., & Laerhoven, K. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08854 · Cited by 31
Predicting Latent Narrative Mood Using Audio and Physiologic Data
Alhanai, T., & Ghassemi, M. M. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10625 · Cited by 33
A Survey of Computational Approaches and Challenges in Multimodal Sentiment Analysis
Huddar, M. G., Sannakki, S. S., & Rajpurohit, V. S. (2019) · International Journal of Computer Sciences and Engineering · doi:10.26438/ijcse/v7i1.876883 · Cited by 31
Hybrid Mutimodal Fusion for Dimensional Emotion Recognition
Ma, Z., Ma, F., Sun, B., & Shutao Li, A. (2021) · doi:10.1145/3475957.3484457 · Cited by 29
Factored Latent-Dynamic Conditional Random Fields for single and multi-label sequence modeling
Neogi, S., & Dauwels, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108236 · Cited by 32

Authors: Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G.
Affiliations: Technical University of Munich (DE)
Corresponding author: Wöllmer, Martin
ORCID: Kaiser, Moritz
Eyben, Florian
Schuller, Björn
Rigoll, Gerhard
Venue: Image and Vision Computing
Publisher: Elsevier BV
Published: 2013-02
Volume: 31
Issue: 2
Pages: 153-163
ISSN: 0262-8856 (ISSN-L)
DOI: 10.1016/j.imavis.2012.03.001
OpenAlex: W2026243162
Semantic Scholar: 752c6738a1ef1c18943636bc3c98a4dbd6455b33
Cited by: 288
References: 81
Citations by year: 2026: 4 · 2025: 15 · 2024: 16 · 2023: 13 · 2022: 17 · 2021: 39 · 2020: 22 · 2019: 38 · 2018: 32 · 2017: 31 · 2016: 21 · 2015: 19 · 2014: 11 · 2013: 8
FWCI: 17.14
Type: journal article
Language: en
Open access: green
mediatum.ub.tum.de
Licence: other-oa
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations: Image and Vision Computing (publishedVersion)
mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
OPUS (Augsburg University) (submittedVersion, cc-by-nc-nd)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords: audio-visual emotion recognition, long short-term memory, arousal, valence, expectations, power, long-range context modeling, video features, human-computer interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks