The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.
Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling
Open accessAbstract
In this paper, we apply a context-sensitive technique for multimodal emotion recognition based on feature-level fusion of acoustic and visual cues. We use bidirectional Long Short-Term Memory (BLSTM) networks which, unlike most other emotion recognition approaches, exploit long-range contextual information for modeling the evolution of emotion within a conversation. We focus on recognizing dimensional emotional labels, which enables us to classify both prototypical and nonprototypical emotional expressions contained in a large audiovisual database. Subject-independent experiments on various classification tasks reveal that the BLSTM network approach generally prevails over standard classification techniques such as Hidden Markov Models or Support Vector Machines, and achieves F1-measures of the order of 72 %, 65 %, and 55 % for the discrimination of three clusters in emotional space and the distinction between three levels of valence and activation, respectively. Index Terms: emotion recognition, multimodality, long shortterm memory, hidden markov models, context modeling
Overview
CoChat Summary
TL;DR
The paper presents a context-sensitive multimodal emotion recognition framework that fuses acoustic and facial features and uses bidirectional LSTM networks to model long-range context for dimensional valence/activation and emotional-space clusters. In subject-independent experiments on IEMOCAP, BLSTM generally outperformed HMM and SVM, achieving F1-measures of 72.34% for three clusters, 65.18% for three valence levels, and 55.18% for three activation levels.
Key findings
- BLSTM achieved the best F1-measure for three-cluster discrimination (72.34%), four clusters (62.78%), and five clusters (54.84%); for three valence levels it reached 65.18% F1, versus 63.66% for unidirectional LSTM.
- For three activation levels, HMM+LM achieved the highest F1 (56.89%), while BLSTM reached 55.18%; the paper attributes this to class imbalance, since most utterances were labeled medium activation and training data for the remaining two activation classes was insufficient.
- Feature selection showed modality-specific patterns: for valence, on average 84% of selected features were facial, whereas for activation only 44% were facial; unimodal HMM results confirmed facial features were better for valence and acoustic features better for activation.
- Context modeling helped: HMM+LM improved over HMM for valence and activation, and BLSTM generally prevailed over HMM and SVM, except for the five-cluster task where LSTM and BLSTM showed no significant difference.
Methods
- Design: feature-level fusion of acoustic and visual cues; bidirectional Long Short-Term Memory networks for long-range contextual modeling, compared against fully-connected 3-state HMMs with a first-order language model and polynomial-kernel SVMs trained with SMO.
- Data: IEMOCAP, approximately 12 hours of audio-visual data from five mixed-gender pairs of actors (ten subjects total), with utterances annotated for valence, activation, and dominance on integer scales from one to five; experiments used cyclic leave-one-speaker-out cross validation, with mean test and training utterances of 498 ± 60 and 4475 ± 61 across folds.
- Features and analysis: 46 facial markers reduced via Principal Feature Analysis to 30 features plus first derivatives (60-dim), normalized per speaker; audio features included 12 MFCCs, 27 Mel Frequency Band coefficients, pitch, and energy plus first derivatives, extracted at 25 ms frame rate with 50 ms window; utterance-level statistical functionals were used for LSTM and SVM, with correlation-based feature subset selection yielding 66–224 features; BLSTM used 128 memory blocks per direction, learning rate 10−5, and zero-mean Gaussian input noise with standard deviation 0.6.
Limitations
- Stated: activation classification suffers from class imbalance, as the majority of utterances are labeled medium activation and the remaining two activation classes have insufficient training data.
- Stated future work: dynamic modeling of low-level features using a multimodal LSTM framework and inclusion of linguistic information were not yet examined.
- Evident from the text: the experiments use an acted IEMOCAP corpus with scripted and improvised scenarios, so real-life generalization remains untested despite the subject-independent setup.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 3 Not checked · 17 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such data are a relatively large portion of the database (approximately 17 % of the total utterances).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BLSTM v+f 67.31 52.53 58.46 55.18”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, multimodal classification outperforms unimodal AER.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all cluster prediction tasks, we observe similar trends: LSTM modeling prevails over HMM and SVM classification”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BLSTM network approach generally prevails over standard classification techniques”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
7 more claims
Create a free account to see all 20 claims and every source verdict.
- The quote only partly states this claim. 1
- Probably not a substantial claim. 5
- It may state two separate claims as one. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Experimental Results and Discussion
“For four and five clusters they achieve F1-measures of 62.78 % and 55.00 % respectively.”
§ 4. Experimental Results and Discussion
“LSTM v+f 56.19 53.89 56.25 55.00 BLSTM v+f 56.31 53.76 56.13 54.84”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Self-citationSupervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Active and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 461
Mechrabian, A. (1968) · Medical Entomology and Zoology · Cited by 459
OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418
Self-citationFeature selection using principal feature analysis
Lu, Y., Cohen, I., Zhou, X. S., & Tian, Q. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291297 · Cited by 341
Wollmer, M., Schuller, B., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 165
Self-citation
Research works printed but not linked: 1
- [14][14] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book. Entropic Cambridge Research Laboratory, Cambridge, England, 2006. 2365
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016818 · Cited by 860
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
“Deep learning is derived from the family of machine learning, which is a broader learning technique for data representation such as emotions [57], [58].”
BackgroundA Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
“…was considerable variation in model-level fusion methods, but bidirectional long short-term memories [Eyben et al. 2010; Metallinou et al. 2012; W¨ollmer et al. 2010, 2013a], various HMM-based approaches (error-weighted semicoupled HMMs [Lin et al. (e.g., Jiang et al. [2011], Paleari et al.…”
BackgroundConversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L.-P., & Zimmermann, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1193 · Cited by 463
“Our work, which performs context-sensitive recognition (Wöllmer et al., 2010) uses three modalities: audio, visual and text.”
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401
“PCA and its variants (e.g., Principal Feature Analysis, or PFA [21]) are broadly used in the emotion recognition literature [22] [23] [24].”
MethodGunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360
“Although in a stricter sense not seen as part of the visual modality, motion capture systems have also been utilized for recording the relationship between affect dimensions and facial feature information [84], affect dimensions and body posture (e.”
BackgroundMethodSelf-citationA deep learning approach for detecting traffic accidents from social media data
Zhang, Z., He, Q., Gao, J., & Ni, M. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2017.11.027 · Cited by 343
Emotion representation, analysis and synthesis in continuous space: A survey
Gunes, H., Schuller, B., Pantic, M., & Cowie, R. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771357 · Cited by 322
“, [36]), and affect dimensions and body posture (e.”
BackgroundMethodInfluentialSelf-citationLSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288
“Apart from preliminary experiments using facial marker information as additional input modality [13] and a recent study on subject dependent recognition of arousal and valence [27], LSTM architectures have hardly been applied for audiovisual emotion recognition.”
BackgroundMethodResultInfluentialSelf-citationA Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space
Xia, R., & Liu, Y. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2512598 · Cited by 183
“Then we apply the K-means clustering algorithm on the activation-valence space to generate cluster-level labels for each sentence, following other previous work [34], [35].”
MethodSpeech emotion recognition approaches in human computer interaction
Ramakrishnan, S., & El Emary, I. M. M. (2013) · Telecommunication Systems · doi:10.1007/s11235-011-9624-z · Cited by 187
“Vocal emotions are also an important constituent in multi-modal human computer interaction [17, 18].”
BackgroundSurvey on audiovisual emotion recognition: databases, features, and data fusion strategies
Wu, C.-H., Lin, J.-C., & Wei, W.-L. (2014) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2014.11 · Cited by 182
Learning Alignment for Multimodal Emotion Recognition from Speech
Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3247 · Cited by 170
“In emotion recognition, many studies extracted features from audio, visual, or textual domains and then fuse them either in the feature levels or decision levels [18, 19, 20].”
BackgroundA survey of speech emotion recognition in natural environment
Shah Fahad, M., Ranjan, A., Yadav, J., & Deepak, A. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2020.102951 · Cited by 161
Farzad, A., Mashayekhi, H., & Hassanpour, H. (2019) · Neural Computing and Applications · doi:10.1007/s00521-017-3210-6 · Cited by 158
“LSTMs are also employed for generation [8], translation [9], emotion recognition [10], acoustic modeling [11], and synthesis [12] of speech.”
MethodContext-Sensitive Learning for Enhanced Audiovisual Emotion Classification
Metallinou, A., Wöllmer, M., Katsamanis, A., Eyben, F., Schuller, B. W., & Narayanan, S. S. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.40 · Cited by 175
“Concerning the valence results, we observe that facial features are much more effective in classifiying valence than voice features, which agrees with our previous findings [8].”
MethodResultSelf-citationCOGMEN: COntextualized GNN based Multimodal Emotion recognitioN
Joshi, A., Bhat, A., Jain, A., Singh, A., & Modi, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.306 · Cited by 254
Multimodal Machine Learning: A Survey and TaxonomyPreprint
Baltrušaitis, T., Ahuja, C., & Morency, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.1705.09406 · Cited by 5,194
Optimizing semantic LSTM for spam detection
Jain, G., Шарма, М., & Agarwal, B. (2018) · International Journal of Information Technology · doi:10.1007/s41870-018-0157-5 · Cited by 182
“The bi-directional LSTM has also shown good results for emotion recognition through speech and facial expression [28].”
BackgroundHuman-Computer Interaction for Recognizing Speech Emotions Using Multilayer Perceptron Classifier
Alnuaim, A. A., Zakariah, M., Shukla, P. K., Alhadlaq, A., Hatamleh, W. A., Tarazi, H., Sureshbabu, R., & Ratna, R. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/6005446 · Cited by 177
“In addition, voice emotions play a significant role in multimodal human-computer contact [10, 11].”
BackgroundAdvances, Challenges, and Opportunities in Automatic Facial Expression Recognition
Martínez, B., & Valstar, M. (2016) · Springer eBooks · doi:10.1007/978-3-319-25958-1_4 · Cited by 176
“[149] actually target multimodal emotion recognition using LSTM-NN.”
Multi-Task Semi-Supervised Adversarial Autoencoding for Speech Emotion Recognition
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Epps, J., & Schuller, B. W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2983669 · Cited by 154
“We interpreted activation as low, medium, high, and valence as negative, neutral, positive, as used in [30], [102].”
MethodSelf-citationMultimodal Categorization of Crisis Events in Social Media
Abavisani, M., Wu, L., Hu, S., Tetreault, J., & Jaimes, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01469 · Cited by 188
“In deep multimodal learning, neural networks are used to integrate the complementary information from multiple representations (modalities) of the same phenomena [60, 43, 3, 12, 2, 44].”
BackgroundD-vlog: Multimodal Vlog Dataset for Depression Detection
Yoon, J., Kang, C., Kim, S., & Han, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21483 · Cited by 210
“With the advancements in deep learning techniques, neural networks can be used for multimodal fusion (Wöllmer et al. 2010; Nicolaou, Gunes, and Pantic 2011; Wöllmer et al. 2013).”
MethodDeep learning approaches for speech emotion recognition: state of the art and research challenges
Jahangir, R., Teh, Y. W., Hanif, F., & Mujtaba, G. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09874-7 · Cited by 142
“DL is a machine learning branch that is entirely based on artificial neural networks (ANNs) for automatic feature extraction from raw data such as speech utterances [211].”
BackgroundOutput-associative RVM regression for dimensional and continuous emotion prediction
Nicolaou, M. A., Güneş, H., & Pantić, M. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2011.12.005 · Cited by 107
“use multimodal acted data that contain face (obtained from motion capture and video) and audio information, and recognize 3–5 levels of A-V values using various classification techniques [30].”
MethodMetallinou, A., Katsamanis, A., & Narayanan, S. S. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.08.018 · Cited by 118
“LSTM networks have been applied in a variety of pattern recognition applications, including phoneme classification [48], audio-visual emotion classification [49], and regression for tracking continuous emotions [13].”
BackgroundSelf-citationVariational Autoencoders for Learning Latent Representations of Speech Emotion: A Preliminary Study
Latif, S., Rana, R. K., Qadir, J., & Epps, J. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1568 · Cited by 122
“Researchers have also used LSTM networks on the IEMOCAP speech corpus and have shown that they perform better than powerful methods like Hidden Markov Models [19, 20].”
MethodAffective learning: Improving engagement and enhancing learning with affect-aware feedback
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutiérrez-Santos, S., Wiedmann, M., & Rummel, N. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9188-z · Cited by 105
“Recent research (such as D’Mello and Graesser, 2010; Paleari et al., 2009; W¨ollmer et al., 2010; Jiang et al., 2011) focuses on a combination of input stimuli to detect affective states.”
D’Mello, S. K., & Kory-Westlund, J. M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388686 · Cited by 122
Multi-Aspect-Aware Bidirectional LSTM Networks for Synthetic Aperture Radar Target Recognition
Zhang, F., Chen, H., Yin, Q., Li, W., Li, H.-C., & Hong, W. (2017) · IEEE Access · doi:10.1109/access.2017.2773363 · Cited by 105
“Recent works have indicated that LSTM outperforms hidden Markov models (HMMs) in modeling the stochastic sequences [38], [39], which was dominated by HMMs in the early 2000s.”
BackgroundDecoding subjective emotional arousal from EEG during an immersive virtual reality experience
Hofmann, S., Klotzsche, F., Mariola, A., Nikulin, V. V., Villringer, A., & Gaebler, M. (2021) · eLife · doi:10.7554/elife.64812 · Cited by 110
Meng, H., Bianchi-Berthouze, N., Deng, Y.-D., Cheng, J., & Cosmas, J. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2418092 · Cited by 89
“[69] used bidirectional LSTM networks to exploit long-range contextual information for modeling the evolution of emotions within a conversation.”
BackgroundMethodKhosla, R., & Chu, M.-T. (2013) · ACM Transactions on Management Information Systems · doi:10.1145/2544104 · Cited by 99
Human-Computer Interaction with Detection of Speaker Emotions Using Convolution Neural Networks
Alnuaim, A. A., Zakariah, M., Alhadlaq, A., Shashidhar, C., Hatamleh, W. A., Tarazi, H., Shukla, P. K., & Ratna, R. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7463091 · Cited by 103
“Vocal emotions are crucial in multimodal human-computer contact [3,4].”
BackgroundA Review of Generalizable Transfer Learning in Automatic Emotion Recognition
Feng, K., & Chaspari, T. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00009 · Cited by 90
“Multi-modal sources of data have been widely used in automatic emotion recognition tasks in order to supplement information from multiple modalities and reduce potential bias from one single signal (Busso et al., 2004; Wöllmer et al., 2010).”
MethodA neural network based algorithm for speaker localization in a multi-room environment
Vesperini, F., Vecchiotti, P., Principi, E., Squartini, S., & Piazza, F. (2016) · Machine learning for signal processing ... · doi:10.1109/mlsp.2016.7738817 · Cited by 80
“As result, the proposed algorithm outperforms the reference one, providing an average localization error, expressed in terms of RMSE, equal to 525 mm against 1465 mm. Concluding, we also assessed the algorithm performance when a real VAD, recently proposed by some of the authors, is used.”
MethodLeveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection
Singhal, S., Pandey, T., Mrig, S., Shah, R. R., & Kumaraguru, P. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524650 · Cited by 86
“Idea has been incorporated in numerous existing multimodal techniques including early and late fusion [10], hybrid fusion [3] and fusionmethods enumerated from deep learningmethods [24, 25, 38].”
BackgroundKim, Y., & Provost, E. M. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638344 · Cited by 79
“There is a large body of work in tracking feature-level emotion structure, including HMMs and Bidirectional Long Short-Term Memory (BLSTM) systems.”
BackgroundAffective State Level Recognition in Naturalistic Facial and Vocal Expressions
Meng, H., & Bianchi‐Berthouze, N. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2253768 · Cited by 80
“[45] used bidirectional long short-term memory (BLSTM) networks to exploit long-range contextual information for modeling the evolution of emotions within a conversation.”
BackgroundText recognition using deep BLSTM networks
Ray, A., Rajeswar, S., & Chaudhury, S. (2015) · International Conference on Advances in Pattern Recognition · doi:10.1109/icapr.2015.7050699 · Cited by 77
“Long Short Term Memory based Recurrent Neural network architecture has been widely used for speech recognition [7], [8], text recognition [9], social signal prediction [10], emotion recognition [11] and time series prediction problems since it has the ability of sequence learning.”
BackgroundFusing pairwise modalities for emotion recognition in conversations
Fan, C., Lin, J., Mao, R., & Cambria, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102306 · Cited by 85
“In this context, RNNs autonomously process each modality before subsequently combination [22–24].”
MethodAcademic Performance Prediction Based on Multisource, Multifeature Behavioral Data
Zhao, L., Chen, K., Song, J., Zhu, X., Sun, J., Caulfield, B., & Namee, B. M. (2020) · IEEE Access · doi:10.1109/access.2020.3002791 · Cited by 72
“LSTM-based features have been applied in many fields, including for example emotion recognition [59], [60], traffic forecast [61] and video action classification [62].”
MethodEvaluating deep learning models for sentiment classification
Ay, B., Talo, M., Hallaç, İ. R., & Aydın, G. (2018) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.4783 · Cited by 66
Multi-head attention fusion networks for multi-modal speech emotion recognition
Zhang, J., Xing, L., Tan, Z., Wang, H., & Wang, K. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108078 · Cited by 71
Multimodal deep learning based on multiple correspondence analysis for disaster management
Pouyanfar, S., Tao, Y., Tian, H., Chen, S., & Shyu, M. (2018) · World Wide Web · doi:10.1007/s11280-018-0636-4 · Cited by 62
“Several fusion models [4, 23, 58] target the facial expressions by fusing the speech models with the visual models.”
MethodAttention-based multimodal sentiment analysis and emotion recognition using deep neural networks
Aslam, A., Sargano, A. B., & Habib, Z. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110494 · Cited by 64
Meng, H., & Bianchi‐Berthouze, N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_49 · Cited by 61
“Long Short-Term Memory (LSTM) Recurrent Neural Networks have been successfully used for modelling the relationship between observations [15] [2] [9] [16].”
MethodMulti-label, multi-task CNN approach for context-based emotion recognition
Bendjoudi, I., Vanderhaegen, F., Hamad, D., & Dornaika, F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.11.007 · Cited by 52
A multitask approach to continuous five-dimensional affect sensing in natural speech
Eyben, F., Wöllmer, M., & Schuller, B. W. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2133366.2133372 · Cited by 58
Self-citationDetecting Depression with Word-Level Multimodal Fusion
Rohanian, M., Hough, J., & Purver, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2283 · Cited by 83
“In other related tasks using multimodal fusion to predict a cognitive state, there has been work on combining temporal information from two or more modalities in a recurrent approach in audio/visual emotion classification [19] and image captioning tasks [20].”
BackgroundOutput-associative RVM regression for dimensional and continuous emotion prediction
Nicolaou, M. A., Güneş, H., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771396 · Cited by 48
Emotion recognition in spontaneous and acted dialogues
Tian, L., Moore, J. D., & Lai, C. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344645 · Cited by 49
“Another application of deep learning methods uses Denoising Autoencoders to model gender information, which is shown to help with the emotion recognition task [37].”
MethodDialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.229 · Cited by 50
“Pio-neers have explored the advantages of combining facial expressions and speech signals to predict emotions (Tzirakis et al., 2017; Wöllmer et al., 2010; Datcu and Rothkrantz, 2014; Zeng et al., 2007).”
BackgroundMulti-modal Multi-cultural Dimensional Continues Emotion Recognition in Dyadic Interactions
Zhao, J., Li, R., Chen, S., & Jin, Q. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266313 · Cited by 42
“Previous works [9, 28] have proved that the LSTM network can capture longterm temporal information and achieve significant improvement than non-temporal models.”
BackgroundProvost, E. M. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638345 · Cited by 45
“The work of Lee et al. [4] approximated humans’ ability to both identify salient regions of interest and to integrate cues over time.”
BackgroundDeep emotion recognition in textual conversations: a survey
Pereira, P., Moniz, H., & Carvalho, J. P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11010-y · Cited by 97
“While early works [53, 120] have tackled challenges from conversational context modelling, to speaker-specific modelling and emotion dynamics, resorting to, for example, gated networks, recent ERC approaches have leveraged the novel developments in Deep Learning, such as Transformers and…”
A hierarchical framework for modeling multimodality and emotional evolution in affective dialogs
Metallinou, A., Katsamanis, A., & Narayanan, S. S. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288399 · Cited by 37
“In [7] authors used speech cues from the past utterance of a speaker and his interlocutor to inform emotion recognition of the current utterance, while in our previous work[6], we modeled the evolution of emotions of a single speaker and exploited this (somewhat constrained) temporal context when classifying the curre…”
BackgroundResultInfluentialSelf-citationEmotion Classification from Speech and Text in Videos Using a Multimodal Approach
Caschera, M. C., Grifoni, P., & Ferri, F. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6040028 · Cited by 43
“In [75], the authors applied fusion at the feature-level for recognizing the dimensional emotional labels of audio and visual signals rather than categorical emotional tags (e.”
MethodSurvey of deep emotion recognition in dynamic data using facial, speech and textual cues
Zhang, T., & Tan, Z. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17944-9 · Cited by 46
“A large number of studies propose novel fusion strategies to tackle multimodal data based on concatenation [69–71, 76, 135–145].”
Predicting Latent Narrative Mood Using Audio and Physiologic Data
Alhanai, T., & Ghassemi, M. M. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10625 · Cited by 39
“…utilized in emotion detection for speech (Stuhlsatz et al. 2011; Li et al. 2013; Han, Yu, and Tashev 2014), audio/video (Kahou et al. 2013; Wöllmer et al. 2010), audio/video/text (Wöllmer et al. 2013), and physiologic data (Haag et al. 2004; Wagner, Kim, and André 2005; Walter et al.…”
MethodWöllmer, M., Metallinou, A., Katsamanis, N., Schuller, B. W., & Narayanan, S. S. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288834 · Cited by 30
Self-citationLinking emotions to behaviors through deep transfer learning
Li, H., Baucom, B., & Georgiou, P. (2020) · PeerJ Computer Science · doi:10.7717/peerj-cs.246 · Cited by 33
Hashmi, M. S. A., Ibrahim, M., Bajwa, I. S., et al. (2022) · Sensors · doi:10.3390/s22051983 · Cited by 28
“, hand-writing recognition [48], emotion recognition [49], generating music composition [50], and speech recognition [51], etc.”
BackgroundDeepConversion: Voice conversion with limited parallel training data
Zhang, M., Şişman, B., Zhao, L., & Li, H. (2020) · Speech Communication · doi:10.1016/j.specom.2020.05.004 · Cited by 27
Towards Emotion Recognition in Human Computer Interaction
Palm, G., & Glodek, M. (2012) · Smart innovation, systems and technologies · doi:10.1007/978-3-642-35467-0_32 · Cited by 32
A hierarchical static-dynamic framework for emotion classification
Mower, E., & Narayanan, S. S. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946960 · Cited by 28
“In [3] the authors utilized bi-directional long-short term modeling and found that the incorporation of within-utterance context improved the overall classification accuracy.”
MethodSelf-citationHuang, C.-W., & Narayanan, S. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02901 · Cited by 27
“s observation, previous studies have also investigated the use of architectures that explicitly model the temporal dynamics, such as hidden Markov models (HMM) [19] or recurrent neural networks (RNN) [20], [21], [22] for recognizing human emotion in speech. Furthermore, there is a growing trend in combining CNN and RN…”
BackgroundMethodInfluentialSelf-citationFu, Y., Yuan, S., Zhang, C., & Cao, J. (2023) · Electronics · doi:10.3390/electronics12224714 · Cited by 53
Multimodal Learning for Identifying Opportunities for Empathetic Responses
Tavabi, L., Stefanov, K., Gilani, S. N., Traum, D. R., & Soleymani, M. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353750 · Cited by 39
“Existing work mostly leverage online datasets that benefit from large amounts of data [9, 27, 28, 30], or use highly curated offline datasets that adopt professional actors for predefined and highly expressive scenarios [25, 30, 33].”
BackgroundFedMEKT: Distillation-based embedding knowledge transfer for multimodal federated learning
Le, H. Q., Nguyen, M. N. H., Thwal, C. M., Qiao, Y., Zhang, C., & Hong, C. S. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107017 · Cited by 34
Black, M. P., Georgiou, P., Katsamanis, A., Baucom, B., & Narayanan, S. S. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-23 · Cited by 23
“For example, improved emotion recognition has been reported when fusing audio/language/discourse features [1] and audio/video features [19].”
BackgroundSelf-citationInferring Emotions From Large-Scale Internet Voice Data
Jia, J., Zhou, S., Yin, Y., Wu, B., Chen, W., Meng, F., & Wang, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2887016 · Cited by 25
“In [60] [61] [62] [63], to model conversation emotion, the authors use bidirectional Long Short Term Memory (BLSTM) networks”
MethodSelf-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition
Yang, H., Gao, X., Wu, J., Gan, T., Ding, N., Jiang, F., & Nie, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.390 · Cited by 28
“Previous work in ERC can be roughly divided into unimodal (Yu et al., 2019; Shen et al., 2021a; Wang et al., 2020) and multimodal approaches (Datcu and Rothkrantz, 2015; Wöllmer et al., 2010).”
BackgroundLSTM Based Cross-corpus and Cross-task Acoustic Emotion Recognition
Kaya, H., Fedotov, D., Yeşilkanat, A., Verkholyak, O., Zhang, Y., & Karpov, A. A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2298 · Cited by 22
“Moreover, there is a growing research body on dimensional affect recognition with Long-Short-Term-Memory (LSTM) Recurrent Neural Networks (RNN) [7, 8, 9].”
BackgroundCombining Facial Expression and Touch for Perceiving Emotional Valence
Tsalamlal, M. Y., Amorim, M., Martin, J., & Ammi, M. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2631469 · Cited by 21
“mediated communication [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39].”
Audio Sentiment Analysis using Spectrogram and Bag-of- Visual- Words
Luitel, S., & Anwar, M. M. (2022) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri54793.2022.00052 · Cited by 22
“analysis rather than individual modalities such as text, audio, or visual [19] [20] [21].”
BackgroundEnergy wasting at internet data centers due to fear
Fernández-Montes, A., Fernández-Cerero, D., Abril, L. G., Álvarez-García, J. A., & Ortega, J. A. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2015.06.018 · Cited by 20
Verkholyak, O., Kaya, H., & Karpov, A. A. (2019) · Informatics and Automation · doi:10.15622/sp.18.1.30-56 · Cited by 20
Domain and writer adaptation of offline Arabic handwriting recognition using deep neural networks
Jemni, S. K., Ammar, S., & Kessentini, Y. (2021) · Neural computing & applications (Print) · doi:10.1007/s00521-021-06520-7 · Cited by 16
“LSTMs have been widely used for speech recognition [4], text recognition [5], emotion recognition [6] and time series prediction problems [7] thanks to their ability of sequence learning.”
MethodEmojilization: An Automated Method For Speech to Emoji-Labeled Text
Hu, J., Xu, Q., Fu, L. P., & Xu, Y. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3313071 · Cited by 19
“Martin Wöllmer et al. have used the Bidirectional Long Short-Term Memory (LSTM) network in a multi-emotion recognition technology that is context-sensitive and based on two-modal fusion [16].”
MethodVariational Autoencoders for Learning Latent Representations of Speech Emotion.Preprint
Latif, S., Rana, R. K., Qadir, J., & Epps, J. (2017) · arXiv (Cornell University) · Cited by 24
PKUSE at SemEval-2019 Task 3: Emotion Detection with Emotion-Oriented Neural Attention Network
Ma, L., Zhang, L., Ye, W., & Hu, W. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2049 · Cited by 17
“Some methods (W¨ollmer et al., 2010; Metallinou et al., 2012; Poria et al., 2017; Chernykh et al., 2017) used recurrent neural network to model the sequence of utterances for emotion detection.”
MethodDialogueTRM: Exploring the Intra- and Inter-Modal Emotional Behaviors in the ConversationPreprint
Mao, Y., Sun, Q. Y., Liu, G., Wang, X., Gao, W., Li, X., & Shen, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07637 · Cited by 47
“Pioneers have explored the advantages of combining facial expressions and speech signals to predict emotions (Tzirakis et al. 2017; Wöllmer et al. 2010; Datcu and Rothkrantz 2014; Zeng et al. 2007).”
BackgroundHwang, W., Kang, D., & Kim, D. (2021) · IET Signal Processing · doi:10.1049/sil2.12076 · Cited by 19
Contrastive Learning for Multimodal Classification of Crisis related Tweets
Mandal, B., Khanal, S., & Caragea, D. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3648143 · Cited by 20
“Multimodal deep learning has been used for a variety of tasks and modalities, including audio-visual learning for speech classification [33], multimodal emotion recognition [12, 54], image-text representation learning for image captioning [5], question answering [4], text-image matching [42], etc.”
BackgroundMultimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion
Zhao, J. M., Dong, W., Shi, L., Qiang, W., Kuang, Z., Xu, D., & An, T. (2022) · Sensors · doi:10.3390/s22155528 · Cited by 15
“[13] fused audio and video modalities for the first time and used bidirectional long and short-term memory neural networks to conduct multi-modal emotion analysis, whose experimental results showed that this multi-modality recognition effect was superior to the single-modality results.”
ResultPredicting Emotionally Salient Regions Using Qualitative Agreement of Deep Neural Network Regressors
Parthasarathy, S., & Busso, C. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2878715 · Cited by 19
“Regressors trained with LSTMs have shown superior performance in predicting emotional attributes [25], [41], [42].”
BackgroundError Reduction Network for DBLSTM-based Voice Conversion
Zhang, M., Şişman, B., Rallabandi, S. S., Li, H., & Li, Z. (2018) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipa.2018.8659543 · Cited by 15
“er, DBLSTM has been successfully used in various tasks in the field of speech and language processing, such as Automatic Speech Recognition (ASR) [28–30], speech synthesis [31] and emotion recognition [32, 33]. Although the DBLSTM and DNN based voice conversion frameworks can achieve good voice conversion performance,…”
MethodAtif, M., & Franzoni, V. (2022) · Future Internet · doi:10.3390/fi14050142 · Cited by 16
Wiki2Prop: A Multimodal Approach for Predicting Wikidata Properties from Wikipedia
Luggen, M., Audiffren, J., Difallah, D. E., & Cudré-Mauroux, P. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450082 · Cited by 14
Horstmann, A. C., Mühl, L., Köppen, L., Lindhaus, M., Storch, D., Bühren, M., Röttgers, H. R., & Krajewski, J. (2022) · Robotics · doi:10.3390/robotics11060141 · Cited by 16
“Emotion recognition predominantly focuses on facial expressions and paralinguistic features [66–68] which constitute core disciplines in machine learning and pattern recognition [69].”
Multimodal GRU with directed pairwise cross-modal attention for sentiment analysis
Qin, Z., Luo, Q., Zang, Z., & Fu, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-93023-3 · Cited by 20
Sentiment analysis from textual to multimodal features in digital environments
Caschera, M. C., Ferri, F., & Grifoni, P. (2016) · International Conference on Management of Emergent Digital EcoSystems (MEDES) · doi:10.1145/3012071.3012089 · Cited by 17
Semi-supervised Multimodal Emotion Recognition with Improved Wasserstein GANs
Liang, J., Chen, S., & Jin, Q. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023144 · Cited by 17
“In this task, the visual features, on the contrary, are more effective in classifying valence which agrees with the previous research [26].”
ResultThe Audio-Visual Arabic Dataset for Natural Emotions
Shaqra, F. A., Duwairi, R., & Al‐Ayyoub, M. (2019) · International Conference on Future Internet of Things and Cloud (FiCloud) · doi:10.1109/ficloud.2019.00054 · Cited by 22
“face, voice, gesture, EGG signal) [1, 2, 3, 4, 5, 6].”
Interaction and Transition Model for Speech Emotion Recognition in Dialogue
Zhang, R., Ando, A., Kobashikawa, S., & Aono, Y. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-713 · Cited by 13
Metadata
- Authors
- Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S.
- Affiliations
- Technical University of Munich (DE)
- University of Southern California (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 2362-2365
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2149940198
- Semantic Scholar
- 923c20f9eacd1d76d95f5528654c9578906aa09b
- Cited by
- 201
- References
- 14
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 12 · 2023: 13 · 2022: 20 · 2021: 17 · 2020: 18 · 2019: 20 · 2018: 19 · 2017: 10 · 2016: 8 · 2015: 14 · 2014: 4 · 2013: 10 · 2012: 17
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- green
- mediatum.ub.tum.de
- Licence
- other-oa
- Locations
- Interspeech (USB) (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multimodal emotion recognition, bidirectional LSTM, feature-level fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.