Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Open accessIntegrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep learning is used for learning joint feature representations for multimodal sensor data in human affect recognition.
“Learning joint feature representations for multimodal data. In multimodal approaches, DNNs are leveraged to learn joint feature representations from multiple unimodal feature representations to accomplish feature-level fusion [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
Ringeval et al. [65] found that while the given physiological signal has lower predictive power than audiovisual signals, both are complementary.
“They found that while the given physiological signal has lower predictive power than audiovisual signals, both are complementary.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
One study suggested that a smaller window size (2s) could be the best choice.
“One study suggested that a smaller window size (2s) could be the best choice [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
Depending on the nature of the source audio, it can be beneficial to apply HSFs to frame-level LLDs according to a sliding window before applying the RNN.
“Depending on the nature of the source audio, it can be beneficial to apply HSFs to frame-level LLDs according to a sliding window before applying the RNN [151], [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
Ringeval et al. concluded that in feature-level fusion, ECG data helps for prediction of valence, but not arousal.
“They concluded that in featurelevel fusion, ECG data helps for prediction of valence, but not arousal.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
The literature generally reports that decision-level fusion works better for affect recognition given the datasets and models currently used.
“To date, the literature generally reports that decision-level fusion works better for affect recognition given the datasets and models currently used [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
Several other studies based on handcrafted features found that decision-level fusion on top of individual LSTM models leads to better performance.
“However, several other studies based on handcrafted features found that decision-level fusion on top of individual LSTM models leads to better performance [65], [129].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · see citing works
A Real-Time QRS Detection Algorithm
Pan, J., & Tompkins, W. J. (1985) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.1985.325532 · Cited by 7,855 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · see citing works
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Two-Frame Motion Estimation Based on Polynomial Expansion
Farnebäck, G. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45103-x_50 · Cited by 3,010 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
Self-citationDawson, M., Schell, A., & Filion, D. L. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511546396.007 · Cited by 2,425 · see citing works
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · see citing works
Intransitivity of preferences.
Tversky, A. (1969) · Psychological Review · doi:10.1037/h0026750 · Cited by 2,250 · see citing works
Cacioppo, J. T., Tassinary, L. G., Berntson, G. G., Shan, H., Mason, P., Kolb, B., Whishaw, I. Q., Geuter, S., Lindquist, M. A., Wager, T. D., et al. (2016) · Cambridge University Press eBooks · doi:10.1017/9781107415782 · Cited by 2,032 · see citing works
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & la Torre, F. D. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.75 · Cited by 1,917 · see citing works
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312 · see citing works
Recent developments in openSMILE, the munich open-source multimedia feature extractor
Eyben, F., Weninger, F. J., Gross, F., & Schuller, B. W. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502224 · Cited by 1,265 · see citing works
Self-citationEmotional speech recognition: Resources, features, and methods
Ververidis, D., & Kotropoulos, C. L. (2006) · Speech Communication · doi:10.1016/j.specom.2006.04.003 · Cited by 927 · see citing works
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter, S., Bengio, Y., Informatik, F. F., Bengio, Y., Frasconi, P., & Schmidhuber, J. (2001) · Cited by 863 · see citing works
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · see citing works
Self-citationSchuller, B. W., Steidl, S., Batliner, A. M., Vinciarelli, A., Scherer, K. R., Ringeval, F., Chétouani, M., Weninger, F. J., Eyben, F., Marchi, E., et al. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-56 · Cited by 732 · see citing works
Self-citationDynamic markers of altered gait rhythm in amyotrophic lateral sclerosis
Hausdorff, J. M., Lertratanakul, A., Cudkowicz, M., Peterson, A. L., Kaliton, D., & Goldberger, A. L. (2000) · Journal of Applied Physiology · doi:10.1152/jappl.2000.88.6.2045 · Cited by 502 · see citing works
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 466 · see citing works
Nicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · see citing works
How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions
Yan, W., Wu, Q., Liang, J., Chen, Y.-H., & Fu, X. (2013) · Journal of Nonverbal Behavior · doi:10.1007/s10919-013-0159-8 · Cited by 446 · see citing works
The Vera am Mittag German audio-visual emotional speech database
Grimm, M., Kroschel, K., & Narayanan, S. S. (2008) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2008.4607572 · Cited by 413 · see citing works
Gunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · see citing works
Self-citationA Field Guide to Dynamical Recurrent Networks
Kolen, J. F., & Kremer, S. C. (2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037 · Cited by 338 · see citing works
Douglas‐Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O. M., McRorie, M., Martin, J., Devillers, L. Y., Abrilian, S., Batliner, A. M., et al. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_43 · Cited by 324 · see citing works
openSMILE - The Munich Versatile and Fast Open-Source Audio Feature Extractor
Eyben, F. ;. (2013) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 322 · see citing works
Self-citationThe Effects of a Normative Intervention on Group Decision-Making Performance
Hall, J., & Watson, W. (1970) · Human Relations · doi:10.1177/001872677002300404 · Cited by 313 · see citing works
On the Acoustics of Emotion in Audio: What Speech, Music, and Sound have in Common
Weninger, F. J., Eyben, F., Schuller, B. W., Mortillaro, M., & Scherer, K. R. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00292 · Cited by 277 · see citing works
Self-citationSocial psychophysiology and emotion : theory and clinical applications
Wagner, H. L. (1988) · Cited by 241 · see citing works
Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity.
LEVENSON, R. W. (1988) · Cited by 217 · see citing works
Building autonomous sensitive artificial listeners
Schroder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D., ter Maat, M., McKeown, G., Pammi, S., Pantic, M., et al. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.34 · Cited by 185 · see citing works
Self-citationBilchick, K. C., & Berger, R. D. (2006) · Journal of Cardiovascular Electrophysiology · doi:10.1111/j.1540-8167.2006.00501.x · Cited by 183 · see citing works
Robust full‐motion recovery of head by dynamic templates and re‐registration techniques
Xiao, J., Moriyama, T., Takeo Kanade, A., & Cohn, J. F. (2003) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.10048 · Cited by 180 · see citing works
Real-time Speech and Music Classification by Large Audio Feature Space Extraction
Eyben, F. (2015) · Springer theses · doi:10.1007/978-3-319-27299-3 · Cited by 161 · see citing works
Self-citationBimodal Emotion Recognition using Speech and Physiological Changes
Kim, J. (2007) · doi:10.5772/4754 · Cited by 145 · see citing works
Eyben, F., Wöllmer, M., Graves, A., Schuller, B. W., Douglas‐Cowie, E., & Cowie, R. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0032-6 · Cited by 137 · see citing works
Self-citationSupport Vector Regression for Automatic Recognition of Spontaneous Emotions in Speech
Grimm, M., Kroschel, K., & Narayanan, S. S. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367262 · Cited by 114 · see citing works
Robust continuous prediction of human emotions using multiscale dynamic cues
Nicolle, J., Rapp, V., Bailly, K., Prévost, L., & Chétouani, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388783 · Cited by 113 · see citing works
The INTERSPEECH 2014 computational paralinguistics challenge: cognitive & physical load
Schuller, B. W., Steidl, S., Batliner, A. M., Epps, J., Eyben, F., Ringeval, F., Marchi, E., & Zhang, Y. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-104 · Cited by 108 · see citing works
Self-citationCorrecting Time-Continuous Emotional Labels by Modeling the Reaction Lag of Evaluators
Mariooryad, S., & Busso, C. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2334294 · Cited by 107 · see citing works
Robust Speech Recognition and Understanding
Grimm, M., & Kroschel, K. (2007) · doi:10.5772/35 · Cited by 98 · see citing works
Frame vs. Turn-Level: Emotion Recognition from Speech Considering Static and Dynamic Processing
Vlasenko, B., Schuller, B. W., Wendemuth, A., & Rigoll, G. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_13 · Cited by 92 · see citing works
Self-citationAudio-Visual Classification and Fusion of Spontaneous Affective Data in Likelihood Space
Nicolaou, M. A., Güneş, H., & Pantić, M. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.900 · Cited by 74 · see citing works
Audio-visual emotion recognition using an emotion space concept
Kanluan, I., Grimm, M., & Kroschel, K. (2008) · Cited by 66 · see citing works
Analysis and Compensation of the Reaction Lag of Evaluators in Continuous Emotional Annotations
Mariooryad, S., & Busso, C. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.21 · Cited by 58 · see citing works
A multitask approach to continuous five-dimensional affect sensing in natural speech
Eyben, F., Wöllmer, M., & Schuller, B. W. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2133366.2133372 · Cited by 53 · see citing works
Self-citationTracking changes in continuous emotion states using body language and prosodic cues
Metallinou, A., Katsamanis, A., Wang, Y., & Narayanan, S. S. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946939 · Cited by 42 · see citing works
The INTERSPEECH 2014 Computational paralinguistics challenge: cognitive & physical load
Schuller, B., Steidl, S., Batliner, A., Epps, J., Eyben, F., Ringeval, F., Marchi, E., & Zhang, Y. (2019) · OPUS (Augsburg University) · Cited by 36 · see citing works
Self-citationAutomatic Segmentation of Spontaneous Data using Dimensional Labels from Multiple Coders
Nicolaou, M. A., Güneş, H., & Pantić, M. (2010) · University of Twente Research Information · Cited by 32 · see citing works
Extracting complexity waveforms from one-dimensional signals
Kalauzi, A. J., Bojić, T., & Rakić, L. (2009) · Nonlinear Biomedical Physics · doi:10.1186/1753-4631-3-8 · Cited by 27 · see citing works
Time-Scale Feature Extractions for Emotional Speech Characterization
Chétouani, M., Mahdhaoui, A., & Ringeval, F. (2009) · Cognitive Computation · doi:10.1007/s12559-009-9016-9 · Cited by 26 · see citing works
Self-citation2D Face Alignment and Pose Estimation Based on 3D Facial Models
Chen, S.-C., Wu, C., Lin, S.-Y., & Hung, Y. (2012) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2012.60 · Cited by 10 · see citing works
The Intrinsic Recurrent Support Vector Machine
Schneegaß, D., Schäfer, A. M., & Martinetz, T. (2007) · Cited by 5 · see citing works
Flagged first, then by citations
5 of the 63 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A review on the long short-term memory model
Houdt, G. V., Mosquera, C., & Nápoles, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09838-1 · Cited by 2,066 · see citing works
“Manyotherapplicationsexistfortimeseriespredictionscenarios, like emotion ratings (Ringeval et al. 2015), topic evolution in text streams (Lu et al. 2019), building energy usage (Wang et al. 2019b), carbon emission levels (Huang et al. 2019b), flight delays (McCarthy et al. 2019), financial trading…”
BackgroundAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904 · see citing works
“As a second baseline algorithm, we utilised a BLSTM-DRNNs with the architecture preserved from [5, 21], i.”
MethodInfluentialSelf-citationEnd-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752 · see citing works
Self-citationValstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988258 · Cited by 516 · see citing works
“A normalisation technique based on the Evaluator Weighted Estimator [30], is used prior to the computation of the gold-standard, i. e., the average of all ratings for each subject [36].”
BackgroundMethodInfluentialSelf-citationFacial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 354 · see citing works
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236 · see citing works
“averaging, as in [29], to more complex methods such as linear and multi-linear regression [15,25,26,28,30,37,38], SVR [39], random forests [32] or Kalman filters based [34–36].”
MethodInfluentialDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Leveraging recent advances in deep learning for audio-Visual emotion recognition
Schoneveld, L., Othmani, A., & Abdelkawy, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.007 · Cited by 243 · see citing works
“The performances of the proposed approach have been evaluated using the REmote COLlaborative and affective interactions (RECOLA) corpus (Ringeval et al. [2013]).”
MethodAVEC 2018 Workshop and Challenge
Ringeval, F., Schuller, B. W., Valstar, M., Cowie, R., Kaya, H., Schmitt, M., Amiriparian, S., Cummins, N., Lalanne, D., Michaud, A., et al. (2018) · doi:10.1145/3266302.3266316 · Cited by 149 · see citing works
“Even though individual ratings of emotion can be modelled all together in a multi-task learning framework [58], the dominant practice is to summarise the annotations for each recording into a single time-series, referred as gold-standard in the literature3, that can be then easily processed by any machine learning alg…”
BackgroundSelf-citationMultimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition
Chen, S., Jin, Q., Zhao, J., & Wang, S. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133949 · Cited by 147 · see citing works
“Ringeval et al. [16] explore two types of multi-task learning for dimension emotion recognition.”
Background(2015) · doi:10.1145/2808196.2811642 · Cited by 178 · see citing works
Self-citationSchmitt, M., Ringeval, F., & Schuller, B. W. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1124 · Cited by 135 · see citing works
“ported by the annotators are not contemporaneous, due to the reaction time of the raters, this delay must be taken into account when using a non-context aware machine learning algorithm [25].”
MethodSelf-citationFacial Emotion Expressions in Human–Robot Interaction: A Survey
Rawal, N., & Stock‐Homburg, R. (2022) · International Journal of Social Robotics · doi:10.1007/s12369-022-00867-0 · Cited by 130 · see citing works
“There are very few studies that combine facial data with both audio and physiological data [107, 106] and studies that analyze all modalities (face, voice, text, body gestures and physiological signals) have not been found.”
BackgroundLong Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811634 · Cited by 124 · see citing works
“Except the above methods, [10] adds window to the label and features.”
BackgroundMethodResultInfluentialSemi-Supervised Speech Emotion Recognition With Ladder Networks
Parthasarathy, S., & Busso, C. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3023632 · Cited by 119 · see citing works
“Previous studies have shown the benefits of training with CCC as the objective function over the MSE [12], [39], [50].”
BackgroundResultA Review of Recurrent Neural Network-Based Methods in Computational Physiology
Mao, S., & Sejdić, E. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3145365 · Cited by 128 · see citing works
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811638 · Cited by 112 · see citing works
“In work [20], two types of multi-task learning are introduced: one by learning each rater’s individual track and the other by learning both dimensions simultaneously.”
BackgroundMethodInfluentialHow deep neural networks can improve emotion recognition on video data
Khorrami, P., Le Paine, T., Brady, K., Dagli, C., & Huang, T. S. (2016) · 2016 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2016.7532431 · Cited by 107 · see citing works
“While many works [12, 13, 14, 15] have trained recurrent neural networks (RNNs) on hand-crafted features, few works [16, 17] have considered the effects of using a CNN as input to the RNN.”
BackgroundAVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and ChallengePreprint
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01600 · Cited by 121 · see citing works
“A normalisation technique based on the Evaluator Weighted Estimator [30], is used prior to the computation of the gold-standard, i. e., the average of all ratings for each subject [36].”
BackgroundMethodInfluentialSelf-citationAutomatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101 · see citing works
“Some studies have applied a ConvNet with a different number of layers to physiological signals to classify different emotions [28], [30]–[32].”
BackgroundCurriculum Learning for Speech Emotion Recognition From Crowdsourced Labels
Lotfian, R., & Busso, C. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2898816 · Cited by 97 · see citing works
“This metric has been previously used in related studies [55], including the audio/visual emotion challenge (AVEC) in 2015 [56].”
MethodMulti-Modal Recurrent Attention Networks for Facial Expression Recognition
Lee, J., Kim, S., Kim, S., & Sohn, K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2996086 · Cited by 96 · see citing works
“To alleviate this limitation, dimensional emotion descriptions [8], [12]–[15] have attracted much attention, where emotions are described in a continuous space, consisting of”
BackgroundMuzammel, M., Salam, H., & Othmani, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106433 · Cited by 88 · see citing works
Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction
Brady, K., Gwon, Y., Khorrami, P., Godoy, E., Campbell, W., Dagli, C., & Huang, T. S. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988264 · Cited by 110 · see citing works
Han, J., Zhang, Z., Schmitt, M., Pantić, M., & Schuller, B. W. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123383 · Cited by 85 · see citing works
“To estimate the performance of the proposed framework, we employ Concordance Correlation Coe cient (CCC) [20] as a main metric since it has been rst proposed in [27] and then widely used for continuous emotion recognition [28, 41, 48].”
MethodSelf-citationZhang, X., Pan, J., Shen, J. G., Din, Z. U., Li, J., Lu, D., Wu, M., & Hu, B. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981440 · Cited by 84 · see citing works
A Review on Facial Expression Recognition: Feature Extraction and Classification
Zhao, X., & Zhang, S. (2016) · IETE Technical Review · doi:10.1080/02564602.2015.1117403 · Cited by 75 · see citing works
Towards Robust Deep Neural Networks for Affect and Depression Recognition from Speech
Othmani, A., Kadoch, D., Bentounes, K. A., Rejaibi, E., Alfred, R., & Hadid, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_1 · Cited by 74 · see citing works
Enhanced semi-supervised learning for multimodal emotion recognition
Zhang, Z.-X., Ringeval, F., Dong, B., Coutinho, E., Marchi, E., & Schuller, B. (2016) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2016.7472666 · Cited by 83 · see citing works
“For the purpose of evaluating the different SSL approaches, we chose the RECOLA database [26] [27].”
BackgroundMethodInfluentialSelf-citationFacial Image Analysis for Fully Automatic Prediction of Difficult Endotracheal Intubation
Cuendet, G. L., Schoettker, P., Yüce, A., Sorci, M., Gao, H., Perruchoud, C., & Thiran, J. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2457032 · Cited by 71 · see citing works
“face-analysis methods, is on the rise in areas, such as marketing and emotion analysis [11], [12], face-tracking systems to increase safety in cars [13], [14] as well as in medicine [15]–[17]”
BackgroundSelf-citationTellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 75 · see citing works
“Decision Fusion combines the unimodal emotion predictions Y V and Y A (e.g., [71]).”
BackgroundResultSelf-citationHuang, Z., Dang, T., Cummins, N., Stasak, B., Le, P. N., Sethu, V., & Epps, J. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811640 · Cited by 69 · see citing works
“Common systems for learning decision level fusion weights include either Linear Regression (LR) [9, 12], or SVR [18].”
BackgroundMulti-head attention fusion networks for multi-modal speech emotion recognition
Zhang, J., Xing, L., Tan, Z., Wang, H., & Wang, K. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108078 · Cited by 66 · see citing works
A Survey on Automatic Multimodal Emotion Recognition in the Wild
Sharma, G., & Dhall, A. (2020) · Intelligent systems reference library · doi:10.1007/978-3-030-51870-7_3 · Cited by 60 · see citing works
Speech Databases, Speech Features, and Classifiers in Speech Emotion Recognition: A Review
Dar, G. H. M., & Delhibabu, R. (2024) · IEEE Access · doi:10.1109/access.2024.3476960 · Cited by 62 · see citing works
“The integration of SER in mental health monitoring holds the potential to transform mental healthcare, making it more accessible and boosting people’s overall well-being [14].”
BackgroundMethods for Facial Expression Recognition with Applications in Challenging Situations
Pise, A. A., Alqahtani, M. A., Verma, P., Purushothama, K., Karras, D. A., Prathibha, S., & Halifa, A. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9261438 · Cited by 55 · see citing works
“Researchers are now focusing their efforts on developing and commercializing powerful multimodal deep learning architectures and databases, such as the fusion of audio and visual modalities investigated by Zhang et al. [124] and Ringeval et al. [125] for audio-visual and physiological modalities.”
BackgroundWeninger, F. J., Ringeval, F., Marchi, E., & Schuller, B. W. (2016) · OPUS (Augsburg University) · Cited by 55 · see citing works
Self-citationToward User-Independent Emotion Recognition Using Physiological Signals
Albraikan, A. A., Tobón, D. P., & Saddik, A. E. (2018) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2867221 · Cited by 55 · see citing works
“pleasure and arousal [19], [26], [27], [29].”
Strength modelling for real-worldautomatic continuous affect recognition from audiovisual signals
Han, J., Zhang, Z., Cummins, N., Ringeval, F., & Schuller, B. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2016.11.020 · Cited by 53 · see citing works
“This finding is in agreement with similar results in the literature [8, 10, 26].”
BackgroundMethodResultInfluentialSelf-citationContinuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54 · see citing works
“Ringeval et al. [32] add average window to the labels and features to smooth the labels and decrease the noise.”
BackgroundZhang, Z., Ringeval, F., Han, J., Deng, J., Marchi, E., & Schuller, B. W. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-998 · Cited by 52 · see citing works
“However, while good performance has been reported in research papers under laboratory conditions [6], or with systems tailored towards specific databases [7], real-life applications of ERS still remain an open challenge.”
BackgroundSelf-citationDynamic Difficulty Awareness Training for Continuous Emotion Prediction
Zhang, Z., Han, J., Coutinho, E., & Schuller, B. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2871949 · Cited by 52 · see citing works
“For example, acoustic features empirically outperform visual features for arousal estimation, whereas the opposite occurs for valence estimation [43].”
BackgroundMethodSelf-citation(2019) · doi:10.1109/smc.2019.8914655 · Cited by 52 · see citing works
“H3C 1K9, Montr´ ´eal, QC, Canada. juan-david.silva-ortega.1@ens.etsmtl.ca, patrick.cardinal@etsmtl.ca, alessandro.koerich@etsm tl.ca the recognition of different affective dimensions. Ringeval et al. [10] investigated the relevance of using machine learning algorithms to integrate contextual information in the modelin…”
BackgroundMethodDiscriminant multi-label manifold embedding for facial Action Unit detection
Yüce, A., Gao, H., & Thiran, J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7284871 · Cited by 49 · see citing works
“SIFT features have recently been successfully applied on AU detection on the GEMEP-FERA and CK+ databases ([9],[18]).”
MethodSelf-citationPrediction-based learning for continuous emotion recognition in speech
Han, J., Zhang, Z., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953109 · Cited by 48 · see citing works
“The obtained labels were then resampled to a constant 40ms frame rate, and averaged over all raters by considering interevaluator agreement, to provide a ‘gold standard’ [21].”
MethodSelf-citationContinuous Estimation of Emotions in Speech by Dynamic Cooperative Speaker Models
Mencattini, A., Martinelli, E., Ringeval, F., Schuller, B. W., & Natale, C. D. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2531664 · Cited by 47 · see citing works
“According to previous work [27], we consider the 65 acoustic low level descriptors (LLDs) and their first order derivatives (producing 130 LLDs in total) that were used for the INTERPSEECH Computational Paralinguistic challengE since its 2013 edition [42].”
BackgroundMethodResultInfluentialSelf-citationReinforcement online learning for emotion prediction by using physiological signals
Liu, W., Zhang, L., Tao, D., & Cheng, J. S. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.06.004 · Cited by 46 · see citing works
Spectral and Cepstral Audio Noise Reduction Techniques in Speech Emotion Recognition
Pohjalainen, J., Ringeval, F., Zhang, Z., & Schuller, B. W. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967306 · Cited by 45 · see citing works
“Obtained labels were then resampled to a constant 40 ms frame rate and averaged over all raters by considering inter-evaluator agreement, to provide a ‘gold standard’ [23].”
MethodSelf-citationEnd-to-end learning for dimensional emotion recognition from physiological signals
Keren, G., Kirschstein, T., Marchi, E., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019533 · Cited by 45 · see citing works
“…dimensional emotion prediction task is the concordance correlation coefficient (CCC) [15], computed over concatenated instances [16]: where ρ is the Pearson’s correlation coefficient between the predictions and the emotion labels, µ x , µ y are the means of the two signals and σ…”
BackgroundSelf-citationSiddiqui, M. F. H., Dhakal, P., Yang, X., & Javaid, A. Y. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6060047 · Cited by 46 · see citing works
Review and Challenges of Technologies for Real-Time Human Behavior Monitoring
Dávila-Montero, S., Dana-Lê, J. A., Bente, G., Hall, A., & Mason, A. J. (2021) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2021.3060617 · Cited by 34 · see citing works
“Although similarities in window length can be identified in the literature, there is still no clear consensus regarding the optimum window length for a given modality and behavior [177].”
BackgroundPhysiological-Based Emotion Detection and Recognition in a Video Game Context
Yang, W., Rifqi, M., Marsala, C., & Pinna, A. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489125 · Cited by 43 · see citing works
“This observation is consistent with the work in [38] where ECG is more accurate for classifying valence and EDA for classifying arousal.”
ResultAnalysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants
Nalepa, G. J., Kutt, K., Giżycka, B., Jemioło, P., & Bobek, S. (2019) · Sensors · doi:10.3390/s19112509 · Cited by 41 · see citing works
Hantke, S., Weninger, F. J., Kurle, R., Ringeval, F., Batliner, A. M., Mousa, A. E.-D., & Schuller, B. W. (2016) · PLoS ONE · doi:10.1371/journal.pone.0154486 · Cited by 41 · see citing works
“In CP, there is a long tradition to go from multi-way classification problems to dimensional modelling [58] and recognition through regression, e. g., emotion recognition in the arousal-valence space [41, 59, 60].”
BackgroundSelf-citationZhang, Z., Han, J., Deng, J., Xu, X., Ringeval, F., & Schuller, B. W. (2018) · IEEE Access · doi:10.1109/access.2018.2821192 · Cited by 40 · see citing works
“These findings are in consistence with the ones reported by previous studies [38], [49].”
BackgroundResultSelf-citationEmoTour: Estimating Emotion and Satisfaction of Users Based on Behavioral Cues and Audiovisual Data
Matsuda, Y., Fedotov, D., Takahashi, Y., Arakawa, Y., Yasumoto, K., & Minker, W. (2018) · Sensors · doi:10.3390/s18113978 · Cited by 40 · see citing works
Vector score alpha integration for classifier late fusion
Safont, G., Salazar, A., & Vergara, L. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.05.014 · Cited by 36 · see citing works
Gahlan, N., & Sethia, D. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106353 · Cited by 36 · see citing works
Galvanic Skin Response Data Classification for Emotion Detection
Setyohadi, D. B., Kusrohmaniah, S., Gunawan, S. B., Pranowo, P., & Prabuwono, A. S. (2018) · International Journal of Electrical and Computer Engineering (IJECE) · doi:10.11591/ijece.v8i5.pp4004-4014 · Cited by 34 · see citing works
Continuous Emotion Recognition in Speech — Do We Need Recurrence?
Schmitt, M., Cummins, N., & Schuller, B. W. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2710 · Cited by 34 · see citing works
“Moreover, ERS is nowadays not commonly performed on discrete chunks of audio, but rather in a ‘time-continuous’ manner, assigning a level of arousal and valence to each time-stamp of an audio (or audio-visual) data stream [8].”
BackgroundMethodSelf-citationMuscle activation strategies of people with early-stage Parkinson’s during walking
Keloth, S. M., Arjunan, S. P., Raghav, S., & Kumar, D. (2021) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-021-00932-1 · Cited by 34 · see citing works
“Reduced production of dopamine neurotransmitter in the substantia nigra leads to the excessive inhibition of the basal ganglia loop which causes loss of habitual patterns associated with walking [2] and decreased range of limb movement [3].”
BackgroundVijayakumar, S., Flynn, R., & Murray, N. (2020) · Irish Signals and Systems Conference · doi:10.1109/issc49989.2020.9180193 · Cited by 33 · see citing works
“Emotion recognition using EEG signals, combined with machine learning, pattern recognition [8] and deep learning algorithms [4], [9], [10], [11] has emerged as a powerful approach to the recognition of emotions since it can extract the cognitive state of the user from the CNS. Studies have also…”
BackgroundCoutinho, E., & Schuller, B. W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179289 · Cited by 32 · see citing works
Self-citationAudio-video emotional response mapping based upon Electrodermal Activity
Sharma, V. K., Prakash, N. R., & Kalra, P. (2018) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.024 · Cited by 31 · see citing works
Building a three-level multimodal emotion recognition framework
Garcia-Garcia, J. M., Lozano, M. D., Penichet, V. M. R., & Law, E. L. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13254-8 · Cited by 32 · see citing works
“For a long time, the shortage of multimodal affective databases limited the amount of works that could be done in this field, even though some multimodal databases have appeared in the last years (e.g., DEAP [36], RECOLA [57], BP4D+ [73]).”
BackgroundA machine learning based system for the automatic evaluation of aphasia speech
Kohlschein, C., Schmitt, M., Schuller, B., Jeschke, S., & Werner, C. J. (2017) · 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom.2017.8210766 · Cited by 29 · see citing works
“In contrast to conventional feed-forward or recurrent neural networks, Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) are well-suited for classification or detection tasks in sequential data, such as speech signals [33], [34].”
BackgroundSelf-citationTang, D., Kuppens, P., Geurts, L., & van Waterschoot, T. (2021) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-021-00208-5 · Cited by 29 · see citing works
“The mean squared error (MSE) or mean absolute error (MAE) loss on the sample level as used in [3] does not consider this correlation.”
BackgroundMethodHui, T., & Sherratt, R. S. (2016) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-016-0409-9 · Cited by 29 · see citing works
“Research on Emotion Prediction normally investigates the features extraction and classifi-cation methods on contexts stimulating emotion using machine learning algorithms (Ringeval et al. 2015; Fair-hurst et al. 2015), or focuses on the prediction algorithms under controlled contextual stimuli.”
iAware: A Real-Time Emotional Biofeedback System Based on Physiological Signals
Albraikan, A., Hafidh, B., & Saddik, A. E. (2018) · IEEE Access · doi:10.1109/access.2018.2885279 · Cited by 31 · see citing works
Kim, Y., & Kim, J. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462011 · Cited by 27 · see citing works
“employed a normalization technique to increase the inter-rater agreement, while preserving the original balancing of the dimensional ratings [2]; Shah et al.”
MethodHolistic Affect Recognition Using PaNDA: Paralinguistic Non-Metric Dimensional Analysis
Zhang, Y., Weninger, F. J., Björn, S., & Picard, R. W. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961881 · Cited by 26 · see citing works
“A wealth of research pertains to multi-label learning for affect recognition, using a single database labeled with a few affective dimensions, predominantly arousal/ valence/ dominance [12, 13, 14].”
BackgroundA systematic review of emotion recognition using cardio-based signals
Ismail, S. N. M. S., Aziz, N. A. A., Ibrahim, S. Z., & Mohamad, M. S. (2023) · ICT Express · doi:10.1016/j.icte.2023.09.001 · Cited by 29 · see citing works
Action Units and Their Cross-Correlations for Prediction of Cognitive Load during Driving
Yüce, A., Gao, H., Cuendet, G. L., & Thiran, J. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2584042 · Cited by 25 · see citing works
Self-citationPredicting the distribution of emotion perception: capturing inter-rater variability
Zhang, B., Essl, G., & Provost, E. M. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136792 · Cited by 25 · see citing works
“However, most work using dimensional labels focuses on predicting either the mean evaluation across multiple evaluators [10, 13, 17, 19, 21, 36] or the classes caterogized from the mean evaluation [18, 28].”
BackgroundResultInfluentialSaluky, S., & Bahiyah, N. (2023) · International Journal of Education and Humanities · doi:10.58557/(ijeh).v3i2.149 · Cited by 25 · see citing works
“Recognizing the need for innovative approaches that embrace flexibility, asynchronous video presents itself as a promising solution (Ringeval et al., 2015).”
BackgroundJointly Aligning and Predicting Continuous Emotion Annotations
Khorram, S., McInnis, M. G., & Provost, E. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2917047 · Cited by 25 · see citing works
“[33] applied a long short-term memory (LSTM) network [38] with an analysis window to deal with simultaneous modeling of reaction delays and emotion labels.”
Keloth, S. M., Viswanathan, R., Jelfs, B., Arjunan, S. P., Raghav, S., & Kumar, D. (2019) · Biosensors · doi:10.3390/bios9020059 · Cited by 24 · see citing works
“Loss of dopamine in the substantia nigra leading to the excessive inhibition of the basal ganglia loop leads to the loss of habitual patterns [70] associated with walking and also causes rigid movement and decreased range of limb movement [77].”
BackgroundEmotion Recognition from Physiological Signals using Multi-Hypergraph Neural Networks
Zhu, J., Zhao, X., Hu, H., & Gao, Y. (2019) · 2019 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00111 · Cited by 23 · see citing works
“Missing data will also lead to parameter estimation bias and hypothesis testing errors of inference, which will negatively influence the learning methods for emotion recognition [10].”
BackgroundA neuro-advertising property video recommendation system
Kaklauskas, A., Zavadskas, E. K., Banaitis, A., Meidutė‐Kavaliauskienė, I., Liberman, A., Dziţac, S., Ubartė, I., Binkyte, A., Čerkauskas, J., Kuzminskė, A., et al. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.07.011 · Cited by 24 · see citing works
A Computational Model for Managing Impressions of an Embodied Conversational Agent in Real-Time
Biancardi, B., Wang, C., Mancini, M., Cafaro, A., Chanel, G., & Pélachaud, C. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925495 · Cited by 23 · see citing works
“Facial signals can reveal the users’ affective states, and potentially formed impressions, faster compared to other modalities such as heart rate and body temperature [23], [24].”
Monocular 3D Facial Expression Features for Continuous Affect Recognition
Pei, E., Oveneke, M. C., Zhao, Y. H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3026894 · Cited by 22 · see citing works
Multi-channel EEG-based emotion recognition in the presence of noisy labels
Li, C., Hou, Y., Song, R., Cheng, J., Liu, Y., & Chen, X. (2022) · Science China Information Sciences · doi:10.1007/s11432-021-3439-2 · Cited by 22 · see citing works
Towards Conditional Adversarial Training for Predicting Emotions from Speech
Han, J., Zhang, Z., Ren, Z. X., Ringeval, F., & Schuller, B. W. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462579 · Cited by 21 · see citing works
“Henceforth, LSTM-RNNs (or BLSTM for its bi-directional version) is frequently employed for time-and value-continuous SER as it has been found very effective [1–3].”
MethodSelf-citationFPGA Acceleration of LSTM Based on Data for Test Flight
Sun, Z., Zhu, Y., Zheng, Y., Wu, H., Cao, Z., Xiong, P., Hou, J., Huang, T., & Que, Z. (2018) · IEEE International Conference on Smart Cloud (SmartCloud) · doi:10.1109/smartcloud.2018.00009 · Cited by 21 · see citing works
“Different from other hot prediction issues in LSTM model such as speech recognition [3][4], human emotion perception[5] , question answering[6] and traffic monitoring[7], the flight data prediction puts forward requirements in two aspects.”
BackgroundHuang, Z., & Epps, J. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2821135 · Cited by 21 · see citing works
“For arousal and valence ground truth, ratings from 6 annotators on RECOLA and SEMAINE were normalized using the normalization technique [41] before averaging across all annotators, as per the AVEC 2016 challenge [39].”
MethodA Novel Method for Human Bias Correction of Continuous- Time Annotations
Booth, B. M., Mundnich, K., & Narayanan, S. S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461645 · Cited by 20 · see citing works
“Many other feature-based methods have been proposed [6, 7, 8, 9, 10] and perform reasonably well on various data sets, but require an informative set of features to be extracted from the stimuli for alignment and fusion.”
A Novel LSTM Approach for Asynchronous Multivariate Time Series Prediction
Ma, K. F., & Leung, H. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851792 · Cited by 20 · see citing works
“When employing more variables in the model, or combining multiple univariate prediction models [10], the number of possible combination of lags grows combinatorically [11] and it becomes difficult to compute optimal lags.”
MethodMultimodal-multisensor affect detection
D’Mello, S. K., Bosch, N., & Chen, H. (2018) · ACM eBooks · doi:10.1145/3107990.3107998 · Cited by 19 · see citing works
“Ringeval et al. [2015a] fused video, audio, and physiology and showed advantages of LSTMs and BLSTMs compared to feed-forward neural networks (this study is discussed in more detail below).”
BackgroundAn efficient model-level fusion approach for continuous affect recognition from audiovisual signals
Pei, E., Jiang, D., & Hichem Sahli, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.09.037 · Cited by 19 · see citing works
On-the-Fly Facial Expression Prediction Using LSTM Encoded Appearance-Suppressed Dynamics
Baddar, W. J., Lee, S., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2957465 · Cited by 19 · see citing works
An Investigation of Emotion Prediction Uncertainty Using Gaussian Mixture Regression
Dang, T., Sethu, V., Epps, J., & Eliathamby Ambikairajah, A. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-512 · Cited by 19 · see citing works
“Several studies [4-7] have showed the importance of taking information from multiple raters into account for both categorical emotion recognition and continuous emotion prediction systems, or have argued that emotion attributes should be ranked instead of trying to predict absolute ordinal values.”
BackgroundDesign and Implementation of LSTM Accelerator Based on FPGA
Zhang, W., Ge, F., Cui, C., Yang, Y., Zhou, F., & Wu, N. (2020) · IEEE International Conference on Communication Technology (ICCT) · doi:10.1109/icct50939.2020.9295665 · Cited by 19 · see citing works
“LSTM is widely used in text recognition [5], scene analysis [6] and sentiment analysis [7].”
MethodDimensional Affect Recognition from HRV: An Approach Based on Supervised SOM and ELM
Bugnon, L., Calvo, R., & Milone, D. H. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2763943 · Cited by 17 · see citing works
“Many of the revised works ([57], [58], [59], [61]) were based on the LSTM model, which was introduced for this database in [52] and it is considered a state-of-the-art model for dimensional affect estimation [7], [57].”
BackgroundSupervised Contrastive Learning for Affect Modelling
Pinitas, K., Makantasis, K., Liapis, A., & Yannakakis, G. N. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556584 · Cited by 17 · see citing works
“In particular, the above results show that arousal is better captured in the provided audio, visual, and physiology features obtained by concatenating ECG and EDA features [35] [50] [46].”
BackgroundRETRACTED ARTICLE: Recognition of emotions in speech using deep CNN and RESNET
Lakshmi, K. L., Muthulakshmi, P., Nithya, A. A., Jeyavathana, R. B., Usharani, R., Das, N. S., & Devi, G. N. R. (2023) · Soft Computing · doi:10.1007/s00500-023-07969-5 · Cited by 18 · see citing works
RetractedOutlier Processing in Multimodal Emotion Recognition
Zhang, G., Luo, T., Pedrycz, W., El-Meligy, M. A., Sharaf, M. A. F., & Li, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2981760 · Cited by 17 · see citing works
“In a fusion strategy, the fusion method of feature-level and decision-level is frequently used [22].”
BackgroundMethodInfluentialReal-world automatic continuous affect recognition from audiovisual signals
Tzirakis, P., Zafeiriou, S. P., & Schuller, B. (2019) · Multimodal Behavior Analysis in the Wild · doi:10.1016/b978-0-12-814601-9.00028-6 · Cited by 20 · see citing works
Self-citationJi, X., Sun, L., & Huang, K. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101379 · Cited by 15 · see citing works
Analyzing the expression of annoyance during phone calls to complaint services
Irastorza, J., & Torres, M. I. (2016) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom.2016.7804533 · Cited by 15 · see citing works
“In this approach recurrent neural networks have been proposed to integrate contextual information and then predict emotion in continuous time to just deal with arousal and valence [19] [20].”
BackgroundModeling Emotions in Robotic Socially Believable Behaving Systems
Esposito, A. A., & Jain, L. C. (2016) · Intelligent systems reference library · doi:10.1007/978-3-319-31056-5_2 · Cited by 15 · see citing works
- Authors
- Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B.
- Affiliations
- University of Fribourg (CH)
- Technical University of Munich (DE)
- École Polytechnique Fédérale de Lausanne (CH)
- University of Passau (DE)
- Corresponding author
- Ringeval, Fabien
- ORCID
- Venue
- Pattern Recognition Letters
- Publisher
- Elsevier BV
- Published
- 2015-11
- Volume
- 66
- Pages
- 22-30
- ISSN
- 0167-8655 (ISSN-L)
- OpenAlex
- W2005418748
- Cited by
- 202
- References
- 63
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 4 · 2023: 15 · 2022: 14 · 2021: 14 · 2020: 17 · 2019: 25 · 2018: 38 · 2017: 23 · 2016: 30 · 2015: 9
- FWCI
- 26.12
- Type
- journal article
- Language
- en
- Open access
- green
- infoscience.epfl.ch
- Locations
- Pattern Recognition Letters (publishedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- OPUS (Augsburg University) (submittedVersion, cc-by-nc-nd)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Seventh Framework Programme
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- automatic emotion recognition, arousal, valence, long short-term memory, recurrent neural networks, multimodal fusion, feature-level fusion, decision fusion, audiovisual information, physiological signals, ECG, electrodermal activity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
