The DOI matches its Crossref record; 58 of the 63 references it cites were found in a registry or an index.
Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Open accessOverview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Learning joint feature representations for multimodal data. In multimodal approaches, DNNs are leveraged to learn joint feature representations from multiple unimodal feature representations to accomplish feature-level fusion [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
In the citing paper
“They found that while the given physiological signal has lower predictive power than audiovisual signals, both are complementary.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
In the citing paper
“One study suggested that a smaller window size (2s) could be the best choice [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
In the citing paper
“Depending on the nature of the source audio, it can be beneficial to apply HSFs to frame-level LLDs according to a sliding window before applying the RNN [151], [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
In the citing paper
“They concluded that in featurelevel fusion, ECG data helps for prediction of valence, but not arousal.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
In the citing paper
“To date, the literature generally reports that decision-level fusion works better for affect recognition given the datasets and models currently used [65].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
In the citing paper
“However, several other studies based on handcrafted features found that decision-level fusion on top of individual LSTM models leads to better performance [65], [129].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [65]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
A Real-Time QRS Detection Algorithm
Pan, J., & Tompkins, W. J. (1985) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.1985.325532 · Cited by 7,855
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 6,152
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Two-Frame Motion Estimation Based on Polynomial Expansion
Farnebäck, G. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45103-x_50 · Cited by 3,112
Cited by
Sorted by citations
A review on the long short-term memory model
Houdt, G. V., Mosquera, C., & Nápoles, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09838-1 · Cited by 2,198
“Manyotherapplicationsexistfortimeseriespredictionscenarios, like emotion ratings (Ringeval et al. 2015), topic evolution in text streams (Lu et al. 2019), building energy usage (Wang et al. 2019b), carbon emission levels (Huang et al. 2019b), flight delays (McCarthy et al. 2019), financial trading…”
BackgroundAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904
“As a second baseline algorithm, we utilised a BLSTM-DRNNs with the architecture preserved from [5, 21], i.”
MethodInfluentialSelf-citationEnd-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752
Self-citationValstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988258 · Cited by 540
“A normalisation technique based on the Evaluator Weighted Estimator [30], is used prior to the computation of the gold-standard, i. e., the average of all ratings for each subject [36].”
BackgroundMethodInfluentialSelf-citationFacial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 370
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236
“averaging, as in [29], to more complex methods such as linear and multi-linear regression [15,25,26,28,30,37,38], SVR [39], random forests [32] or Kalman filters based [34–36].”
MethodInfluentialDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Leveraging recent advances in deep learning for audio-Visual emotion recognition
Schoneveld, L., Othmani, A., & Abdelkawy, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.007 · Cited by 243
“The performances of the proposed approach have been evaluated using the REmote COLlaborative and affective interactions (RECOLA) corpus (Ringeval et al. [2013]).”
MethodAVEC 2018 Workshop and Challenge
Ringeval, F., Schuller, B. W., Valstar, M., Cowie, R., Kaya, H., Schmitt, M., Amiriparian, S., Cummins, N., Lalanne, D., Michaud, A., et al. (2018) · doi:10.1145/3266302.3266316 · Cited by 149
“Even though individual ratings of emotion can be modelled all together in a multi-task learning framework [58], the dominant practice is to summarise the annotations for each recording into a single time-series, referred as gold-standard in the literature3, that can be then easily processed by any machine learning alg…”
BackgroundSelf-citationMultimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition
Chen, S., Jin, Q., Zhao, J., & Wang, S. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133949 · Cited by 147
“Ringeval et al. [16] explore two types of multi-task learning for dimension emotion recognition.”
Background(2015) · doi:10.1145/2808196.2811642 · Cited by 178
Self-citationSchmitt, M., Ringeval, F., & Schuller, B. W. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1124 · Cited by 135
“ported by the annotators are not contemporaneous, due to the reaction time of the raters, this delay must be taken into account when using a non-context aware machine learning algorithm [25].”
MethodSelf-citationFacial Emotion Expressions in Human–Robot Interaction: A Survey
Rawal, N., & Stock‐Homburg, R. (2022) · International Journal of Social Robotics · doi:10.1007/s12369-022-00867-0 · Cited by 152
“There are very few studies that combine facial data with both audio and physiological data [107, 106] and studies that analyze all modalities (face, voice, text, body gestures and physiological signals) have not been found.”
BackgroundLong Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811634 · Cited by 124
“Except the above methods, [10] adds window to the label and features.”
BackgroundMethodResultInfluentialSemi-Supervised Speech Emotion Recognition With Ladder Networks
Parthasarathy, S., & Busso, C. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3023632 · Cited by 119
“Previous studies have shown the benefits of training with CCC as the objective function over the MSE [12], [39], [50].”
BackgroundResultA Review of Recurrent Neural Network-Based Methods in Computational Physiology
Mao, S., & Sejdić, E. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3145365 · Cited by 128
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811638 · Cited by 114
“In work [20], two types of multi-task learning are introduced: one by learning each rater’s individual track and the other by learning both dimensions simultaneously.”
BackgroundMethodInfluentialHow deep neural networks can improve emotion recognition on video data
Khorrami, P., Le Paine, T., Brady, K., Dagli, C., & Huang, T. S. (2016) · 2016 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2016.7532431 · Cited by 107
“While many works [12, 13, 14, 15] have trained recurrent neural networks (RNNs) on hand-crafted features, few works [16, 17] have considered the effects of using a CNN as input to the RNN.”
BackgroundAVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and ChallengePreprint
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01600 · Cited by 129
“A normalisation technique based on the Evaluator Weighted Estimator [30], is used prior to the computation of the gold-standard, i. e., the average of all ratings for each subject [36].”
BackgroundMethodInfluentialSelf-citationAutomatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101
“Some studies have applied a ConvNet with a different number of layers to physiological signals to classify different emotions [28], [30]–[32].”
BackgroundCurriculum Learning for Speech Emotion Recognition From Crowdsourced Labels
Lotfian, R., & Busso, C. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2898816 · Cited by 103
“This metric has been previously used in related studies [55], including the audio/visual emotion challenge (AVEC) in 2015 [56].”
MethodMulti-Modal Recurrent Attention Networks for Facial Expression Recognition
Lee, J., Kim, S., Kim, S., & Sohn, K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2996086 · Cited by 100
“To alleviate this limitation, dimensional emotion descriptions [8], [12]–[15] have attracted much attention, where emotions are described in a continuous space, consisting of”
BackgroundMuzammel, M., Salam, H., & Othmani, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106433 · Cited by 91
Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction
Brady, K., Gwon, Y., Khorrami, P., Godoy, E., Campbell, W., Dagli, C., & Huang, T. S. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988264 · Cited by 110
Han, J., Zhang, Z., Schmitt, M., Pantić, M., & Schuller, B. W. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123383 · Cited by 89
“To estimate the performance of the proposed framework, we employ Concordance Correlation Coe cient (CCC) [20] as a main metric since it has been rst proposed in [27] and then widely used for continuous emotion recognition [28, 41, 48].”
MethodSelf-citationZhang, X., Pan, J., Shen, J. G., Din, Z. U., Li, J., Lu, D., Wu, M., & Hu, B. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981440 · Cited by 84
A Review on Facial Expression Recognition: Feature Extraction and Classification
Zhao, X., & Zhang, S. (2016) · IETE Technical Review · doi:10.1080/02564602.2015.1117403 · Cited by 78
Towards Robust Deep Neural Networks for Affect and Depression Recognition from Speech
Othmani, A., Kadoch, D., Bentounes, K. A., Rejaibi, E., Alfred, R., & Hadid, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_1 · Cited by 79
Enhanced semi-supervised learning for multimodal emotion recognition
Zhang, Z.-X., Ringeval, F., Dong, B., Coutinho, E., Marchi, E., & Schuller, B. (2016) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2016.7472666 · Cited by 83
“For the purpose of evaluating the different SSL approaches, we chose the RECOLA database [26] [27].”
BackgroundMethodInfluentialSelf-citationFacial Image Analysis for Fully Automatic Prediction of Difficult Endotracheal Intubation
Cuendet, G. L., Schoettker, P., Yüce, A., Sorci, M., Gao, H., Perruchoud, C., & Thiran, J. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2457032 · Cited by 71
“face-analysis methods, is on the rise in areas, such as marketing and emotion analysis [11], [12], face-tracking systems to increase safety in cars [13], [14] as well as in medicine [15]–[17]”
BackgroundSelf-citationTellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 75
“Decision Fusion combines the unimodal emotion predictions Y V and Y A (e.g., [71]).”
BackgroundResultSelf-citationHuang, Z., Dang, T., Cummins, N., Stasak, B., Le, P. N., Sethu, V., & Epps, J. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811640 · Cited by 69
“Common systems for learning decision level fusion weights include either Linear Regression (LR) [9, 12], or SVR [18].”
BackgroundMulti-head attention fusion networks for multi-modal speech emotion recognition
Zhang, J., Xing, L., Tan, Z., Wang, H., & Wang, K. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108078 · Cited by 66
A Survey on Automatic Multimodal Emotion Recognition in the Wild
Sharma, G., & Dhall, A. (2020) · Intelligent systems reference library · doi:10.1007/978-3-030-51870-7_3 · Cited by 64
Speech Databases, Speech Features, and Classifiers in Speech Emotion Recognition: A Review
Dar, G. H. M., & Delhibabu, R. (2024) · IEEE Access · doi:10.1109/access.2024.3476960 · Cited by 68
“The integration of SER in mental health monitoring holds the potential to transform mental healthcare, making it more accessible and boosting people’s overall well-being [14].”
BackgroundMethods for Facial Expression Recognition with Applications in Challenging Situations
Pise, A. A., Alqahtani, M. A., Verma, P., Purushothama, K., Karras, D. A., Prathibha, S., & Halifa, A. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9261438 · Cited by 64
“Researchers are now focusing their efforts on developing and commercializing powerful multimodal deep learning architectures and databases, such as the fusion of audio and visual modalities investigated by Zhang et al. [124] and Ringeval et al. [125] for audio-visual and physiological modalities.”
BackgroundWeninger, F. J., Ringeval, F., Marchi, E., & Schuller, B. W. (2016) · OPUS (Augsburg University) · Cited by 55
Self-citationToward User-Independent Emotion Recognition Using Physiological Signals
Albraikan, A. A., Tobón, D. P., & Saddik, A. E. (2018) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2867221 · Cited by 57
“pleasure and arousal [19], [26], [27], [29].”
Strength modelling for real-worldautomatic continuous affect recognition from audiovisual signals
Han, J., Zhang, Z., Cummins, N., Ringeval, F., & Schuller, B. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2016.11.020 · Cited by 53
“This finding is in agreement with similar results in the literature [8, 10, 26].”
BackgroundMethodResultInfluentialSelf-citationContinuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54
“Ringeval et al. [32] add average window to the labels and features to smooth the labels and decrease the noise.”
BackgroundZhang, Z., Ringeval, F., Han, J., Deng, J., Marchi, E., & Schuller, B. W. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-998 · Cited by 52
“However, while good performance has been reported in research papers under laboratory conditions [6], or with systems tailored towards specific databases [7], real-life applications of ERS still remain an open challenge.”
BackgroundSelf-citationDynamic Difficulty Awareness Training for Continuous Emotion Prediction
Zhang, Z., Han, J., Coutinho, E., & Schuller, B. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2871949 · Cited by 53
“For example, acoustic features empirically outperform visual features for arousal estimation, whereas the opposite occurs for valence estimation [43].”
BackgroundMethodSelf-citation(2019) · doi:10.1109/smc.2019.8914655 · Cited by 52
“H3C 1K9, Montr´ ´eal, QC, Canada. juan-david.silva-ortega.1@ens.etsmtl.ca, patrick.cardinal@etsmtl.ca, alessandro.koerich@etsm tl.ca the recognition of different affective dimensions. Ringeval et al. [10] investigated the relevance of using machine learning algorithms to integrate contextual information in the modelin…”
BackgroundMethodDiscriminant multi-label manifold embedding for facial Action Unit detection
Yüce, A., Gao, H., & Thiran, J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7284871 · Cited by 50
“SIFT features have recently been successfully applied on AU detection on the GEMEP-FERA and CK+ databases ([9],[18]).”
MethodSelf-citationPrediction-based learning for continuous emotion recognition in speech
Han, J., Zhang, Z., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953109 · Cited by 48
“The obtained labels were then resampled to a constant 40ms frame rate, and averaged over all raters by considering interevaluator agreement, to provide a ‘gold standard’ [21].”
MethodSelf-citationContinuous Estimation of Emotions in Speech by Dynamic Cooperative Speaker Models
Mencattini, A., Martinelli, E., Ringeval, F., Schuller, B. W., & Natale, C. D. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2531664 · Cited by 47
“According to previous work [27], we consider the 65 acoustic low level descriptors (LLDs) and their first order derivatives (producing 130 LLDs in total) that were used for the INTERPSEECH Computational Paralinguistic challengE since its 2013 edition [42].”
BackgroundMethodResultInfluentialSelf-citationReinforcement online learning for emotion prediction by using physiological signals
Liu, W., Zhang, L., Tao, D., & Cheng, J. S. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.06.004 · Cited by 46
Spectral and Cepstral Audio Noise Reduction Techniques in Speech Emotion Recognition
Pohjalainen, J., Ringeval, F., Zhang, Z., & Schuller, B. W. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967306 · Cited by 45
“Obtained labels were then resampled to a constant 40 ms frame rate and averaged over all raters by considering inter-evaluator agreement, to provide a ‘gold standard’ [23].”
MethodSelf-citationEnd-to-end learning for dimensional emotion recognition from physiological signals
Keren, G., Kirschstein, T., Marchi, E., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019533 · Cited by 48
“…dimensional emotion prediction task is the concordance correlation coefficient (CCC) [15], computed over concatenated instances [16]: where ρ is the Pearson’s correlation coefficient between the predictions and the emotion labels, µ x , µ y are the means of the two signals and σ…”
BackgroundSelf-citationSiddiqui, M. F. H., Dhakal, P., Yang, X., & Javaid, A. Y. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6060047 · Cited by 46
Review and Challenges of Technologies for Real-Time Human Behavior Monitoring
Dávila-Montero, S., Le, J. A., Bente, G., Hall, A. T., & Mason, A. J. (2021) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2021.3060617 · Cited by 48
“Although similarities in window length can be identified in the literature, there is still no clear consensus regarding the optimum window length for a given modality and behavior [177].”
BackgroundPhysiological-Based Emotion Detection and Recognition in a Video Game Context
Yang, W., Rifqi, M., Marsala, C., & Pinna, A. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489125 · Cited by 43
“This observation is consistent with the work in [38] where ECG is more accurate for classifying valence and EDA for classifying arousal.”
ResultAnalysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants
Nalepa, G. J., Kutt, K., Giżycka, B., Jemioło, P., & Bobek, S. (2019) · Sensors · doi:10.3390/s19112509 · Cited by 41
Hantke, S., Weninger, F. J., Kurle, R., Ringeval, F., Batliner, A. M., Mousa, A. E.-D., & Schuller, B. W. (2016) · PLoS ONE · doi:10.1371/journal.pone.0154486 · Cited by 41
“In CP, there is a long tradition to go from multi-way classification problems to dimensional modelling [58] and recognition through regression, e. g., emotion recognition in the arousal-valence space [41, 59, 60].”
BackgroundSelf-citationZhang, Z., Han, J., Deng, J., Xu, X., Ringeval, F., & Schuller, B. W. (2018) · IEEE Access · doi:10.1109/access.2018.2821192 · Cited by 40
“These findings are in consistence with the ones reported by previous studies [38], [49].”
BackgroundResultSelf-citationEmoTour: Estimating Emotion and Satisfaction of Users Based on Behavioral Cues and Audiovisual Data
Matsuda, Y., Fedotov, D., Takahashi, Y., Arakawa, Y., Yasumoto, K., & Minker, W. (2018) · Sensors · doi:10.3390/s18113978 · Cited by 40
Vector score alpha integration for classifier late fusion
Safont, G., Salazar, A., & Vergara, L. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.05.014 · Cited by 36
Gahlan, N., & Sethia, D. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106353 · Cited by 36
Galvanic Skin Response Data Classification for Emotion Detection
Setyohadi, D. B., Kusrohmaniah, S., Gunawan, S. B., Pranowo, P., & Prabuwono, A. S. (2018) · International Journal of Electrical and Computer Engineering (IJECE) · doi:10.11591/ijece.v8i5.pp4004-4014 · Cited by 34
Continuous Emotion Recognition in Speech — Do We Need Recurrence?
Schmitt, M., Cummins, N., & Schuller, B. W. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2710 · Cited by 38
“Moreover, ERS is nowadays not commonly performed on discrete chunks of audio, but rather in a ‘time-continuous’ manner, assigning a level of arousal and valence to each time-stamp of an audio (or audio-visual) data stream [8].”
BackgroundMethodSelf-citationMuscle activation strategies of people with early-stage Parkinson’s during walking
Keloth, S. M., Arjunan, S. P., Raghav, S., & Kumar, D. (2021) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-021-00932-1 · Cited by 34
“Reduced production of dopamine neurotransmitter in the substantia nigra leads to the excessive inhibition of the basal ganglia loop which causes loss of habitual patterns associated with walking [2] and decreased range of limb movement [3].”
BackgroundVijayakumar, S., Flynn, R., & Murray, N. (2020) · Irish Signals and Systems Conference · doi:10.1109/issc49989.2020.9180193 · Cited by 33
“Emotion recognition using EEG signals, combined with machine learning, pattern recognition [8] and deep learning algorithms [4], [9], [10], [11] has emerged as a powerful approach to the recognition of emotions since it can extract the cognitive state of the user from the CNS. Studies have also…”
BackgroundCoutinho, E., & Schuller, B. W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179289 · Cited by 33
Self-citationAudio-video emotional response mapping based upon Electrodermal Activity
Sharma, V. K., Prakash, N. R., & Kalra, P. (2018) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.024 · Cited by 31
Building a three-level multimodal emotion recognition framework
Garcia-Garcia, J. M., Lozano, M. D., Penichet, V. M. R., & Law, E. L. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13254-8 · Cited by 32
“For a long time, the shortage of multimodal affective databases limited the amount of works that could be done in this field, even though some multimodal databases have appeared in the last years (e.g., DEAP [36], RECOLA [57], BP4D+ [73]).”
BackgroundA machine learning based system for the automatic evaluation of aphasia speech
Kohlschein, C., Schmitt, M., Schuller, B., Jeschke, S., & Werner, C. J. (2017) · 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom.2017.8210766 · Cited by 29
“In contrast to conventional feed-forward or recurrent neural networks, Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) are well-suited for classification or detection tasks in sequential data, such as speech signals [33], [34].”
BackgroundSelf-citationTang, D., Kuppens, P., Geurts, L., & van Waterschoot, T. (2021) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-021-00208-5 · Cited by 30
“The mean squared error (MSE) or mean absolute error (MAE) loss on the sample level as used in [3] does not consider this correlation.”
BackgroundMethodHui, T., & Sherratt, R. S. (2016) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-016-0409-9 · Cited by 32
“Research on Emotion Prediction normally investigates the features extraction and classifi-cation methods on contexts stimulating emotion using machine learning algorithms (Ringeval et al. 2015; Fair-hurst et al. 2015), or focuses on the prediction algorithms under controlled contextual stimuli.”
iAware: A Real-Time Emotional Biofeedback System Based on Physiological Signals
Albraikan, A., Hafidh, B., & Saddik, A. E. (2018) · IEEE Access · doi:10.1109/access.2018.2885279 · Cited by 31
Kim, Y., & Kim, J. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462011 · Cited by 28
“employed a normalization technique to increase the inter-rater agreement, while preserving the original balancing of the dimensional ratings [2]; Shah et al.”
MethodHolistic Affect Recognition Using PaNDA: Paralinguistic Non-Metric Dimensional Analysis
Zhang, Y., Weninger, F. J., Björn, S., & Picard, R. W. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961881 · Cited by 26
“A wealth of research pertains to multi-label learning for affect recognition, using a single database labeled with a few affective dimensions, predominantly arousal/ valence/ dominance [12, 13, 14].”
BackgroundA systematic review of emotion recognition using cardio-based signals
Ismail, S. N. M. S., Aziz, N. A. A., Ibrahim, S. Z., & Mohamad, M. S. (2023) · ICT Express · doi:10.1016/j.icte.2023.09.001 · Cited by 29
Action Units and Their Cross-Correlations for Prediction of Cognitive Load during Driving
Yüce, A., Gao, H., Cuendet, G. L., & Thiran, J. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2584042 · Cited by 25
Self-citationPredicting the distribution of emotion perception: capturing inter-rater variability
Zhang, B., Essl, G., & Provost, E. M. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136792 · Cited by 27
“However, most work using dimensional labels focuses on predicting either the mean evaluation across multiple evaluators [10, 13, 17, 19, 21, 36] or the classes caterogized from the mean evaluation [18, 28].”
BackgroundResultInfluentialSaluky, S., & Bahiyah, N. (2023) · International Journal of Education and Humanities · doi:10.58557/(ijeh).v3i2.149 · Cited by 25
“Recognizing the need for innovative approaches that embrace flexibility, asynchronous video presents itself as a promising solution (Ringeval et al., 2015).”
BackgroundJointly Aligning and Predicting Continuous Emotion Annotations
Khorram, S., McInnis, M. G., & Provost, E. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2917047 · Cited by 25
“[33] applied a long short-term memory (LSTM) network [38] with an analysis window to deal with simultaneous modeling of reaction delays and emotion labels.”
Keloth, S. M., Viswanathan, R., Jelfs, B., Arjunan, S. P., Raghav, S., & Kumar, D. (2019) · Biosensors · doi:10.3390/bios9020059 · Cited by 24
“Loss of dopamine in the substantia nigra leading to the excessive inhibition of the basal ganglia loop leads to the loss of habitual patterns [70] associated with walking and also causes rigid movement and decreased range of limb movement [77].”
BackgroundEmotion Recognition from Physiological Signals using Multi-Hypergraph Neural Networks
Zhu, J., Zhao, X., Hu, H., & Gao, Y. (2019) · 2019 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00111 · Cited by 23
“Missing data will also lead to parameter estimation bias and hypothesis testing errors of inference, which will negatively influence the learning methods for emotion recognition [10].”
BackgroundA neuro-advertising property video recommendation system
Kaklauskas, A., Zavadskas, E. K., Banaitis, A., Meidutė‐Kavaliauskienė, I., Liberman, A., Dziţac, S., Ubartė, I., Binkyte, A., Čerkauskas, J., Kuzminskė, A., et al. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.07.011 · Cited by 24
A Computational Model for Managing Impressions of an Embodied Conversational Agent in Real-Time
Biancardi, B., Wang, C., Mancini, M., Cafaro, A., Chanel, G., & Pélachaud, C. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925495 · Cited by 23
“Facial signals can reveal the users’ affective states, and potentially formed impressions, faster compared to other modalities such as heart rate and body temperature [23], [24].”
Monocular 3D Facial Expression Features for Continuous Affect Recognition
Pei, E., Oveneke, M. C., Zhao, Y. H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3026894 · Cited by 22
Multi-channel EEG-based emotion recognition in the presence of noisy labels
Li, C., Hou, Y., Song, R., Cheng, J., Liu, Y., & Chen, X. (2022) · Science China Information Sciences · doi:10.1007/s11432-021-3439-2 · Cited by 22
Towards Conditional Adversarial Training for Predicting Emotions from Speech
Han, J., Zhang, Z., Ren, Z. X., Ringeval, F., & Schuller, B. W. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462579 · Cited by 21
“Henceforth, LSTM-RNNs (or BLSTM for its bi-directional version) is frequently employed for time-and value-continuous SER as it has been found very effective [1–3].”
MethodSelf-citationFPGA Acceleration of LSTM Based on Data for Test Flight
Sun, Z., Zhu, Y., Zheng, Y., Wu, H., Cao, Z., Xiong, P., Hou, J., Huang, T., & Que, Z. (2018) · IEEE International Conference on Smart Cloud (SmartCloud) · doi:10.1109/smartcloud.2018.00009 · Cited by 21
“Different from other hot prediction issues in LSTM model such as speech recognition [3][4], human emotion perception[5] , question answering[6] and traffic monitoring[7], the flight data prediction puts forward requirements in two aspects.”
BackgroundHuang, Z., & Epps, J. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2821135 · Cited by 21
“For arousal and valence ground truth, ratings from 6 annotators on RECOLA and SEMAINE were normalized using the normalization technique [41] before averaging across all annotators, as per the AVEC 2016 challenge [39].”
MethodA Novel Method for Human Bias Correction of Continuous- Time Annotations
Booth, B. M., Mundnich, K., & Narayanan, S. S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461645 · Cited by 20
“Many other feature-based methods have been proposed [6, 7, 8, 9, 10] and perform reasonably well on various data sets, but require an informative set of features to be extracted from the stimuli for alignment and fusion.”
A Novel LSTM Approach for Asynchronous Multivariate Time Series Prediction
Ma, K. F., & Leung, H. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851792 · Cited by 23
“When employing more variables in the model, or combining multiple univariate prediction models [10], the number of possible combination of lags grows combinatorically [11] and it becomes difficult to compute optimal lags.”
MethodMultimodal-multisensor affect detection
D’Mello, S. K., Bosch, N., & Chen, H. (2018) · ACM eBooks · doi:10.1145/3107990.3107998 · Cited by 20
“Ringeval et al. [2015a] fused video, audio, and physiology and showed advantages of LSTMs and BLSTMs compared to feed-forward neural networks (this study is discussed in more detail below).”
BackgroundAn efficient model-level fusion approach for continuous affect recognition from audiovisual signals
Pei, E., Jiang, D., & Hichem Sahli, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.09.037 · Cited by 21
On-the-Fly Facial Expression Prediction Using LSTM Encoded Appearance-Suppressed Dynamics
Baddar, W. J., Lee, S., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2957465 · Cited by 22
An Investigation of Emotion Prediction Uncertainty Using Gaussian Mixture Regression
Dang, T., Sethu, V., Epps, J., & Eliathamby Ambikairajah, A. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-512 · Cited by 19
“Several studies [4-7] have showed the importance of taking information from multiple raters into account for both categorical emotion recognition and continuous emotion prediction systems, or have argued that emotion attributes should be ranked instead of trying to predict absolute ordinal values.”
BackgroundDesign and Implementation of LSTM Accelerator Based on FPGA
Zhang, W., Ge, F., Cui, C., Yang, Y., Zhou, F., & Wu, N. (2020) · IEEE International Conference on Communication Technology (ICCT) · doi:10.1109/icct50939.2020.9295665 · Cited by 19
“LSTM is widely used in text recognition [5], scene analysis [6] and sentiment analysis [7].”
MethodDimensional Affect Recognition from HRV: An Approach Based on Supervised SOM and ELM
Bugnon, L. A., Calvo, R. A., & Milone, D. H. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2763943 · Cited by 17
“Many of the revised works ([57], [58], [59], [61]) were based on the LSTM model, which was introduced for this database in [52] and it is considered a state-of-the-art model for dimensional affect estimation [7], [57].”
BackgroundSupervised Contrastive Learning for Affect Modelling
Pinitas, K., Makantasis, K., Liapis, A., & Yannakakis, G. N. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556584 · Cited by 17
“In particular, the above results show that arousal is better captured in the provided audio, visual, and physiology features obtained by concatenating ECG and EDA features [35] [50] [46].”
BackgroundRETRACTED ARTICLE: Recognition of emotions in speech using deep CNN and RESNET
Lakshmi, K. L., Muthulakshmi, P., Nithya, A. A., Jeyavathana, R. B., Usharani, R., Das, N. S., & Devi, G. N. R. (2023) · Soft Computing · doi:10.1007/s00500-023-07969-5 · Cited by 18
RetractedOutlier Processing in Multimodal Emotion Recognition
Zhang, G., Luo, T., Pedrycz, W., El-Meligy, M. A., Sharaf, M. A. F., & Li, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2981760 · Cited by 17
“In a fusion strategy, the fusion method of feature-level and decision-level is frequently used [22].”
BackgroundMethodInfluentialReal-world automatic continuous affect recognition from audiovisual signals
Tzirakis, P., Zafeiriou, S. P., & Schuller, B. (2019) · Multimodal Behavior Analysis in the Wild · doi:10.1016/b978-0-12-814601-9.00028-6 · Cited by 20
Self-citationJi, X., Sun, L., & Huang, K. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101379 · Cited by 15
Analyzing the expression of annoyance during phone calls to complaint services
Irastorza, J., & Torres, M. I. (2016) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom.2016.7804533 · Cited by 15
“In this approach recurrent neural networks have been proposed to integrate contextual information and then predict emotion in continuous time to just deal with arousal and valence [19] [20].”
BackgroundModeling Emotions in Robotic Socially Believable Behaving Systems
Esposito, A. A., & Jain, L. C. (2016) · Intelligent systems reference library · doi:10.1007/978-3-319-31056-5_2 · Cited by 15
Metadata
- Authors
- Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B.
- Affiliations
- University of Fribourg (CH)
- Technical University of Munich (DE)
- École Polytechnique Fédérale de Lausanne (CH)
- University of Passau (DE)
- Corresponding author
- Ringeval, Fabien
- ORCID
- Venue
- Pattern Recognition Letters
- Publisher
- Elsevier BV
- Published
- 2015-11
- Volume
- 66
- Pages
- 22-30
- ISSN
- 0167-8655 (ISSN-L)
- OpenAlex
- W2005418748
- Cited by
- 202
- References
- 63
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 4 · 2023: 15 · 2022: 14 · 2021: 14 · 2020: 17 · 2019: 25 · 2018: 38 · 2017: 23 · 2016: 30 · 2015: 9
- FWCI
- 26.12
- Type
- journal article
- Language
- en
- Open access
- green
- infoscience.epfl.ch
- Locations
- Pattern Recognition Letters (publishedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- OPUS (Augsburg University) (submittedVersion, cc-by-nc-nd)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Seventh Framework Programme
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- automatic emotion recognition, arousal, valence, long short-term memory, recurrent neural networks, multimodal fusion, feature-level fusion, decision fusion, audiovisual information, physiological signals, ECG, electrodermal activity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
