Learning deep physiological models of affect
Open accessAbstract
Feature extraction and feature selection are crucial phases in the process of affective modeling. Both, however, incorporate substantial limitations that hinder the development of reliable and accurate models of affect. For the purpose of modeling affect manifested through physiology, this paper builds on recent advances in machine learning with deep learning (DL) approaches. The efficiency of DL algorithms that train artificial neural network models is tested and compared against standard feature extraction and selection approaches followed in the literature. Results on a game data corpus - containing players' physiological signals (i.e., skin conductance and blood volume pulse) and subjective self-reports of affect - reveal that DL outperforms manual ad-hoc feature extraction as it yields significantly more accurate affective models. Moreover, it appears that DL meets and even outperforms affective models that are boosted by automatic feature selection, for several of the scenarios examined. As the DL method is generic and applicable to any affective modeling task, the key findings of the paper suggest that ad-hoc feature extraction and selection - to a lesser degree - could be bypassed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CNNs trained unsupervised via AEs on raw blood volume pulse and skin conductance signals outperformed models based on handcrafted features.
“For this purpose, they used CNNs trained in an unsupervised way via AEs to learn features from raw blood volume pulse and skin conductance signals, which outperformed models based on handcrafted features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [172]
Martinez et al. were the first to learn deep features directly from peripheral physiology.
“Martinez et al. [172] were the first to learn deep features directly from the peripheral physiology.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [172]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Self-citationPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,048 · see citing works
Self-citationApproximate entropy as a measure of system complexity.
Pincus, S. M. (1991) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.88.6.2297 · Cited by 5,811 · see citing works
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · see citing works
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · see citing works
Self-citationConvolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,637 · see citing works
Self-citationFeature Selection for Classification
Dash, M., & Liu, H. (1997) · Intelligent Data Analysis · doi:10.1016/s1088-467x(97)00008-5 · Cited by 3,594 · see citing works
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620 · see citing works
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · see citing works
Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction
Masci, J., Meier, U., Ciresan, D., & Schmidhuber, J. (2011) · International Conference on Artificial Neural Networks · doi:10.1007/978-3-642-21735-7_7 · Cited by 2,185 · see citing works
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1× · see citing works
“The key challenges one faces when attempting to model emotion [2] are inherent in the vague definitions and fuzzy boundaries of emotion, and in the modeling methodology followed.”
Method(1993) · Cited by 1,173 · see citing works
Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier]
Arel, I., Rose, D. C., & Karnowski, T. P. (2010) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2010.938364 · Cited by 1,128 · see citing works
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383157 · Cited by 1,109 · see citing works
Emotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · see citing works
Multimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantić, M., & Pun, T. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 725 · see citing works
Matsugu, M., Mori, K., Mitari, Y., & Kaneda, Y. (2003) · Neural Networks · doi:10.1016/s0893-6080(03)00115-1 · Cited by 715 · see citing works
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 655 · see citing works
Affective Body Expression Perception and Recognition: A Survey
Kleinsmith, A., & Bianchi-Berthouze, N. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.16 · Cited by 646 · see citing works
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592 · see citing works
Wagner, J., Kim, J., & André, E. (2005) · IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2005.1521579 · Cited by 560 · see citing works
Mandryk, R. L., & Atkins, M. S. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.11.011 · Cited by 538 · see citing works
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · Cited by 455 · see citing works
Self-citationMorris, J. D. (1995) · Journal of Advertising Research · Cited by 436 · see citing works
(2013) · doi:10.4324/9780203763247 · Cited by 400 · see citing works
Discriminant analysis of principal components for face recognition
Zhao, W., Chellappa, R., & Krishnaswamy, A. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670971 · Cited by 394 · see citing works
Bi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382 · see citing works
Relationship of Heart Rate Variability to Parasympathetic Effect
Goldberger, J. J., Challapalli, S., Tung, R. H., Parker, M. A., & Kadish, A. H. (2001) · Circulation · doi:10.1161/01.cir.103.15.1977 · Cited by 329 · see citing works
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
El Kaliouby, R., & Robinson, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.427 · Cited by 282 · see citing works
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A. C., Nass, C., & John, O. P. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.10.011 · Cited by 276 · see citing works
Phasic Emotional Reactions to Video Game Events: A Psychophysiological Investigation
Ravaja, N., Saari, T., Salminen, M., Laarni, J., & Kallinen, K. (2006) · Media Psychology · doi:10.1207/s1532785xmep0804_2 · Cited by 263 · see citing works
F�rnkranz, J., & Hllermeier, E. (2010) · Cited by 259 · see citing works
Disentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215 · see citing works
Self-citationCall Center Stress Recognition with Person-Specific Models
Hernandez, J., Morris, R. R., & Picard, R. W. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_16 · Cited by 210 · see citing works
観察 SAM 自己評価マネキン 感情的反応の効果的な異文化間比較測定
Morris, J. D. (1995) · Journal of Advertising Research · Cited by 185 · see citing works
On the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24412-4_3 · Cited by 182 · see citing works
Self-citationComparing SVM and convolutional networks for epileptic seizure prediction from intracranial EEG
Mirowski, P., LeCun, Y., Madhavan, D., & Kuzniecky, R. (2008) · Machine learning for signal processing ... · doi:10.1109/mlsp.2008.4685487 · Cited by 170 · see citing works
Mining features for sequence classification
Lesh, N. B., Zaki, M. J., & Ogihara, M. (1999) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/312129.312275 · Cited by 161 · see citing works
Chen, H., Ng, T. D., Martinez, J., & Schatz, B. R. (1997) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199701)48:1 · Cited by 159 · see citing works
Modeling self-efficacy in intelligent tutoring systems: An inductive approach
McQuiggan, S. W., Mott, B., & Lester, J. C. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9040-y · Cited by 149 · see citing works
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
el Kaliouby, R., & Robinson, P. (2005) · Springer eBooks · doi:10.1007/0-387-27890-7_11 · Cited by 134 · see citing works
Learning to rank with (a lot of) word features
Bai, B., Weston, J., Grangier, D., Collobert, R., Sadamasa, K., Qi, Y., Chapelle, O., & Weinberger, K. Q. (2009) · Information Retrieval · doi:10.1007/s10791-009-9117-9 · Cited by 117 · see citing works
Entertainment modeling through physiology in physical play
Yannakakis, G. N., & Hallam, J. C. T. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2008.06.004 · Cited by 116 · see citing works
Self-citationTowards affective camera control in games
Yannakakis, G. N., Martínez, H. P., & Jhala, A. H. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9078-0 · Cited by 112 · see citing works
Self-citationAutomatic Speech Classification To Five Emotional States Based On Gender Information
Ververidis, D., & Kotropoulos, C. L. (2004) · doi:10.5281/zenodo.38468 · Cited by 105 · see citing works
Entertainment capture through heart rate activity in physical interactive playgrounds
Yannakakis, G. N., Hallam, J. C. T., & Lund, H. H. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9036-7 · Cited by 104 · see citing works
Self-citationUsing Activity-Related Behavioural Features towards More Effective Automatic Stress Detection
Giakoumis, D., Drosou, A., Cipresso, P., Tzovaras, D. K., Hassapis, G. D., Gaggioli, A., & Riva, G. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043571 · Cited by 98 · see citing works
Temporal Pooling And Multiscale Learning For Automatic Annotation And Ranking Of Music Audio.
Hamel, P., Lemieux, S., Bengio, Y., & Douglas Eck, C. (2011) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1418236 · Cited by 96 · see citing works
Self-citationRanking vs. Preference: A Comparative Study of Self-reporting
Yannakakis, G. N., & Hallam, J. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_47 · Cited by 90 · see citing works
Self-citationModeling enjoyment preference from physiological responses in a car racing game
Tognetti, S., Garbarino, M., Bonarini, A., & Matteucci, M. (2010) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/itw.2010.5593337 · Cited by 75 · see citing works
Automatic Recognition of Boredom in Video Games Using Novel Biosignal Moment-Based Features
Giakoumis, D., Tzovaras, D. K., Moustakas, K., & Hassapis, G. D. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.4 · Cited by 70 · see citing works
Classification of EEG for Affect Recognition: An Adaptive Approach
AlZoubi, O., Calvo, R. A., & Stevens, R. H. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-10439-8_6 · Cited by 69 · see citing works
Generic Physiological Features as Predictors of Player Experience
Martínez, H. P., Garbarino, M., & Yannakakis, G. N. (2011) · Affective Computing and Intelligent Interaction · doi:10.1007/978-3-642-24600-5_30 · Cited by 54 · see citing works
Self-citationPredicting Facial Indicators of Confusion with Hidden Markov Models
Grafsgaard, J. F., Boyer, K., & Lester, J. C. (2011) · Affective Computing and Intelligent Interaction · doi:10.1007/978-3-642-24600-5_13 · Cited by 48 · see citing works
Preference learning for affective modeling
Yannakakis, G. N. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349491 · Cited by 46 · see citing works
Self-citationMining multimodal sequential patterns: a case study on affect detection
Martínez, H. P., & Yannakakis, G. N. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070485 · Cited by 42 · see citing works
Self-citationDynamic recognition of basic facial expressions by discrete-time recurrent neural network
Kobayashi, H., & Hara, F. (2005) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.1993.713882 · Cited by 42 · see citing works
Enjoyment recognition from physiological data in a car racing game
Tognetti, S., Garbarino, M., Bonanno, A. T., Matteucci, M., & Bonarini, A. (2010) · International Workshop on Affective Interaction in Natural Environments (AFFINE) · doi:10.1145/1877826.1877830 · Cited by 34 · see citing works
Genetic search feature selection for affective modeling
Martínez, H. P., & Yannakakis, G. N. (2010) · International Workshop on Affective Interaction in Natural Environments (AFFINE) · doi:10.1145/1877826.1877832 · Cited by 21 · see citing works
Self-citationInferring document similarity from hyperlinks
Grangier, D., & Samy Bengio, A. (2005) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1099554.1099666 · Cited by 15 · see citing works
Regularized Auto-Encoders Estimate Local StatisticsPreprint
Alain, G., Bengio, Y., & Rifai, S. (2012) · arXiv (Cornell University) · Cited by 11 · see citing works
Self-citationInterview mit Prof. Dr. Michael M. Richter.
Schneeberger, J. (2005) · Künstliche Intell. · Cited by 6 · see citing works
Detecting human behavior emotional cues in Natural Interaction
Caridakis, G., Asteriadis, S., Karpouzis, K. C., & Kollias, S. (2011) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2011.6004962 · Cited by 4 · see citing works
Dynamic Recognition of Basic Facial Expressions by Discrete-time Recurrent Neural Network
Kobayashi, R. (1993) · Cited by 2 · see citing works
Flagged first, then by citations
5 of the 74 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Zheng, W.-L., & Lu, B.-L. (2015) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2015.2431497 · Cited by 2,477 · see citing works
Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
Cambria, E., & White, B. (2014) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2014.2307227 · Cited by 1,106 · see citing works
A review of emotion recognition using physiological signals
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018) · Sensors · doi:10.3390/s18072074 · Cited by 923 · see citing works
Wearables and the Medical Revolution
Dunn, J., Runge, R., & Snyder, M. (2018) · Personalized Medicine · doi:10.2217/pme-2018-0044 · Cited by 654 · see citing works
A Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 571 · see citing works
Jumping NLP Curves: A Review of Natural Language Processing Research
Cambria, E., & White, B. (2014) · Cited by 455 · see citing works
EEG-based emotion classification using deep belief networks
Zheng, W.-L., Zhu, J.-Y., Peng, Y., & Lu, B.-L. (2014) · 2014 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890166 · Cited by 367 · see citing works
Bota, P. J., Wang, C., Fred, A. L. N., & Silva, H. (2019) · IEEE Access · doi:10.1109/access.2019.2944001 · Cited by 329 · see citing works
Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019) · IEEE Access · doi:10.1109/access.2018.2883213 · Cited by 289 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
CNN-based sensor fusion techniques for multimodal human activity recognition
Münzner, S., Schmidt, P., Reiss, A., Hanselmann, M., Stiefelhagen, R., & Dürichen, R. (2017) · ACM International Symposium on Wearable Computers (ISWC) · doi:10.1145/3123021.3123046 · Cited by 236 · see citing works
Wearable-Based Affect Recognition—A Review
Schmidt, P., Reiss, A., Dürichen, R., & Laerhoven, K. V. (2019) · Sensors · doi:10.3390/s19194079 · Cited by 267 · see citing works
Li, C., Bao, Z., Li, L., & Zhao, Z.-P. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.102185 · Cited by 205 · see citing works
Automatic driver stress level classification using multimodal deep learning
Rastgoo, M. N., Nakisa, B., Maire, F., Rakotonirainy, A., & Chandran, V. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.07.010 · Cited by 189 · see citing works
Virtual Reality for Emotion Elicitation – A Review
Somarathna, R., Bednarz, T. P., & Mohammadi, G. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3181053 · Cited by 180 · see citing works
Automatic Ship Classification from Optical Aerial Images with Convolutional Neural Networks
Gallego, A., Pertusa, A., & Gil, P. (2018) · Remote Sensing · doi:10.3390/rs10040511 · Cited by 172 · see citing works
Deep learning for procedural content generation
Liu, J., Snodgrass, S., Khalifa, A., Risi, S., Yannakakis, G. N., & Togelius, J. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05383-8 · Cited by 174 · see citing works
Self-citationEEG Channel Correlation Based Model for Emotion Recognition
Islam, M. R., Islam, M. M., Rahman, M. M., Mondal, C., Singha, S. K., Ahmad, M., Awal, A., Islam, M. S., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104757 · Cited by 162 · see citing works
Respiration-based emotion recognition with deep learning
Zhang, Q., Chen, X., Zhan, Q., Yang, T., & Xia, S. (2017) · Computers in Industry · doi:10.1016/j.compind.2017.04.005 · Cited by 151 · see citing works
ECG signal classification based on deep CNN and BiLSTM
Cheng, J., Zou, Q., & Zhao, Y. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01736-y · Cited by 158 · see citing works
Gjoreski, M., Gradišek, A., Budna, B., Gams, M., & Poglajen, G. (2020) · IEEE Access · doi:10.1109/access.2020.2968900 · Cited by 150 · see citing works
Movahedi, F., Coyle, J. L., & Sejdić, E. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2727218 · Cited by 132 · see citing works
The Ordinal Nature of Emotions: An Emerging Approach
Yannakakis, G. N., Cowie, R., & Busso, C. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2879512 · Cited by 121 · see citing works
Self-citationOkewu, E., Adewole, P., Misra, S., Maskeliūnas, R., & Damaševičius, R. (2021) · Applied Artificial Intelligence · doi:10.1080/08839514.2021.1922847 · Cited by 137 · see citing works
Lima, A. C. E. S., & de Castro, L. N. (2014) · Neural Networks · doi:10.1016/j.neunet.2014.05.020 · Cited by 117 · see citing works
EOG-based drowsiness detection using convolutional neural networks
Zhu, X., Zheng, W., Lu, B., Chen, X., Chen, S., & Wang, C. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889642 · Cited by 117 · see citing works
Fast Emotion Recognition Based on Single Pulse PPG Signal with Convolutional Neural Network
Lee, M. S., Lee, Y. K., Pae, D.-S., Lim, M. T., Kim, D.-W., & Kang, T. (2019) · Applied Sciences · doi:10.3390/app9163355 · Cited by 114 · see citing works
The ordinal nature of emotions
Yannakakis, G. N., Cowie, R., & Busso, C. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273608 · Cited by 112 · see citing works
Self-citationDon’t Classify Ratings of Affect; Rank Them!
Martínez, H. P., Yannakakis, G. N., & Hallam, J. C. T. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2352268 · Cited by 109 · see citing works
Self-citationDeep neural networks for ultra-short-term wind forecasting
Đalto, M., Matuško, J., & Vašak, M. (2015) · IEEE International Conference on Industrial Technology (ICIT) · doi:10.1109/icit.2015.7125335 · Cited by 108 · see citing works
Federated Self-Supervised Learning of Multisensor Representations for Embedded Intelligence
Saeed, A., Salim, F. D., Ozcelebi, T., & Lukkien, J. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3009358 · Cited by 106 · see citing works
Ganapathy, N., Veeranki, Y. R., & Swaminathan, R. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113571 · Cited by 103 · see citing works
Deep Physiological Affect Network for the Recognition of Human Emotions
Kim, B. H., & Jo, S. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2790939 · Cited by 102 · see citing works
Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101 · see citing works
Rad, N. M., Kia, S. M., Zarbo, C., van Laarhoven, T., Jurman, G., Venuti, P., Marchiori, E., & Furlanello, C. (2017) · Signal Processing · doi:10.1016/j.sigpro.2017.10.011 · Cited by 6 · see citing works
Exploring Deep Physiological Models for Nociceptive Pain Recognition
Thiam, P., Bellmann, P., Kestler, H. A., & Schwenker, F. (2019) · Sensors · doi:10.3390/s19204503 · Cited by 94 · see citing works
Meng, H., Bianchi‐Berthouze, N., Deng, Y. S., Cheng, J., & Cosmas, J. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2418092 · Cited by 94 · see citing works
Federated Self-Supervised Learning of Multisensor Representations for Embedded Intelligence
Saeed, A., Salim, F. D., Özçelebi, T., & Lukkien, J. J. (2024) · Figshare · Cited by 92 · see citing works
A general framework for sensor-based human activity recognition
Köping, L., Shirahama, K., & Grzegorzek, M. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2017.12.025 · Cited by 83 · see citing works
Deep Learning on 1-D Biosignals: a Taxonomy-based Survey
Ganapathy, N., Swaminathan, R., & Deserno, T. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1667083 · Cited by 83 · see citing works
Personalized Emotion Recognition by Personality-Aware High-Order Learning of Physiological Signals
Zhao, S., Gholaminejad, A., Ding, G., Gao, Y., Han, J., & Keutzer, K. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3233184 · Cited by 81 · see citing works
Physiological-signal-based emotion recognition: An odyssey from methodology to philosophy
Li, W., Zhang, Z., & Song, A. (2020) · Measurement · doi:10.1016/j.measurement.2020.108747 · Cited by 79 · see citing works
Lee, M. S., Lee, Y. K., Lim, M. T., & Kang, T. (2020) · Applied Sciences · doi:10.3390/app10103501 · Cited by 77 · see citing works
Recognizing Abnormal Heart Sounds Using Deep LearningPreprint
Rubin, J., Abreu, R., Ganguli, A., Nelaturi, S., Matei, I., & Sricharan, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04642 · Cited by 76 · see citing works
Approaches, Applications, and Challenges in Physiological Emotion Recognition—A Tutorial Overview
Can, Y. S., Mahesh, B., & André, E. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3286445 · Cited by 76 · see citing works
Yang, Y., Gao, Z., Wang, X.-M., Li, Y., Han, J., Marwan, N., & Kurths, J. (2018) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5023857 · Cited by 74 · see citing works
Deep Learning-Based Automatic Detection of Ships: An Experimental Study Using Satellite Images
Patel, K., Bhatt, C., & Mazzeo, P. L. (2022) · Journal of Imaging · doi:10.3390/jimaging8070182 · Cited by 72 · see citing works
Pan, C., Shi, C., Mu, H., Li, J., & Gao, X. (2020) · Applied Sciences · doi:10.3390/app10051619 · Cited by 72 · see citing works
Automated Nociceptive Pain Assessment Using Physiological Signals and a Hybrid Deep Learning Network
Subramaniam, S. D., & Doss, B. P. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3023656 · Cited by 72 · see citing works
Applying Deep Learning to Stereotypical Motor Movement Detection in Autism Spectrum Disorders
Rad, N. M., & Furlanello, C. (2016) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2016.0178 · Cited by 69 · see citing works
Grounding truth via ordinal annotation
Yannakakis, G. N., & Martínez, H. P. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344627 · Cited by 68 · see citing works
Self-citationInvestigating deep learning for fNIRS based BCI
Hennrich, J., Herff, C., Heger, D., & Schultz, T. (2015) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2015.7318984 · Cited by 67 · see citing works
Nath, R. K., Thapliyal, H., & Caban‐Holt, A. (2021) · Journal of Signal Processing Systems · doi:10.1007/s11265-020-01611-5 · Cited by 66 · see citing works
SigRep: Toward Robust Wearable Emotion Recognition With Contrastive Representation Learning
Dissanayake, V., Seneviratne, S., Rana, R. K., Wen, E., Kaluarachchi, T. I., & Nanayakkara, S. C. (2022) · IEEE Access · doi:10.1109/access.2022.3149509 · Cited by 66 · see citing works
The identification of cis-regulatory elements: A review from a machine learning perspective
Li, Y., Chen, C., Kaye, A. M., & Wasserman, W. W. (2015) · Biosystems · doi:10.1016/j.biosystems.2015.10.002 · Cited by 63 · see citing works
Vidyasagar, K. E. C., Kumar, K. R., Sai, G. N. K. A., Ruchita, M., & Saikia, M. J. (2024) · IEEE Access · doi:10.1109/access.2024.3399114 · Cited by 62 · see citing works
Martínez, H. P., & Yannakakis, G. N. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663236 · Cited by 62 · see citing works
Self-citationEsports Athletes and Players: A Comparative Study
Khromov, N., Korotin, A. A., Lange, A., Stepanov, A., Burnaev, E. V., & Somov, A. (2019) · IEEE Pervasive Computing · doi:10.1109/mprv.2019.2926247 · Cited by 60 · see citing works
Dzieżyc, M., Gjoreski, M., Kazienko, P., Saganowski, S., & Gams, M. (2020) · Sensors · doi:10.3390/s20226535 · Cited by 58 · see citing works
The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games
Makantasis, K., Liapis, A., & Yannakakis, G. N. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3060877 · Cited by 59 · see citing works
Self-citationBanville, H. J., & Falk, T. H. (2016) · Brain-Computer Interfaces · doi:10.1080/2326263x.2015.1134958 · Cited by 58 · see citing works
Personality-Aware Personalized Emotion Recognition from Physiological Signals
Zhao, S., Ding, G., Han, J., & Gao, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/230 · Cited by 54 · see citing works
Ho, T. K. K., Gwak, J., Park, C. M., & Song, J. (2019) · IEEE Access · doi:10.1109/access.2019.2900127 · Cited by 54 · see citing works
Learning distributed word representation with multi-contextual mixed embedding
Li, J., Li, J., Fu, X., Masud, M. A., & Huang, J. Z. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.05.045 · Cited by 53 · see citing works
Mcfly: Automated deep learning on time series
van Kuppevelt, D. E., Meijer, C., Huber, F., van der Ploeg, A., Georgievska, S., & van Hees, V. T. (2020) · SoftwareX · doi:10.1016/j.softx.2020.100548 · Cited by 53 · see citing works
Emotion recognition from thermal infrared images using deep Boltzmann machine
Wang, S., He, M., Gao, Z., He, S., & Ji, Q. (2014) · Frontiers of Computer Science · doi:10.1007/s11704-014-3295-3 · Cited by 51 · see citing works
Implicit Analysis of Perceptual Multimedia Experience Based on Physiological Response: A Review
Moon, S.-E., & Lee, J. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2614880 · Cited by 50 · see citing works
The platformer experience dataset
Karpouzis, K. C., Yannakakis, G. N., Shaker, N., & Asteriadis, S. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344647 · Cited by 45 · see citing works
Self-citationYour Gameplay Says It All: Modelling Motivation in Tom Clancy’s The Division
Melhárt, D., Azadvar, A., Canossa, A., Liapis, A., & Yannakakis, G. N. (2019) · IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8848123 · Cited by 46 · see citing works
Self-citationMultimodal Vigilance Estimation Using Deep Learning
Wu, W., Sun, W. J., Wu, Q. M. J., Yang, Y., Zhang, H., Zheng, W., & Lu, B. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3022647 · Cited by 45 · see citing works
From Pixels to Affect: A Study on Games and Player Experience
Makantasis, K., Liapis, A., & Yannakakis, G. N. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925493 · Cited by 42 · see citing works
Self-citationDeep learning framework for detection of hypoglycemic episodes in children with type 1 diabetes
San, P. P., Ling, S. H., & Nguyen, H. T. (2016) · 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2016.7591483 · Cited by 42 · see citing works
Recognition of persisting emotional valence from EEG using convolutional neural networks
Yanagimoto, M., & Sugimoto, C. (2016) · 2016 IEEE 9th International Workshop on Computational Intelligence and Applications (IWCIA) · doi:10.1109/iwcia.2016.7805744 · Cited by 43 · see citing works
Sense and Learn: Self-supervision for omnipresent sensors
Saeed, A., Ungureanu, V., & Gfeller, B. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100152 · Cited by 34 · see citing works
EEG-based driver fatigue detection using hybrid deep generic model
San, P. P., Ling, S. H., Chai, R., Tran, Y., Craig, A. R., & Nguyen, H. T. (2016) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2016.7590822 · Cited by 38 · see citing works
Yannakakis, G. N., Martínez, H. P., & Garbarino, M. (2016) · Socio-affective computing · doi:10.1007/978-3-319-41316-7_7 · Cited by 37 · see citing works
Self-citationAffective Game Computing: A Survey
Yannakakis, G. N., & Melhárt, D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3315689 · Cited by 40 · see citing works
Self-citationAdvanced Ultrasound and Photoacoustic Imaging in Cardiology
Wu, M., Awasthi, N., Rad, N. M., Pluim, J. P. W., & Lopata, R. G. P. (2021) · Sensors · doi:10.3390/s21237947 · Cited by 37 · see citing works
Multimodal Goal Recognition in Open-World Digital Games
Min, W., Mott, B., Rowe, J., Taylor, R., Wiebe, E., Boyer, K. E., & Lester, J. C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v13i1.12939 · Cited by 36 · see citing works
Survey on Feature Extraction and Applications of Biosignals
Supratak, A., Wu, C., Dong, H., Sun, K., & Guo, Y. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-50478-0_8 · Cited by 36 · see citing works
A hybrid intelligent model for acute hypotensive episode prediction with large-scale data
Jiang, D., Tu, G., Jin, D., Wu, K., Liu, C., Zheng, L., & Zhou, T. (2020) · Information Sciences · doi:10.1016/j.ins.2020.08.033 · Cited by 35 · see citing works
Eduardo, A., Aidos, H., & Fred, A. L. N. (2017) · International Conference on Pattern Recognition Applications and Methods · doi:10.5220/0006195404630470 · Cited by 33 · see citing works
Hidden-layer visible deep stacking network optimized by PSO for motor imagery EEG recognition
Tang, X., Zhang, N., Zhou, J., & Liu, Q. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.12.039 · Cited by 32 · see citing works
Emotion Recognition using EEG and Physiological Data for Robot-Assisted Rehabilitation Systems
Gümüşlü, E., Barkana, D. E., & Köse, H. (2020) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3395035.3425199 · Cited by 31 · see citing works
Awasthi, N., Vermeer, L., Fixsen, L. S., Lopata, R. G. P., & Pluim, J. P. W. (2022) · IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control · doi:10.1109/tuffc.2022.3169684 · Cited by 31 · see citing works
Deep Learning Based Affective Model for Speech Emotion Recognition
Zhou, X., Guo, J., & Bie, R. (2016) · 2016 Intl IEEE Conferences on Ubiquitous Intelligence & Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Cloud and Big Data Computing, Internet of People, and Smart World Congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld) · doi:10.1109/uic-atc-scalcom-cbdcom-iop-smartworld.2016.0133 · Cited by 36 · see citing works
On the Generality of Codebook Approach for Sensor-Based Human Activity Recognition
Shirahama, K., & Grzegorzek, M. (2017) · Electronics · doi:10.3390/electronics6020044 · Cited by 29 · see citing works
Emotion recognition techniques using physiological signals and video games –Systematic review–
Callejas-Cuervo, M., Martínez-Tejada, L. A., & Alarcón-Aldana, A. C. (2017) · Revista Facultad de Ingeniería · doi:10.19053/01211129.v26.n46.2017.7310 · Cited by 29 · see citing works
Detecting Expressions with Multimodal Transformers
Parthasarathy, S., & Sundaram, S. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383573 · Cited by 29 · see citing works
EEG-based emotion recognition using discriminative graph regularized extreme learning machine
Zhu, J., Zheng, W., Peng, Y. K., Duan, R.-N., & Lu, B. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889618 · Cited by 29 · see citing works
Stereotypical Motor Movement Detection in Dynamic Feature Space
Rad, N. M., Kia, S. M., Zarbo, C., Jurman, G., Venuti, P., & Furlanello, C. (2016) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2016.0076 · Cited by 29 · see citing works
Rank Pooling Approach for Wearable Sensor-Based ADLs Recognition
Nisar, M. A., Shirahama, K., Li, F., Huang, X., & Grzegorzek, M. (2020) · Sensors · doi:10.3390/s20123463 · Cited by 28 · see citing works
The Value of Cooperation: Minimizing User Costs in Multi-Broker Mobile Cloud Computing Networks
Guan, Z., & Melodia, T. (2015) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2015.2440257 · Cited by 27 · see citing works
A Review on Deep Learning Algorithms for Speech and Facial Emotion Recognition
Latha, C. P., & Priya, M. (2016) · APTIKOM Journal on Computer Science and Information Technologies · doi:10.11591/aptikom.j.csit.118 · Cited by 28 · see citing works
User Evaluation of Affective Dynamic Difficulty Adjustment Based on Physiological Deep Learning
Chanel, G., & Lopes, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50353-6_1 · Cited by 27 · see citing works
Gahlan, N., & Sethia, D. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19467-3 · Cited by 26 · see citing works
Normal Inverse Gaussian Features for EEG-Based Automatic Emotion Recognition
Pusarla, N., Singh, A. K., & Tripathi, S. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3205894 · Cited by 22 · see citing works
Preference Learning for Move Prediction and Evaluation Function Approximation in Othello
Rúnarsson, T. P., & Lucas, S. M. (2014) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2014.2307272 · Cited by 25 · see citing works
Enhanced deep capsule network for EEG-based emotion recognition
ÇİZMECİ, H., & Özcan, C. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-022-02251-x · Cited by 24 · see citing works
- Authors
- Martinez, H. P., Bengio, Y., & Yannakakis, G. N.
- Affiliations
- Université de Montréal (CA)
- University of Malta (MT)
- Venue
- IEEE Computational Intelligence Magazine
- Published
- 2013-05
- Volume
- 8
- Issue
- 2
- Pages
- 20-33
- ISSN
- 1556-603X (ISSN-L)
- OpenAlex
- W2103184652
- Cited by
- 287
- References
- 74
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 18 · 2023: 22 · 2022: 50 · 2021: 23 · 2020: 28 · 2019: 29 · 2018: 32 · 2017: 25 · 2016: 22 · 2015: 17 · 2014: 14 · 2013: 1
- FWCI
- 24.33
- Type
- journal article
- Language
- en
- Open access
- green
- www.um.edu.mt
- Locations
- IEEE Computational Intelligence Magazine (publishedVersion)
- OAR@UM (University of Malta) (submittedVersion, other-oa)
- Funders
- Canada Research Chairs
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
- Keywords
- emotion modeling, physiological signals, deep learning, feature extraction, feature selection, skin conductance, blood volume pulse, self-reported affect, game data, artificial neural networks, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
