The DOI matches its Crossref record; 69 of the 74 references it cites were found in a registry or an index.
Learning deep physiological models of affect
Open accessAbstract
Feature extraction and feature selection are crucial phases in the process of affective modeling. Both, however, incorporate substantial limitations that hinder the development of reliable and accurate models of affect. For the purpose of modeling affect manifested through physiology, this paper builds on recent advances in machine learning with deep learning (DL) approaches. The efficiency of DL algorithms that train artificial neural network models is tested and compared against standard feature extraction and selection approaches followed in the literature. Results on a game data corpus - containing players' physiological signals (i.e., skin conductance and blood volume pulse) and subjective self-reports of affect - reveal that DL outperforms manual ad-hoc feature extraction as it yields significantly more accurate affective models. Moreover, it appears that DL meets and even outperforms affective models that are boosted by automatic feature selection, for several of the scenarios examined. As the DL method is generic and applicable to any affective modeling task, the key findings of the paper suggest that ad-hoc feature extraction and selection - to a lesser degree - could be bypassed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For this purpose, they used CNNs trained in an unsupervised way via AEs to learn features from raw blood volume pulse and skin conductance signals, which outperformed models based on handcrafted features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [172]
In the citing paper
“Martinez et al. [172] were the first to learn deep features directly from the peripheral physiology.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [172]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,330
Self-citationApproximate entropy as a measure of system complexity.
Pincus, S. M. (1991) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.88.6.2297 · Cited by 5,811
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationConvolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,646
Self-citationFeature Selection for Classification
Dash, M., & Liu, H. (1997) · Intelligent Data Analysis · doi:10.1016/s1088-467x(97)00008-5 · Cited by 3,594
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334
Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction
Masci, J., Meier, U., Ciresan, D. C., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21735-7_7 · Cited by 2,137
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1×
“The key challenges one faces when attempting to model emotion [2] are inherent in the vague definitions and fuzzy boundaries of emotion, and in the modeling methodology followed.”
MethodDeep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier]
Arel, I., Rose, D. C., & Karnowski, T. P. (2010) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2010.938364 · Cited by 1,252
(1993) · Cited by 1,173
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383157 · Cited by 1,130
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 988
Emotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944
Multimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantić, M., & Pun, T. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 740
Matsugu, M., Mori, K., Mitari, Y., & Kaneda, Y. (2003) · Neural Networks · doi:10.1016/s0893-6080(03)00115-1 · Cited by 730
Affective Body Expression Perception and Recognition: A Survey
Kleinsmith, A., & Bianchi‐Berthouze, N. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.16 · Cited by 624
Backpropagation: Theory, architectures, and applications
Biron, P. V. (1997) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199701)48:1 · Cited by 620
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
Wagner, J., Kim, J., & André, E. (2005) · IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2005.1521579 · Cited by 560
Mandryk, R. L., & Atkins, M. S. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.11.011 · Cited by 538
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · Cited by 455
Self-citationMorris, J. D. (1995) · Journal of Advertising Research · Cited by 436
(2013) · doi:10.4324/9780203763247 · Cited by 400
Discriminant analysis of principal components for face recognition
Zhao, W., Chellappa, R., & Krishnaswamy, A. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670971 · Cited by 394
Bi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382
Relationship of Heart Rate Variability to Parasympathetic Effect
Goldberger, J. J., Challapalli, S., Tung, R. H., Parker, M. A., & Kadish, A. H. (2001) · Circulation · doi:10.1161/01.cir.103.15.1977 · Cited by 340
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
El Kaliouby, R., & Robinson, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.427 · Cited by 282
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A. C., Nass, C., & John, O. P. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.10.011 · Cited by 276
Phasic Emotional Reactions to Video Game Events: A Psychophysiological Investigation
Ravaja, N., Saari, T., Salminen, M., Laarni, J., & Kallinen, K. (2006) · Media Psychology · doi:10.1207/s1532785xmep0804_2 · Cited by 263
F�rnkranz, J., & Hllermeier, E. (2010) · Cited by 259
Disentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215
Self-citationCall Center Stress Recognition with Person-Specific Models
Hernandez, J., Morris, R. R., & Picard, R. W. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_16 · Cited by 210
観察 SAM 自己評価マネキン 感情的反応の効果的な異文化間比較測定
Morris, J. D. (1995) · Journal of Advertising Research · Cited by 185
On the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24412-4_3 · Cited by 182
Self-citationComparing SVM and convolutional networks for epileptic seizure prediction from intracranial EEG
Mirowski, P., LeCun, Y., Madhavan, D., & Kuzniecky, R. (2008) · Machine learning for signal processing ... · doi:10.1109/mlsp.2008.4685487 · Cited by 172
Mining features for sequence classification
Lesh, N. B., Zaki, M. J., & Ogihara, M. (1999) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/312129.312275 · Cited by 161
Modeling self-efficacy in intelligent tutoring systems: An inductive approach
McQuiggan, S. W., Mott, B., & Lester, J. C. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9040-y · Cited by 155
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
el Kaliouby, R., & Robinson, P. (2005) · Springer eBooks · doi:10.1007/0-387-27890-7_11 · Cited by 140
Learning to rank with (a lot of) word features
Bai, B., Weston, J., Grangier, D., Collobert, R., Sadamasa, K., Qi, Y., Chapelle, O., & Weinberger, K. Q. (2009) · Information Retrieval · doi:10.1007/s10791-009-9117-9 · Cited by 118
Entertainment modeling through physiology in physical play
Yannakakis, G. N., & Hallam, J. C. T. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2008.06.004 · Cited by 116
Self-citationTowards affective camera control in games
Yannakakis, G. N., Martínez, H. P., & Jhala, A. H. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9078-0 · Cited by 113
Self-citationAutomatic Speech Classification To Five Emotional States Based On Gender Information
Ververidis, D., & Kotropoulos, C. L. (2004) · doi:10.5281/zenodo.38468 · Cited by 105
Entertainment capture through heart rate activity in physical interactive playgrounds
Yannakakis, G. N., Hallam, J. C. T., & Lund, H. H. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9036-7 · Cited by 104
Self-citationUsing Activity-Related Behavioural Features towards More Effective Automatic Stress Detection
Giakoumis, D., Drosou, A., Cipresso, P., Tzovaras, D. K., Hassapis, G. D., Gaggioli, A., & Riva, G. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043571 · Cited by 101
Temporal Pooling And Multiscale Learning For Automatic Annotation And Ranking Of Music Audio.
Hamel, P., Lemieux, S., Bengio, Y., & Eck, D. (2011) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1418236 · Cited by 94
Self-citationRanking vs. Preference: A Comparative Study of Self-reporting
Yannakakis, G. N., & Hallam, J. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_47 · Cited by 90
Self-citationModeling enjoyment preference from physiological responses in a car racing game
Tognetti, S., Garbarino, M., Bonarini, A., & Matteucci, M. (2010) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/itw.2010.5593337 · Cited by 75
Automatic Recognition of Boredom in Video Games Using Novel Biosignal Moment-Based Features
Giakoumis, D., Tzovaras, D. K., Moustakas, K., & Hassapis, G. D. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.4 · Cited by 70
Classification of EEG for Affect Recognition: An Adaptive Approach
AlZoubi, O., Calvo, R. A., & Stevens, R. H. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-10439-8_6 · Cited by 69
Generic Physiological Features as Predictors of Player Experience
Martínez, H. P., Garbarino, M., & Yannakakis, G. N. (2011) · Affective Computing and Intelligent Interaction · doi:10.1007/978-3-642-24600-5_30 · Cited by 54
Self-citationPredicting Facial Indicators of Confusion with Hidden Markov Models
Grafsgaard, J. F., Boyer, K., & Lester, J. C. (2011) · Affective Computing and Intelligent Interaction · doi:10.1007/978-3-642-24600-5_13 · Cited by 48
Preference learning for affective modeling
Yannakakis, G. N. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349491 · Cited by 46
Self-citationMining multimodal sequential patterns: a case study on affect detection
Martínez, H. P., & Yannakakis, G. N. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070485 · Cited by 42
Self-citationDynamic recognition of basic facial expressions by discrete-time recurrent neural network
Kobayashi, H., & Hara, F. (2005) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.1993.713882 · Cited by 42
Enjoyment recognition from physiological data in a car racing game
Tognetti, S., Garbarino, M., Bonanno, A. T., Matteucci, M., & Bonarini, A. (2010) · International Workshop on Affective Interaction in Natural Environments (AFFINE) · doi:10.1145/1877826.1877830 · Cited by 34
Genetic search feature selection for affective modeling
Martínez, H. P., & Yannakakis, G. N. (2010) · International Workshop on Affective Interaction in Natural Environments (AFFINE) · doi:10.1145/1877826.1877832 · Cited by 21
Self-citationInferring document similarity from hyperlinks
Grangier, D., & Samy Bengio, A. (2005) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1099554.1099666 · Cited by 15
Regularized Auto-Encoders Estimate Local StatisticsPreprint
Alain, G., Bengio, Y., & Rifai, S. (2012) · arXiv (Cornell University) · Cited by 11
Self-citationInterview mit Prof. Dr. Michael M. Richter.
Schneeberger, J. (2005) · Künstliche Intell. · Cited by 6
Detecting human behavior emotional cues in Natural Interaction
Caridakis, G., Asteriadis, S., Karpouzis, K. C., & Kollias, S. (2011) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2011.6004962 · Cited by 4
Dynamic Recognition of Basic Facial Expressions by Discrete-time Recurrent Neural Network
Kobayashi, R. (1993) · Cited by 2
Research works printed but not linked: 6
- susskind, 2010, The Toronto face dataset, Tech Rep TR- 2010-001
- martínez, 2010, Extending neuro-evolution preference learning through player modeling, Proc IEEE Conf Computational Intelligence and Games, 313
- ekman, 1988, Facial action coding system: A technique for the measurement of facial movement, From Appraisal to Emotion D ifferences Among Unpleasant Feelings Motivation and Emotion, 12, 271
- b engio, 2007, Greedy layerwise training of deep networks, Advances in neural information processing systems, 19, 153
- v yzas, 1998, Affective pattern classification, Proc AAAI Fall Symp Series Emotional and Intelligent The Tangled Knot of Cognition, 176
- 10.1016/j.ijhcs.2008.06.004 · DOI
Cited by
Sorted by citations
Zheng, W.-L., & Lu, B.-L. (2015) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2015.2431497 · Cited by 2,477
Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
Cambria, E., & White, B. (2014) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2014.2307227 · Cited by 1,399
A review of emotion recognition using physiological signals
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018) · Sensors · doi:10.3390/s18072074 · Cited by 923
Wearables and the Medical Revolution
Dunn, J., Runge, R., & Snyder, M. (2018) · Personalized Medicine · doi:10.2217/pme-2018-0044 · Cited by 654
A Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 571
Metadata
- Authors
- Martinez, H. P., Bengio, Y., & Yannakakis, G. N.
- Affiliations
- Université de Montréal (CA)
- University of Malta (MT)
- Venue
- IEEE Computational Intelligence Magazine
- Published
- 2013-05
- Volume
- 8
- Issue
- 2
- Pages
- 20-33
- ISSN
- 1556-603X (ISSN-L)
- OpenAlex
- W2103184652
- Cited by
- 287
- References
- 74
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 18 · 2023: 22 · 2022: 50 · 2021: 23 · 2020: 28 · 2019: 29 · 2018: 32 · 2017: 25 · 2016: 22 · 2015: 17 · 2014: 14 · 2013: 1
- FWCI
- 24.33
- Type
- journal article
- Language
- en
- Open access
- green
- www.um.edu.mt
- Locations
- IEEE Computational Intelligence Magazine (publishedVersion)
- OAR@UM (University of Malta) (submittedVersion, other-oa)
- Funders
- Canada Research Chairs
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
- Keywords
- emotion modeling, physiological signals, deep learning, feature extraction, feature selection, skin conductance, blood volume pulse, self-reported affect, game data, artificial neural networks, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
