Toward machine emotional intelligence: analysis of affective physiological state
Abstract
The ability to recognize emotion is one of the hallmarks of emotional intelligence, an aspect of human intelligence that has been argued to be even more important than mathematical and verbal intelligences. This paper proposes that machine intelligence needs to include emotional intelligence and demonstrates results toward this goal: developing a machine's ability to recognize the human affective state given four physiological signals. We describe difficult issues unique to obtaining reliable affective data and collect a large set of data from a subject trying to elicit and experience each of eight emotional states, daily, over multiple weeks. This paper presents and compares multiple algorithms for feature-based recognition of emotional state from this data. We analyze four physiological signals that exhibit problematic day-to-day variations: The features of different emotions on the same day tend to cluster more tightly than do the features of the same emotion on different days. To handle the daily variations, we propose new features and algorithms and compare their performance. We find that the technique of seeding a Fisher Projection with the results of sequential floating forward search improves the performance of the Fisher Projection and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Psychophysiological measures contain information about affective states.
“Psychophysiological measures were also shown to contain information about affective states [8].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Salovey, P., & Mayer, J. D. (1990) · Imagination, Cognition and Personality · doi:10.2190/dugg-p24e-52wk-6cdg · Cited by 10,484 · see citing works
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382 · see citing works
Self-citationMeasuring facial expressions by computer image analysis
Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999) · Psychophysiology · doi:10.1017/s0048577299971664 · Cited by 424 · see citing works
Facial emotion recognition using multi-modal information
De Silva, L. C., Miyasato, T., & Nakatsu, R. (2002) · Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat. No.97TH8237) · doi:10.1109/icics.1997.647126 · Cited by 319 · see citing works
Multimodal human emotion/expression recognition
Chen, L. S., Huang, T. S., Miyasato, T., & Nakatsu, R. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670976 · Cited by 183 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Detecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207 · see citing works
Self-citationMauss, I. B., & Robinson, M. D. (2009) · Cognition and Emotion · doi:10.1080/02699930802204677 · Cited by 2,078 · see citing works
Barrett, L. F. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00003.x · Cited by 1,859 · see citing works
Affective computing: challenges
Picard, R. W. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00052-1 · Cited by 978 · see citing works
Self-citationSmart wearable systems: Current status and future challenges
Chan, M., Estève, D., Fourniols, J.-Y., Escriba, C., & Campo, E. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.09.003 · Cited by 953 · see citing works
Emotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · see citing works
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works
Fundamentals of physiological computing
Fairclough, S. H. (2009) · Interacting with Computers · doi:10.1016/j.intcom.2008.10.011 · Cited by 608 · see citing works
Cardiovascular, electrodermal, and respiratory response patterns to fear‐ and sadness‐inducing films
Kreibig, S. D., Wilhelm, F. H., Roth, W. T., & Gross, J. J. (2007) · Psychophysiology · doi:10.1111/j.1469-8986.2007.00550.x · Cited by 495 · see citing works
Ecg pattern analysis for emotion detection
Agrafioti, F., Hatzinakos, D., & Anderson, A. K. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.28 · Cited by 476 · see citing works
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · see citing works
Short-term emotion assessment in a recall paradigm
Chanel, G., Kierkels, J. J. M., Soleymani, M., & Pun, T. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.005 · Cited by 367 · see citing works
Gunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · see citing works
Emotional Design in Digital Media for Learning
Plass, J. L., & Kaplan, U. (2016) · Emotions, Technology, Design, and Learning · doi:10.1016/b978-0-12-801856-9.00007-4 · Cited by 377 · see citing works
Empirically Building and Evaluating a Probabilistic Model of User Affect
Conati, C., & Maclaren, H. (2009) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-009-9062-8 · Cited by 328 · see citing works
Barsade, S. G., & Knight, A. P. (2015) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-032414-111316 · Cited by 309 · see citing works
Activity-Aware Mental Stress Detection Using Physiological Sensors
Sun, F.-T., Kuo, C., Cheng, H.-T., Buthpitiya, S., Collins, P., & Griss, M. (2012) · Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-642-29336-8_12 · Cited by 292 · see citing works
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287 · see citing works
Using psycho-physiological measures to assess task difficulty in software development
Fritz, T., Begel, A., Müller, S. C., Yigit-Elliott, S., & Züger, M. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568266 · Cited by 262 · see citing works
Petrantonakis, P. C., & Hadjileontiadis, L. J. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.7 · Cited by 260 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Affective multimodal human-computer interaction
Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101299 · Cited by 234 · see citing works
Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.11.007 · Cited by 202 · see citing works
Multimodal emotion recognition from expressive faces, body gestures and speech
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K. (2007) · IFIP The International Federation for Information Processing · doi:10.1007/978-0-387-74161-1_41 · Cited by 183 · see citing works
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-019-00505-7 · Cited by 182 · see citing works
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171 · see citing works
Frantzidis, C. A., Bratsas, C., Klados, M. A., Konstantinidis, E., Lithari, C. D., Vivas, A. B., Papadelis, C. L., Kaldoudi, E., Pappas, C., & Bamidis, P. D. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2038481 · Cited by 166 · see citing works
Analysis of physiological signals for recognition of boredom, pain, and surprise emotions
Jang, E.-H., Park, B.-J., Park, M.-S., Kim, S.-H., & Sohn, J.-H. (2015) · Journal of Physiological Anthropology · doi:10.1186/s40101-015-0063-5 · Cited by 159 · see citing works
Context-Aware QoE Modelling, Measurement, and Prediction in Mobile Computing Systems
Mitra, K., Zaslavsky, A., & Ahlund, C. (2015) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2013.155 · Cited by 144 · see citing works
Self-Supervised Learning for ECG-Based Emotion Recognition
Sarkar, P., & Etemad, A. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053985 · Cited by 146 · see citing works
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · see citing works
Anxiety level recognition for virtual reality therapy system using physiological signals
Šalkevicius, J., Damaševičius, R., Maskeliunas, R., & Laukienė, I. (2019) · Electronics · doi:10.3390/electronics8091039 · Cited by 135 · see citing works
Kragel, P. A., & LaBar, K. S. (2013) · Emotion · doi:10.1037/a0031820 · Cited by 134 · see citing works
Context-aware experience sampling reveals the scale of variation in affective experience
Hoemann, K., Khan, Z., Feldman, M. J., Nielson, C., Devlin, M., Dy, J., Barrett, L. F., Wormwood, J. B., & Quigley, K. S. (2020) · Scientific Reports · doi:10.1038/s41598-020-69180-y · Cited by 124 · see citing works
Using mental load for managing interruptions in physiologically attentive user interfaces
Chen, D., & Vertegaal, R. (2004) · CHI '04 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/985921.986103 · Cited by 120 · see citing works
Kanjo, E., Younis, E. M. G., & Sherkat, N. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.05.005 · Cited by 118 · see citing works
Rickard, N., Arjmand, H.-A., Bakker, D., & Seabrook, E. (2016) · JMIR Mental Health · doi:10.2196/mental.6202 · Cited by 118 · see citing works
“This extends previous research that demonstrates feasibility of such modeling [40,41,71] by utilizing predictors already established in previous research to be associated with mental health (such as online social networking) rather than only those mobile phone sensors that are convenient to record (such as app use and…”
Emotion Recognition through Physiological Signals for Human-Machine Communication
Maaoui, C., & Pruski, A. (2010) · Cutting Edge Robotics 2010 · doi:10.5772/10312 · Cited by 109 · see citing works
“A significant series of work has been conducted by Picard and colleagues at MIT Lab.”
BackgroundMethodInfluentialExplainable AI and Multi-Modal Causability in Medicine
Holzinger, A. (2021) · i-com · doi:10.1515/icom-2020-0024 · Cited by 87 · see citing works
Khezri, M., Firoozabadi, M., & Sharafat, A. R. (2015) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2015.07.006 · Cited by 89 · see citing works
Mentis, A.-F. A., Lee, D., & Roussos, P. (2024) · Molecular Psychiatry · doi:10.1038/s41380-023-02047-6 · Cited by 75 · see citing works
Emotion recognition using Kinect motion capture data of human gaits
Li, S., Cui, L., Zhu, C., Li, B., Zhao, N., & Zhu, T. (2016) · PeerJ · doi:10.7717/peerj.2364 · Cited by 71 · see citing works
Kukolja, D., Popović, S., Horvat, M., Kovač, B., & Ćosić, K. (2014) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2014.05.006 · Cited by 66 · see citing works
Capturing user engagement via psychophysiology: measures and mechanisms for biocybernetic adaptation
Fairclough, S. H., Gilleade, K., Ewing, K. C., & Roberts, J. (2013) · International Journal of Autonomous and Adaptive Communications Systems · doi:10.1504/ijaacs.2013.050694 · Cited by 60 · see citing works
Modeling Quality of IoT Experience in Autonomous Vehicles
Minovski, D., Ahlund, C., & Mitra, K. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2975418 · Cited by 59 · see citing works
Wu, D., Zhang, J., & Zhao, Q. (2020) · IEEE Access · doi:10.1109/access.2020.3010311 · Cited by 51 · see citing works
Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017) · Australasian Physical & Engineering Sciences in Medicine · doi:10.1007/s13246-017-0571-1 · Cited by 46 · see citing works
Sense and Learn: Self-supervision for omnipresent sensors
Saeed, A., Ungureanu, V., & Gfeller, B. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100152 · Cited by 34 · see citing works
Stress and Perception of Emotional Stimuli: Long-Term Stress Rewiring the Brain
Khosrowabadi, R., & Institute for Cognitive and Brain Sciences, S. B. U. (2018) · Basic and Clinical Neuroscience Journal · doi:10.29252/nirp.bcn.9.2.107 · Cited by 39 · see citing works
A Multimodal Dataset for Mixed Emotion Recognition
Yang, P., Liu, N., Liu, X., Shu, Y., Ji, W., Ren, Z., Sheng, J., Yu, M., Yi, R., Zhang, D., et al. (2024) · Scientific Data · doi:10.1038/s41597-024-03676-4 · Cited by 37 · see citing works
Hussain, M. S., Calvo, R. A., & Chen, F. (2014) · Interacting with Computers · doi:10.1093/iwc/iwt032 · Cited by 34 · see citing works
Affect-aware behaviour modelling and control inside an intelligent environment
Leon, E., Clarke, G., Callaghan, V., & Doctor, F. (2010) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2009.12.002 · Cited by 30 · see citing works
Pre-training in Medical Data: A Survey
Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30 · see citing works
On the Generality of Codebook Approach for Sensor-Based Human Activity Recognition
Shirahama, K., & Grzegorzek, M. (2017) · Electronics · doi:10.3390/electronics6020044 · Cited by 29 · see citing works
Singh, R. R., Conjeti, S., & Banerjee, R. (2011) · 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2011.6082900 · Cited by 27 · see citing works
Comparison of Wearable and Clinical Devices for Acquisition of Peripheral Nervous System Signals
Bizzego, A., Gabrieli, G., Furlanello, C., & Esposito, G. (2020) · Sensors · doi:10.3390/s20236778 · Cited by 25 · see citing works
Arruti, A., Cearreta, I., Álvarez, A., Lazkano, E., & Sierra, B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0108975 · Cited by 21 · see citing works
Qu, C., Brinkman, W.-P., Ling, Y., Wiggers, P., & Heynderickx, I. (2013) · Virtual Reality · doi:10.1007/s10055-013-0231-z · Cited by 16 · see citing works
- Authors
- Picard, R. W., Vyzas, E., & Healey, J.
- Affiliations
- MIT Media Lab (US)
- IBM (United States) (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2001
- Volume
- 23
- Issue
- 10
- Pages
- 1175-1191
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2120945046
- Cited by
- 2,334
- References
- 5
- Citations by year
- 2026: 49 · 2025: 138 · 2024: 103 · 2023: 116 · 2022: 99 · 2021: 105 · 2020: 120 · 2019: 147 · 2018: 111 · 2017: 129 · 2016: 112 · 2015: 110 · 2014: 112 · 2013: 126 · 2012: 120
- FWCI
- 26.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- emotion recognition, physiological data collection, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
