Affective multimodal human-computer interaction
Abstract
Social and emotional intelligence are aspects of human intelligence that have been argued to be better predictors than IQ for measuring aspects of success in life, especially in social interactions, learning, and adapting to what is important. When it comes to machines, not all of them will need such skills. Yet to have machines like computers, broadcast systems, and cars, capable of adapting to their users and of anticipating their wishes, endowing them with the ability to recognize user's affective states is necessary. This article discusses the components of human affect, how they might be integrated into computers, and how far are we from realizing affective multimodal human-computer interaction.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous facial emotion recognition software primarily dealt with offline emotion recognition that causes post-processing of the learner's data.
“Previous software primarily dealt with offline emotion recognition that cause post-processing of the learner's data.”
From Towards multimodal emotion recognition in e-learning environments (2016) · cites this paper as Pantic, Sebe, Cohn, & Huang, 2005
Previous facial emotion recognition software can only manage a small set of expressions from frontal view faces without facial hair or glasses, provided that there is constant illumination.
“The application context is restricted by the fact that such software can only manage a small set of expressions from frontal view faces without facial hair, glasses provided that there is constant illumination.”
From Towards multimodal emotion recognition in e-learning environments (2016) · cites this paper as Pantic, Sebe, Cohn, & Huang, 2005
Cites
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382 · see citing works
The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding
EKMAN, P., & FRIESEN, W. V. (1969) · Semiotica · doi:10.1515/semi.1969.1.1.49 · Cited by 3,316 · see citing works
Lewis, M., & Haviland, J. M. (1993) · Cited by 3,308 · see citing works
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,612 · see citing works
Self-citationToward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · see citing works
Acoustic profiles in vocal emotion expression.
Banse, R., & Scherer, K. R. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.3.614 · Cited by 2,253 · see citing works
A Survey on Visual Surveillance of Object Motion and Behaviors
Hu, W., Tan, T. N., Wang, L., & Maybank, S. J. (2004) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2004.829274 · Cited by 2,132 · see citing works
Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.
Ambady, N., & Rosenthal, R. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.111.2.256 · Cited by 1,925 · see citing works
Russell, J. A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.102 · Cited by 1,640 · see citing works
Watson, D., Weber, K., Assenheimer, J. S., Clark, L. A., Strauss, M. E., & McCormick, R. A. (1995) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.104.1.3 · Cited by 1,423 · see citing works
Watson, D., Weber, K., Assenheimer, J. S., Clark, L. A., Strauss, M. E., & McCormick, R. A. (1995) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.104.1.3 · Cited by 1,423 · see citing works
Web-Based Database for Facial Expression Analysis
Pantić, M., Valstar, M., Rademaker, R. M., & Maat, L. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521424 · Cited by 1,099 · see citing works
Watson, D., Clark, L. A., Weber, K., Assenheimer, J. S., Strauss, M. E., & McCormick, R. A. (1995) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.104.1.15 · Cited by 1,002 · see citing works
Speech emotion recognition using hidden Markov models
Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003) · Speech Communication · doi:10.1016/s0167-6393(03)00099-2 · Cited by 920 · see citing works
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874 · see citing works
Self-citationToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791 · see citing works
Watson, D., Clark, L. A., Weber, K., et al. (1995) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.104.1.15 · Cited by 745 · see citing works
Emotional Intelligence. New York (Bantam) 1995.
Goleman, D. (1995) · Cited by 666 · see citing works
The Psychology of Facial Expression
Mandler, G. (1997) · Cambridge University Press eBooks · doi:10.1017/cbo9780511659911 · Cited by 656 · see citing works
Cultural similarities and differences in display rules
Matsumoto, D. (1990) · Motivation and Emotion · doi:10.1007/bf00995569 · Cited by 652 · see citing works
Keltner, D. J., Ekman, P., Gonzaga, G. C., & Beer, J. S. (2002) · doi:10.1093/oso/9780195126013.003.0022 · Cited by 555 · see citing works
Handbook of methods in nonverbal behavior research
Scherer, K. R., & Ekman, P. (1982) · Cited by 515 · see citing works
Behavioral markers and recognizability of the smile of enjoyment.
Frank, M. G., Ekman, P., & Friesen, W. V. (1993) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.64.1.83 · Cited by 508 · see citing works
Human-centered design considered harmful
Norman, D. A. (2005) · interactions · doi:10.1145/1070960.1070976 · Cited by 487 · see citing works
Behavioral Markers and Recognizability of the Smile of Enjoyment
Frank, M. G., Ekman, P., & Friesen, W. V. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.003.0011 · Cited by 412 · see citing works
Looking at people: sensing for ubiquitous and wearable computing
Pentland, A. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.824823 · Cited by 403 · see citing works
The New Handbook of Methods in Nonverbal Behavior Research
Harrigan, J. A., Rosenthal, R., & Scherer, K. R. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198529620.001.0001 · Cited by 359 · see citing works
Juslin, P. N., & Scherer, K. R. (2005) · doi:10.1093/oso/9780198529613.003.0003 · Cited by 318 · see citing works
THE TIMING OF FACIAL MOTION IN POSED AND SPONTANEOUS SMILES
Cohn, J. F., & Schmidt, K. L. (2004) · International Journal of Wavelets Multiresolution and Information Processing · doi:10.1142/s021969130400041x · Cited by 301 · see citing works
Self-citationSocially aware, computation and communication
Pentlan, A. (2005) · Computer · doi:10.1109/mc.2005.104 · Cited by 255 · see citing works
Audio-visual integration in multimodal communication
Chen, T., & Rao, R. R. (1998) · Proceedings of the IEEE · doi:10.1109/5.664274 · Cited by 252 · see citing works
Has the Internet become indispensable?
Hoffman, D. L., Novak, T., & Venkatesh, A. (2004) · Communications of the ACM · doi:10.1145/1005817.1005818 · Cited by 229 · see citing works
De Silva, L. C., & Ng, P. C. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840655 · Cited by 196 · see citing works
The new ethology of human facial expressions
Fridlund, A. J. (1997) · Cambridge University Press eBooks · doi:10.1017/cbo9780511659911.007 · Cited by 194 · see citing works
Yoshitomi, Y., Kim, S.-I., Kawano, T., & Kilazoe, T. (2002) · Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication. IEEE RO-MAN 2000 (Cat. No.00TH8499) · doi:10.1109/roman.2000.892491 · Cited by 144 · see citing works
Emotion Analysis in Man-Machine Interaction Systems
Balomenos, T., Raouzaiou, A. T., Ioannou, S. V., Drosopoulos, A., Karpouzis, K. C., & Kollias, S. (2005) · Lecture notes in computer science · doi:10.1007/978-3-540-30568-2_27 · Cited by 126 · see citing works
Audio-visual based emotion recognition-a new approach
Song, M., Bu, J., Chen, C., & Li, N. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315276 · Cited by 126 · see citing works
Huang, T. S., & Chen, L. S.-H. (2000) · Cited by 108 · see citing works
Self-citationEmbodied contextual agent in information delivering application
Pélachaud, C., Carofiglio, V., Carolis, B. N. D., de Rosis, F., & Poggi, I. (2002) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/544862.544921 · Cited by 105 · see citing works
Authentic facial expression analysis
Sebe, N., Lew, M. S., Cohen, I., Sun, Y., Gevers, T., & Huang, T. S. (2004) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2004.1301585 · Cited by 101 · see citing works
Self-citationEmotional expressions in audiovisual human computer interaction
Chen, L., & Huang, T. S. (2002) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2000.869630 · Cited by 83 · see citing works
Self-citationUser-centered modeling and evaluation of multimodal interfaces
Oviatt, S. L. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817127 · Cited by 83 · see citing works
Emotion recognition from the facial image and speech signal
Go, H.-J., Kwak, K.-C., Lee, D.-J., & Chun, M.-G. (2003) · Cited by 63 · see citing works
Bimodal expression of emotion by face and voice
Cohn, J. F., & Katz, G. S. (1998) · doi:10.1145/306668.306683 · Cited by 54 · see citing works
Self-citationBoosted learning in dynamic bayesian networks for multimodal speaker detection
Garg, A., Pavlović, V., & Rehg, J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817119 · Cited by 51 · see citing works
Audio-visual speaker detection using dynamic Bayesian networks
Garg, A., Pavlović, V., & Rehg, J. M. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840663 · Cited by 42 · see citing works
Feature analysis for automatic speechreading
Scanlon, P., & Reilly, R. B. (2002) · IEEE Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2001.962802 · Cited by 41 · see citing works
Wierzbicka, A. (1993) · Pragmatics & Cognition · doi:10.1075/pc.1.1.03wie · Cited by 31 · see citing works
Case-based reasoning for user-profiled recognition of emotions from face images
Pantić, M., & Rothkrantz, L. J. M. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2004.1394211 · Cited by 23 · see citing works
Exploiting the dependencies in information fusion
Pan, H., Liang, Z., & Huang, T. S. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1999.784713 · Cited by 21 · see citing works
Self-citationWierzbicka, A. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511521256.004 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
Self-citationAffective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,445 · see citing works
Facial Expression Analysis under Partial Occlusion
Zhang, L., Verma, B., Tjondronegoro, D., & Chandran, V. (2018) · ACM Computing Surveys · doi:10.1145/3158369 · Cited by 1,291 · see citing works
Multimodal human–computer interaction: A survey
Jaimes, A., & Sebe, N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.019 · Cited by 965 · see citing works
Self-citationPantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580 · see citing works
A high-resolution 3D dynamic facial expression database
Yin, L., Chen, X., Sun, Y., Worm, T., & Reale, M. (2008) · 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition · doi:10.1109/afgr.2008.4813324 · Cited by 488 · see citing works
Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409 · see citing works
Bi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382 · see citing works
Emotion recognition using a hierarchical binary decision tree approach
Lee, C.-C., Mower, E., Busso, C., Lee, S., & Narayanan, S. S. (2011) · Speech Communication · doi:10.1016/j.specom.2011.06.004 · Cited by 359 · see citing works
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319 · see citing works
Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech
Castellano, G., Kessous, L., & Caridakis, G. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85099-1_8 · Cited by 267 · see citing works
Lopatovska, I., & Arapakis, I. (2010) · Information Processing & Management · doi:10.1016/j.ipm.2010.09.001 · Cited by 261 · see citing works
Kessous, L., Castellano, G., & Caridakis, G. (2010) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0025-5 · Cited by 256 · see citing works
Frantzidιs, C. A., Bratsas, C., Papadelis, C. L., Konstantinidis, E. I., Pappas, C., & Bamidis, P. D. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2010.2041553 · Cited by 250 · see citing works
Machine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Takeo Kanade, A. (2005) · Lecture notes in computer science · doi:10.1007/11564386_1 · Cited by 226 · see citing works
Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study
Abdollahi, H., Mahoor, M. H., Zandie, R., Siewierski, J., & Qualls, S. H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3143803 · Cited by 209 · see citing works
Gunes, H., & Piccardi, M. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.39 · Cited by 206 · see citing works
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · see citing works
Self-citationHow to distinguish posed from spontaneous smiles using geometric features
Valstar, M., Güneş, H., & Pantić, M. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322202 · Cited by 201 · see citing works
Multimodal emotion recognition from expressive faces, body gestures and speech
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K. (2007) · IFIP The International Federation for Information Processing · doi:10.1007/978-0-387-74161-1_41 · Cited by 183 · see citing works
AVEC 2018 Workshop and Challenge
Ringeval, F., Schuller, B. W., Valstar, M., Cowie, R., Kaya, H., Schmitt, M., Amiriparian, S., Cummins, N., Lalanne, D., Michaud, A., et al. (2018) · doi:10.1145/3266302.3266316 · Cited by 149 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Self-citationTowards multimodal emotion recognition in e-learning environments
Bahreini, K., Nadolski, R., & Westera, W. (2016) · Interactive Learning Environments · doi:10.1080/10494820.2014.908927 · Cited by 137 · see citing works
Deep learning for video classification and captioning
Wu, Z., Yao, T., Fu, Y., & Jiang, Y.-G. (2017) · doi:10.1145/3122865.3122867 · Cited by 137 · see citing works
Artificial intelligence moving serious gaming: Presenting reusable game AI components
Westera, W., Prada, R., Mascarenhas, S., Santos, P. A., Dias, J., Guimarães, M., Georgiadis, K., Nyamsuren, E., Bahreini, K., Yumak, Z., et al. (2020) · Education and Information Technologies · doi:10.1007/s10639-019-09968-2 · Cited by 136 · see citing works
Audio–Visual Affective Expression Recognition Through Multistream Fused HMM
Zeng, Z., Tu, J., Pianfetti, B., & Huang, T. S. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921737 · Cited by 134 · see citing works
Self-citationModeling naturalistic affective states via facial and vocal expressions recognition
Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Raouzaiou, A. T., & Karpouzis, K. C. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181029 · Cited by 133 · see citing works
Interpreting ambiguous emotional expressions
Mower, E., Metallinou, A., Lee, C.-C., Kazemzadeh, A., Busso, C., Lee, S., & Narayanan, S. S. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349500 · Cited by 127 · see citing works
AffectButton: A method for reliable and valid affective self-report
Broekens, J., & Brinkman, W.-P. (2013) · Int. J. Hum. Comput. Stud. · doi:10.1016/j.ijhcs.2013.02.003 · Cited by 141 · see citing works
Beyond Facial Expressions: Learning Human Emotion from Body Gestures
Shan, C., Gong, S., & McOwan, P. W. (2007) · British Machine Vision Conference (BMVC) · doi:10.5244/c.21.43 · Cited by 125 · see citing works
A Multimodal Emotion Detection System during Human–Robot Interaction
Alonso-Martín, F., Malfáz, M., Sequeira, J. S., Gorostiza, J. F., & Salichs, M. A. (2013) · Sensors · doi:10.3390/s131115549 · Cited by 117 · see citing works
Multimodal Human Computer Interaction: A Survey
Jaimes, A., & Sebe, N. (2005) · Lecture notes in computer science · doi:10.1007/11573425_1 · Cited by 115 · see citing works
Self-citationZhang, Y., Song, D., Li, X., Zhang, P., Wang, P., Rong, L., Yu, G., & Wang, B. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.04.003 · Cited by 114 · see citing works
Glenn, T., & Monteith, S. (2014) · Current Psychiatry Reports · doi:10.1007/s11920-014-0523-3 · Cited by 104 · see citing works
Machine analysis of facial behaviour: naturalistic and dynamic behaviour
Pantić, M. (2009) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2009.0135 · Cited by 103 · see citing works
Foundations of human computing
Cohn, J. F. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181043 · Cited by 102 · see citing works
Self-citationModeling Latent Discriminative Dynamic of Multi-dimensional Affective Signals
Ramírez, G. A., Baltrušaitis, T., & Morency, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_51 · Cited by 91 · see citing works
Han, J., Zhang, Z., Cummins, N., & Schuller, B. W. (2019) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2019.2901088 · Cited by 86 · see citing works
Appearance Manifold of Facial Expression
Shan, C., Gong, S., & McOwan, P. W. (2005) · Lecture notes in computer science · doi:10.1007/11573425_22 · Cited by 86 · see citing works
Arapakis, I., Konstas, I., & Jose, J. M. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631336 · Cited by 78 · see citing works
Multimodal deep representation learning for video classification
Tian, H., Tao, Y., Pouyanfar, S., Chen, S., & Shyu, M. (2018) · World Wide Web · doi:10.1007/s11280-018-0548-3 · Cited by 77 · see citing works
Foundations of Human Computing: Facial Expression and Emotion
Cohn, J. F. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_1 · Cited by 73 · see citing works
Self-citationBatliner, A. M., Seppi, D., Steidl, S., & Schuller, B. W. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/782802 · Cited by 72 · see citing works
Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition
Karpouzis, K. C., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A. T., Malatesta, L., & Kollias, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_5 · Cited by 72 · see citing works
Emotion Recognition by Fusing Time Synchronous and Time Asynchronous Representations
Wu, W., Zhang, C., & Woodland, P. C. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414880 · Cited by 70 · see citing works
Analyzing Free-standing Conversational Groups
Alameda-Pineda, X., Yan, Y., Ricci, E., Lanz, O., & Sebe, N. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806238 · Cited by 68 · see citing works
Self-citationTemporal Segmentation of Facial Behavior
la Torre, F. D., Campoy, J., Ambadar, Z., & Cohn, J. F. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408961 · Cited by 66 · see citing works
Self-citationKotti, M., & Paternò, F. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-012-9127-7 · Cited by 74 · see citing works
Altun, H., & Polat, G. (2008) · Expert Systems with Applications · doi:10.1016/j.eswa.2008.10.005 · Cited by 64 · see citing works
MultiSense—Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case
Stratou, G., & Morency, L. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2614300 · Cited by 56 · see citing works
A computational model of empathy for interactive agents
Yalcin, O. N., & DiPaola, S. (2018) · Biologically Inspired Cognitive Architectures · doi:10.1016/j.bica.2018.07.010 · Cited by 71 · see citing works
Multimodal interfaces: Challenges and perspectives
Sebe, N. (2009) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-2009-0003 · Cited by 57 · see citing works
Self-citationZhu, Y., la Torre, F. D., Cohn, J. F., & Zhang, Y. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.10 · Cited by 53 · see citing works
Self-citationEmotion recognition using facial expressions with active appearance models
Ratliff, M. S., & Patterson, E. K. (2008) · Cited by 53 · see citing works
The emerging roles of performance within HCI and interaction design
Macaulay, C., Jacucci, G., O’Neill, S. J., Kankaineen, T., & Simpson, M. (2006) · Interacting with Computers · doi:10.1016/j.intcom.2006.07.001 · Cited by 52 · see citing works
User and context adaptive neural networks for emotion recognition
Caridakis, G., Karpouzis, K. C., & Kollias, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.043 · Cited by 51 · see citing works
Audio-Visual Spontaneous Emotion Recognition
Zeng, Z., Hu, Y., Roisman, G. I., Wen, Z., Fu, Y., & Huang, T. S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_4 · Cited by 51 · see citing works
Self-citationEz-Zaouia, M., & Lavoué, É. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3027434 · Cited by 50 · see citing works
Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
Praveen, R. G., & Alam, M. J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00483 · Cited by 49 · see citing works
A fuzzy logic approach to reliable real-time recognition of facial emotions
Bahreini, K., van der Vegt, W., & Westera, W. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7250-z · Cited by 48 · see citing works
Audio-visual emotion recognition in adult attachment interview
Zeng, Z., Hu, Y., Fu, Y., Huang, T. S., Roisman, G. I., & Wen, Z. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181028 · Cited by 46 · see citing works
Self-citationA comparison of general vs personalised affective models for the prediction of topical relevance
Arapakis, I., Αθανασάκος, Κ., & Jose, J. M. (2010) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1835449.1835512 · Cited by 44 · see citing works
Sun, Y., Reale, M. J., & Yin, L. (2008) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2008.4813336 · Cited by 44 · see citing works
Gaze-X: Adaptive, Affective, Multimodal Interface for Single-User Office Scenarios
Maat, L., & Pantic, M. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-72348-6_13 · Cited by 44 · see citing works
Image-based memes as sentiment predictors
French, J. H. (2017) · International Conference on Information Society · doi:10.23919/i-society.2017.8354676 · Cited by 42 · see citing works
Context-Aware Automated Analysis and Annotation of Social Human--Agent Interactions
Baur, T., Mehlmann, G., Damian, I., Lingenfelser, F., Wagner, J., Lugrin, B., André, E., & Gebhard, P. (2015) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2764921 · Cited by 40 · see citing works
Sesin, A. (2008) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2007.05.0075 · Cited by 39 · see citing works
Sentiment analysis of COVID-19 cases in Greece using Twitter data
Samaras, L., García‐Barriocanal, E., & Sicilia, M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120577 · Cited by 38 · see citing works
Multimodal user’s affective state analysis in naturalistic interaction
Caridakis, G., Karpouzis, K. C., Wallace, M., Kessous, L., & Amir, N. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0030-8 · Cited by 38 · see citing works
Driver frustration detection from audio and video in the wild
Abdić, I., Fridman, L., McDuff, D., Marchi, E., Reimer, B., & Schuller, B. W. (2016) · OPUS (Augsburg University) · Cited by 35 · see citing works
Multimodal Transformer with Learnable Frontend and Self Attention for Emotion Recognition
Dutta, S., & Ganapathy, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747723 · Cited by 34 · see citing works
Joint Multi-Scale Multimodal Transformer for Emotion Using Consumer Devices
Khan, M., Ahmad, J., Gueaieb, W., Masi, G., Karray, F. O., & Saddik, A. E. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3532322 · Cited by 33 · see citing works
Mourão, A., & Magalhães, J. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502115 · Cited by 33 · see citing works
Guo, Y., Siddiqui, F., Zhao, Y., Chellappa, R., & Lo, S.-Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02495-3 · Cited by 32 · see citing works
Building a three-level multimodal emotion recognition framework
Garcia-Garcia, J. M., Lozano, M. D., Penichet, V. M. R., & Law, E. L. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13254-8 · Cited by 32 · see citing works
Maat, L., & Pantić, M. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181032 · Cited by 32 · see citing works
Multimodal Affect Recognition: Current Approaches and Challenges
Osman, H. A., & Falk, T. (2017) · InTech eBooks · doi:10.5772/65683 · Cited by 29 · see citing works
LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations
Dutta, S., & Ganapathy, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889998 · Cited by 30 · see citing works
Recognizing Facial Expressions Automatically from Video
Shan, C., & Braspenning, R. (2009) · Springer eBooks · doi:10.1007/978-0-387-93808-0_18 · Cited by 30 · see citing works
Dynamic cascades with bidirectional bootstrapping for spontaneous facial action unit detection
Zhu, Y., la Torre, F. D., Cohn, J. F., & Zhang, Y. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349603 · Cited by 30 · see citing works
Self-citationA Report on the First International Workshop on Research Methods in Animal-Computer Interaction
Zamansky, A., Roshier, M., Mancini, C., Collins, E. C., Hall, C., Grillaert, K., Morrison, A., North, S., & Wirman, H. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3052759 · Cited by 28 · see citing works
Multimodal Emotion Recognition from Low-Level Cues
Pantić, M., Caridakis, G., André, E., Kim, J., Karpouzis, K. C., & Kollias, S. (2010) · Cognitive technologies · doi:10.1007/978-3-642-15184-2_8 · Cited by 28 · see citing works
Zhang, L., Tjondronegoro, D., & Chandran, V. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.09.005 · Cited by 27 · see citing works
Guthier, B., Abaalkhail, R., Alharthi, R., & Saddik, A. E. (2015) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/iccnc.2015.7069418 · Cited by 26 · see citing works
Chen, H., Zeng, W., Chen, C., Cai, L., Wang, F., Shi, Y., Wang, L., Zhang, W., Li, Y., Yan, H., et al. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107848 · Cited by 26 · see citing works
Frontiers of Multimedia Research
Chang, S.-F. (2017) · doi:10.1145/3122865 · Cited by 26 · see citing works
Melder, W. A., Truong, K. P., den Uyl, M. J., van Leeuwen, D. A., Neerincx, M. A., Loos, L. R., & Plum, B. S. (2007) · doi:10.1145/1290128.1290134 · Cited by 24 · see citing works
Red Hen Lab: Dataset and Tools for Multimodal Human Communication Research
Joo, J., Steen, F. F., & Turner, M. B. (2017) · Künstliche Intell. · doi:10.1007/s13218-017-0505-9 · Cited by 24 · see citing works
Vision-Based Game Interface Using Human Gesture
Park, H. S., Jung, D. J., & Kim, H. J. (2006) · Lecture notes in computer science · doi:10.1007/11949534_66 · Cited by 24 · see citing works
M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5492 · Cited by 23 · see citing works
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398 · Cited by 23 · see citing works
Braun, M., Pfleging, B., & Alt, F. (2018) · Multimodal Technologies and Interaction · doi:10.3390/mti2040075 · Cited by 23 · see citing works
Zhang, L., Tjondronegoro, D., Chandran, V., & Eggink, J. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-2497-5 · Cited by 22 · see citing works
Incongruity-Aware Cross-Modal Attention for Audio-Visual Fusion in Dimensional Emotion Recognition
Praveen, R. G., & Alam, J. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3422823 · Cited by 21 · see citing works
Empirical analysis of multiple modalities for emotion recognition using convolutional neural network
Jaswal, R. A., & Dhingra, S. (2023) · Measurement Sensors · doi:10.1016/j.measen.2023.100716 · Cited by 20 · see citing works
Pantić, M. (2006) · Lecture notes in computer science · doi:10.1007/11825890_2 · Cited by 20 · see citing works
Evaluating the Emotional State of a User Using a Webcam
Magdin, M., Turčáni, M., & Hudec, L. (2016) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2016.4112 · Cited by 19 · see citing works
Valstar, M. (2019) · ACM eBooks · doi:10.1145/3233795.3233807 · Cited by 18 · see citing works
- Authors
- Pantic, M., Sebe, N., Cohn, J. F., & Huang, T.
- Affiliations
- Delft University of Technology, The Netherlands
- University of Amsterdam (NL)
- University of Pittsburgh (US)
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the 13th annual ACM international conference on Multimedia
- Publisher
- ACM
- Published
- 2005-11-06
- Pages
- 669-676
- OpenAlex
- W2028975938
- Cited by
- 234
- References
- 50
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 8 · 2023: 8 · 2022: 23 · 2021: 6 · 2020: 10 · 2019: 9 · 2018: 11 · 2017: 9 · 2016: 7 · 2015: 12 · 2014: 6 · 2013: 14 · 2012: 8
- FWCI
- 25.50
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Institutional Research Information System (Università degli Studi di Trento) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- affective computing, multimodal human-computer interaction, affective states, social-emotional intelligence, user modeling, adaptive systems, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
