Emotion Recognition Based on Joint Visual and Audio Cues
Abstract
Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. However, one necessary ingredient for natural interaction is still missing - emotions. This paper describes the problem of bimodal emotion recognition and advocates the use of probabilistic graphical models when fusing the different modalities. We test our audio-visual emotion recognition approach on 38 subjects with 11 HCI-related affect states. The experimental results show that the average person-dependent emotion recognition accuracy is greatly improved when both visual and audio information are used in classification.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879 · see citing works
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874 · see citing works
Self-citationToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791 · see citing works
Mechrabian, A. (1968) · Medical Entomology and Zoology · Cited by 401 · see citing works
Affective multimodal human-computer interaction
Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101299 · Cited by 234 · see citing works
Self-citationMULTIMODAL EMOTION RECOGNITION
Sebe, N., Cohen, I., & Huang, T. S. (2005) · Handbook of Pattern Recognition and Computer Vision · doi:10.1142/9789812775320_0021 · Cited by 201 · see citing works
Self-citationDe Silva, L. C., & Ng, P. C. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840655 · Cited by 196 · see citing works
Yoshitomi, Y., Kim, S.-I., Kawano, T., & Kilazoe, T. (2002) · Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication. IEEE RO-MAN 2000 (Cat. No.00TH8499) · doi:10.1109/roman.2000.892491 · Cited by 144 · see citing works
Audio-visual based emotion recognition-a new approach
Song, M., Bu, J., Chen, C., & Li, N. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315276 · Cited by 126 · see citing works
Modeling Multimodal Expression of User’s Affective Subjective Experience
Bianchi‐Berthouze, N., & Lisetti, C. (2002) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1013365332180 · Cited by 96 · see citing works
Emotional expressions in audiovisual human computer interaction
Chen, L., & Huang, T. S. (2002) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2000.869630 · Cited by 83 · see citing works
Self-citationConnected vibrations: a modal analysis approach for non-rigid motion tracking
Hai, T., & Huang, T. S. (2002) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1998.698685 · Cited by 74 · see citing works
Self-citationBimodal HCI-related affect recognition
Zeng, Z., Tu, J., Liu, M., Zhang, T., Rizzolo, N., Zhang, Z., Huang, T. S., Roth, D., & Levinson, S. E. (2004) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1027933.1027958 · Cited by 63 · see citing works
Self-citationBoosted learning in dynamic bayesian networks for multimodal speaker detection
Garg, A., Pavlović, V., & Rehg, J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817119 · Cited by 51 · see citing works
Greenhaus, J. H., & A.Callanan, G. (2006) · Encyclopedia of Career Development · doi:10.4135/9781412952675.n92 · Cited by 15 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · see citing works
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Self-citationError weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121 · see citing works
A hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98 · see citing works
Soladié, C., Salam, H., Pelachaud, C., Stoiber, N., & Séguier, R. (2012) · Proceedings of the 14th ACM international conference on Multimodal interaction · doi:10.1145/2388676.2388782 · Cited by 33 · see citing works
User Preference and Categories for Error Responses in Conversational User Interfaces
Yuan, S., Brüggemeier, B., Hillmann, S., & Michael, T. (2020) · Proceedings of the 2nd Conference on Conversational User Interfaces · doi:10.1145/3405755.3406126 · Cited by 16 · see citing works
- Authors
- Sebe, N., Cohen, I., Gevers, T., & Huang, T. S.
- Affiliations
- University of Amsterdam (NL)
- Hewlett-Packard (United States) (US)
- University of Illinois Urbana-Champaign (US)
- Venue
- 18th International Conference on Pattern Recognition (ICPR'06)
- Publisher
- IEEE
- Published
- 2006
- Pages
- 1136-1139
- ISSN
- 1041-3278 (ISSN-L)
- OpenAlex
- W2142653084
- Cited by
- 202
- References
- 15
- Citations by year
- 2025: 1 · 2024: 7 · 2023: 6 · 2022: 22 · 2021: 12 · 2020: 8 · 2019: 11 · 2018: 9 · 2017: 17 · 2016: 12 · 2015: 18 · 2014: 9 · 2013: 11 · 2012: 13
- FWCI
- 19.02
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Iris (University of Trento) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- emotion recognition, audiovisual integration, multimodal fusion, probabilistic graphical models, human-computer interaction, affective computing, affective states
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
