Multiple kernel learning for emotion recognition in the wild
Abstract
We propose a method to automatically detect emotions in unconstrained settings as part of the 2013 Emotion Recognition in the Wild Challenge [16], organized in conjunction with the ACM International Conference on Multimodal Interaction (ICMI 2013). Our method combines multiple visual descriptors with paralinguistic audio features for multimodal classification of video clips. Extracted features are combined using Multiple Kernel Learning and the clips are classified using an SVM into one of the seven emotion categories: Anger, Disgust, Fear, Happiness, Neutral, Sadness and Surprise. The proposed method achieves competitive results, with an accuracy gain of approximately 10% above the challenge baseline.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Because of the subtleness of facial configurations, most predesigned representations are dynamically extracting the appearance.
“Because of the subtleness of facial configurations most predesigned representations are dynamically extracting the appearance [262], [263], [264], [265].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [263]
Cites
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · see citing works
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
Least Squares Support Vector Machine Classifiers
Suykens, J., & Vandewalle, J. (1999) · Neural Processing Letters · doi:10.1023/a:1018628609742 · Cited by 9,481 · see citing works
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · see citing works
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,281 · see citing works
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Locality-constrained Linear Coding for image classification
Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., & Gong, Y. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540018 · Cited by 3,100 · see citing works
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846 · see citing works
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
Vedaldi, A., & Fulkerson, B. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1874249 · Cited by 2,675 · see citing works
Coding facial expressions with Gabor wavelets
Lyons, M. J., Akamatsu, S., Kamachi, M., & Gyoba, J. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670949 · Cited by 2,214 · see citing works
Face detection, pose estimation, and landmark localization in the wild
Zhu, X., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248014 · Cited by 2,171 · see citing works
Advances in Neural Information Processing Systems 18
Murray, I., Mackay, D., Zoubin Ghahramani, A., & Skilling, J. (2006) · Cited by 1,927 · see citing works
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & la Torre, F. D. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.75 · Cited by 1,917 · see citing works
Advances in Neural Information Processing Systems 21
Griffiths, T. L., Lucas, C. G., Williams, J. J., & Kalish, M. L. (2009) · neural information processing systems · Cited by 1,854 · see citing works
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., Blitzer, J. C., & Saul, L. K. (2005) · Cited by 1,693 · see citing works
Active Appearance Models Revisited
Matthews, I. A., & Baker, S. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029666.37597.d3 · Cited by 1,562 · see citing works
Chapter 2 Building the gist of a scene: the role of global image features in recognition
Oliva, A., & Torralba, A. B. (2006) · Progress in brain research · doi:10.1016/s0079-6123(06)55002-2 · Cited by 1,441 · see citing works
Representing shape with a spatial pyramid kernel
Bosch, A., Zisserman, A., & Muñoz, X. (2007) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1282280.1282340 · Cited by 1,407 · see citing works
Web-Based Database for Facial Expression Analysis
Pantić, M., Valstar, M., Rademaker, R. M., & Maat, L. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521424 · Cited by 1,099 · see citing works
Blur Insensitive Texture Classification Using Local Phase Quantization
Ojansivu, V., & Heikkilä, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69905-7_27 · Cited by 1,043 · see citing works
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & B, T. J. S. A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.799905 · Cited by 999 · see citing works
Self-citationFacial and vocal expressions of emotion
Russell, J. A., Bachorowski, J.-A., & Fernández-Dols, J.-M. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145102 · Cited by 873 · see citing works
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · see citing works
The devil is in the details: an evaluation of recent feature encoding methods
Chatfield, K., Lempitsky, V., Vedaldi, A., & Zisserman, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.76 · Cited by 834 · see citing works
Feature Detection and Tracking with Constrained Local Models
Cristinacce, D., & Cootes, T. F. (2006) · British Machine Vision Conference (BMVC) · doi:10.5244/c.20.95 · Cited by 736 · see citing works
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 664 · see citing works
The INTERSPEECH 2010 paralinguistic challenge
Schuller, B. W., Steidl, S., Batliner, A. M., Burkhardt, F., Devillers, L. Y., Müller, C. A., & Narayanan, S. S. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-739 · Cited by 577 · see citing works
More generality in efficient multiple kernel learning
Varma, M., & Babu, B. R. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553510 · Cited by 400 · see citing works
Fully Automatic Facial Action Unit Detection and Temporal Analysis
Valstar, M., & Pantić, M. (2006) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2006.85 · Cited by 349 · see citing works
Facial emotion recognition using multi-modal information
De Silva, L. C., Miyasato, T., & Nakatsu, R. (2002) · Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat. No.97TH8237) · doi:10.1109/icics.1997.647126 · Cited by 319 · see citing works
Emotion recognition using PHOG and LPQ features
Dhall, A., Asthana, A., Goecke, R., & Gedeon, T. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771366 · Cited by 221 · see citing works
Emotion recognition in the wild challenge 2013
Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531739 · Cited by 178 · see citing works
Computing Spatio-Temporal Representations of Human Faces
Yacoob, Y., & Davis, L. S. (1994) · Cited by 158 · see citing works
Multimodal approaches for emotion recognition: a survey
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2005) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.600746 · Cited by 154 · see citing works
Pattern Recognition (ICPR), 2012 21st International Conference on
Beyan, C., & Fisher, B. M. (2012) · Cited by 143 · see citing works
Computing spatio-temporal representations of human faces
Yacoob, & Davis (1994) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1994.323812 · Cited by 141 · see citing works
Emotion Analysis in Man-Machine Interaction Systems
Balomenos, T., Raouzaiou, A. T., Ioannou, S. V., Drosopoulos, A., Karpouzis, K. C., & Kollias, S. (2005) · Lecture notes in computer science · doi:10.1007/978-3-540-30568-2_27 · Cited by 126 · see citing works
Huang, T. S., & Chen, L. S.-H. (2000) · Cited by 108 · see citing works
AAM Derived Face Representations for Robust Facial Action Recognition
Lucey, S., Matthews, I. A., Hu, C., Ambadar, Z., la Torre, F. D., & Cohn, J. F. (2006) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fgr.2006.17 · Cited by 106 · see citing works
Exploring Bag of Words Architectures in the Facial Expression Domain
Sikka, K., Wu, T., Susskind, J., & Bartlett, M. S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33868-7_25 · Cited by 94 · see citing works
Self-citationAdding the affective dimension: a new look in speech analysis and synthesis.
Scherer, K. R. (1996) · Cited by 89 · see citing works
Learning non-redundant codebooks for classifying complex objects
Zhang, W., Surve, A. S., Fern, X. Z., & Dietterich, T. G. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553533 · Cited by 71 · see citing works
Volume Local Phase Quantization for Blur-Insensitive Dynamic Texture Classification
Päivärinta, J., Rahtu, E., & Heikkilä, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21227-7_34 · Cited by 70 · see citing works
A framework for modeling appearance change in image sequences
Black, M. J., Fleet, D. J., & Yacoob, Y. (2002) · International Conference on Computer Vision · doi:10.1109/iccv.1998.710788 · Cited by 67 · see citing works
Weakly supervised pain localization using multiple instance learning
Sikka, K., Dhall, A., & Bartlett, M. S. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553762 · Cited by 62 · see citing works
Self-citationLOW-LEVEL FUSION OF AUDIO AND VIDEO FEATURE FOR MULTI-MODAL EMOTION RECOGNITION
Wimmer, M., Schuller, B. W., Arsić, D., Rigoll, G., & Radig, B. (2008) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0001082801450151 · Cited by 52 · see citing works
Facial action recognition in face profile image sequences
Pantić, M., Patras, I., & Rothkruntz, L. (2003) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2002.1035712 · Cited by 46 · see citing works
Bucak, S. S., Jin, R., & Jain, A. K. (2010) · Cited by 33 · see citing works
Analysis of high-level features for vocal emotion recognition
Atassi, H., Esposito, A. A., & Smékal, Z. (2011) · International Conference on Telecommunications and Signal Processing (TSP) · doi:10.1109/tsp.2011.6043708 · Cited by 19 · see citing works
BAG OF WORDS FOR LARGE SCALE OBJECT RECOGNITION - Properties and Benchmark
Aly, M., Munich, M. E., & Perona, P. (2011) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0003311402990306 · Cited by 12 · see citing works
Audio-Visual Data Association for Face Expression Analysis
Tawari, A., & Trivedi, M. M. (2013) · Cited by 4 · see citing works
Valstar, M., Pantić, M., & Patras, I. (2005) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305 · see citing works
Video-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235 · see citing works
The more the merrier: Analysing the affect of a group of people in images
Dhall, A., Joshi, J., Sikka, K., Goecke, R., & Sebe, N. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163151 · Cited by 93 · see citing works
Self-citationFacial expression recognition in the wild based on multimodal texture features
Sun, B., Li, L., Zhou, G., & He, J. (2016) · Journal of Electronic Imaging · doi:10.1117/1.jei.25.6.061407 · Cited by 74 · see citing works
EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge
Palash, M., & Bhargava, B. (2023) · arXiv · doi:10.48550/arxiv.2306.08657 · Cited by 25 · see citing works
“In [27], authors combined multiple visual and audio features for automatic emotion recognition using kernel and Support Vector Machines (SVM) methods.”
Background
- Authors
- Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M.
- Affiliations
- University of California San Diego (US)
- Venue
- Proceedings of the 15th ACM on International conference on multimodal interaction
- Publisher
- ACM
- Published
- 2013-12-09
- Pages
- 517-524
- OpenAlex
- W2143492886
- Semantic Scholar
- 13afc4f8d08f766479577db2083f9632544c7ea6
- Cited by
- 119
- References
- 55
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 5 · 2023: 6 · 2022: 21 · 2021: 9 · 2020: 12 · 2019: 9 · 2018: 8 · 2017: 3 · 2016: 15 · 2015: 16 · 2014: 7
- FWCI
- 17.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- multiple kernel learning, image descriptors, video clips, support vector machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
