The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition
Abstract
Facial expression is central to human experience, but most previous databases and studies are limited to posed facial behavior under controlled conditions. In this paper, we present a novel facial expression database, Real-world Affective Face Database (RAF-DB), which contains approximately 30 000 facial images with uncontrolled poses and illumination from thousands of individuals of diverse ages and races. During the crowdsourcing annotation, each image is independently labeled by approximately 40 annotators. An expectation-maximization algorithm is developed to reliably estimate the emotion labels, which reveals that real-world faces often express compound or even mixture emotions. A cross-database study between RAF-DB and CK+ database further indicates that the action units of real-world emotions are much more diverse than, or even deviate from, those of laboratory-controlled emotions. To address the recognition of multi-modal expressions in the wild, we propose a new deep locality-preserving convolutional neural network (DLP-CNN) method that aims to enhance the discriminative power of deep features by preserving the locality closeness while maximizing the inter-class scatter. Benchmark experiments on 7-class basic expressions and 11-class compound expressions, as well as additional experiments on CK+, MMI, and SFEW 2.0 databases, show that the proposed DLP-CNN outperforms the state-of-the-art handcrafted features and deep learning-based methods for expression recognition in the wild. To promote further study, we have made the RAF database, benchmarks, and descriptor encodings publicly available to the research community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Eigenfaces vs. Fisherfaces: recognition using class specific linear projection
Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598228 · Cited by 11,707
The expression of the emotions in man and animals.
Darwin, C. (1872) · J. Murray eBooks · doi:10.1037/10001-000 · Cited by 11,657
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
What is Twitter, a social network or a news media?
Kwak, H., Lee, C., Park, H., & Moon, S. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772751 · Cited by 6,652
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
Locality Preserving Projections
He, X., & Niyogi, P. (2003) · Cited by 4,119
A Discriminative Feature Learning Approach for Deep Face Recognition
Wen, Y., Zhang, K., Li, Z., & Qiao, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_31 · Cited by 4,077
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · cited 2×
“The deep feature x ∈ Rd denotes deep convolutional activation features (DeCaf) [13] taken from the final hidden layer, i.”
BackgroundMethodCaffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 3,752
Ekman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
Martínez, A. M. (1998) · Medical Entomology and Zoology · Cited by 2,830
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Presentation and validation of the Radboud Faces Database
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & van Knippenberg, A. (2010) · Cognition and Emotion · doi:10.1080/02699930903485076 · Cited by 2,420
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Coding facial expressions with Gabor wavelets
Lyons, M., Akamatsu, S., Kamachi, M., & Gyoba, J. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670949 · Cited by 2,210
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & De la Torre, F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.75 · Cited by 1,896
Liu, C., & Wechsler, H. (2002) · IEEE Transactions on Image Processing · doi:10.1109/tip.2002.999679 · Cited by 1,886
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1×
“AUTOMATIC facial expression recognition has made significant progress in the past two decades [56], [78].”
BackgroundUniversals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
Russell, J. A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.102 · Cited by 1,640 · cited 1×
“The basic emotional expressions are therefore not universal enough to generalize expressions displayed on the human face [61].”
BackgroundGross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2010) · Image and Vision Computing · doi:10.1016/j.imavis.2009.08.002 · Cited by 1,621
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild
Li, S., Deng, W., & Du, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.277 · Cited by 1,470
Self-citationA 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
YouTube: online video and participatory culture
(2010) · Choice Reviews Online · doi:10.5860/choice.47-2989 · Cited by 1,131
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477450 · Cited by 1,086
Automatic classification of single facial images
Lyons, M. J., Budynek, J., & Akamatsu, S. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.817413 · Cited by 1,075
Web-Based Database for Facial Expression Analysis
Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005) · 2005 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2005.1521424 · Cited by 1,039
Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
Whitehill, J., Wu, T., Bergsma, J., Movellan, J. R., & Ruvolo, P. (2009) · Cited by 1,004
An experimental comparison of performance measures for classification
Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.010 · Cited by 889
Compound facial expressions of emotion
Du, S., Tao, Y., & Martinez, A. M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1322355111 · Cited by 806
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · cited 1×
“In DTAGN [30], two different models were combined to extract temporal appearance and geometry features simultaneously.”
MethodBP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database
Zhang, X., Yin, L., Cohn, J. F., Canavan, S., Reale, M., Horowitz, A., Liu, P., & Girard, J. M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.002 · Cited by 745
The Expression Of The Emotions In Man And Animals
Darwin, C. (1998) · doi:10.1093/oso/9780195112719.001.0001 · Cited by 732
Deep Learning using Linear Support Vector Machines
Tang, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0239 · Cited by 708
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657 · cited 4×
““SFEW best [33]”, “SFEW second [77]” and “SFEW third [54]” indicate the best single model result of the winner, the runner-up and the second runner-up in EmotiW 2015, respectively.”
BackgroundMethodInfluentialFacial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.233 · Cited by 650
Benitez-Quiroz, C. F., Srinivasan, R., & Martinez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.600 · Cited by 607
Image based Static Facial Expression Recognition with Multiple Deep Network Learning
Yu, Z., & Zhang, C. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830595 · Cited by 598
(2017) · The SAGE Encyclopedia of Communication Research Methods · doi:10.4135/9781483381411.n178 · Cited by 559
A high-resolution 3D dynamic facial expression database
Yin, L., Chen, X., Sun, Y., Worm, T., & Reale, M. (2008) · 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition · doi:10.1109/afgr.2008.4813324 · Cited by 488
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 412
Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition
Liu, M., Shan, S., Wang, R., & Chen, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.226 · Cited by 392
Combining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391
Learning active facial patches for expression analysis
Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., & Metaxas, D. N. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247974 · Cited by 369
Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns
Levi, G., & Hassner, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830587 · Cited by 336
Face Expression Recognition and Analysis: The State of the Art
Bettadapura, V. (2012) · arXiv · doi:10.48550/arxiv.1203.6722 · Cited by 336
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16817-3_10 · Cited by 329
Self-citationA Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition
Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., & Yan, K. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2598092 · Cited by 314
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305
Kim, B.-K., Roh, J., Dong, S.-Y., & Lee, S.-Y. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0209-0 · Cited by 243
YouTube: Online Video and Participatory Culture
(2010) · Online Information Review · doi:10.1108/14684521011037052 · Cited by 235
McDuff, D., el Kaliouby, R., Senechal, T., Amr, M., Cohn, J. F., & Picard, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.130 · Cited by 234
AU-aware Deep Networks for facial expression recognition
Liu, M., Li, S., Shan, S., & Chen, X. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553734 · Cited by 218
Self-citationPoell, T., & Borra, E. (2012) · Journalism · doi:10.1177/1464884911431533 · Cited by 215
Manifold based analysis of facial expression
Chang, Y., Hu, C., Feris, R., & Turk, M. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.08.006 · Cited by 215
Introducing the Geneva Multimodal Emotion Portrayal (GEMEP) corpus
Bänziger, & Scherer, T. A. (2010) · Cited by 209
Eleftheriadis, S., Rudovic, O., & Pantic, M. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2375634 · Cited by 206
Flickr and public image-sharing: distant closeness and photo exhibition
Van House, N. A. (2007) · CHI '07 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1240866.1241068 · Cited by 181
Emotion recognition in the wild challenge 2013
Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531739 · Cited by 178
Haar Features for FACS AU Recognition
Whitehill, J., & Omlin, C. W. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.61 · Cited by 162
Robust Representation and Recognition of Facial Emotions Using Extreme Sparse Learning
Shojaeilangari, S., Yau, W., Nandakumar, K., Li, J., & Teoh, E. K. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2416634 · Cited by 160
Facial expression recognition via deep learning
Lv, Y., Feng, Z., & Xu, C. (2014) · 2014 International Conference on Smart Computing · doi:10.1109/smartcomp.2014.7043872 · Cited by 122
Facial animation parameters extraction and expression recognition using Hidden Markov Models
Pardàs, M., & Bonafonte, A. (2002) · Signal Processing: Image Communication · doi:10.1016/s0923-5965(02)00078-4 · Cited by 81
Learning Bases of Activity for Facial Expression Recognition
Sariyanidi, E., Gunes, H., & Cavallaro, A. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662237 · Cited by 73
Learning Expressionlets via Universal Manifold Model for Dynamic Facial Expression Recognition
Liu, M., Shan, S., Wang, R., & Chen, X. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2615424 · Cited by 56
Multi-task mid-level feature learning for micro-expression recognition
He, J., Hu, J., Lu, X., & Zheng, W. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.029 · Cited by 53
Towards Facial Expression Recognition in the Wild: A New Database and Deep Recognition System
Peng, X., Xia, Z., Li, L., & Feng, X. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.192 · Cited by 39
Emotion recognition based on a novel triangular facial feature extraction method
Huang, K., Huang, S., & Kuo, Y.-H. (2010) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596374 · Cited by 29
Cited by
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Self-citationRegion Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
Wang, K., Peng, X., Yang, J., Meng, D., & Qiao, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2956143 · Cited by 940
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
Khare, S. K., Blanes-Vidal, V., Nadimi, E. S., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102019 · Cited by 469
Zhao, Z., Liu, Q., & Wang, S. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3093397 · Cited by 375
Metadata
- Authors
- Li, S., & Deng, W.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2019-01
- Volume
- 28
- Issue
- 1
- Pages
- 356-370
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W2889978276
- Cited by
- 660
- References
- 81
- FWCI
- 69.10
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.