The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion
Open accessAbstract
Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1×
“[51] proposed Pyramid Vision Transformer (PVT) for pixel-level dense prediction, which can work as the feature extraction backbone without convolutions.”
BackgroundLucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287 · cited 1×
“In addition, feature fusion can enrich the representative ability of the whole networks, which effectively improves the generalization ability and the recognition performance[38],[39],[40].”
BackgroundTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2020) · arXiv · doi:10.48550/arxiv.2012.09841 · Cited by 2,514
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,444
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Self-citationRecognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
Wang, K., Peng, X., Yang, J., Meng, D., & Qiao, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2956143 · Cited by 940
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916
(2016) · Cited by 844
Facial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 778
Self-citationSuppressing Uncertainties for Large-Scale Facial Expression Recognition
Wang, K., Peng, X., Yang, J., Lu, S., & Qiao, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00693 · Cited by 772
The Expression Of The Emotions In Man And Animals
Darwin, C. (1998) · doi:10.1093/oso/9780195112719.001.0001 · Cited by 732
Deep Learning using Linear Support Vector Machines
Tang, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0239 · Cited by 708
End-to-End Video Instance Segmentation with Transformers
Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., & Xia, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00863 · Cited by 679
Li, S., & Deng, W. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2868382 · Cited by 660
Self-citationAutomatic facial expression recognition using features of salient facial patches
Happy, S. L., & Routray, A. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2386334 · Cited by 599
Robust facial expression recognition using local binary patterns
Shan, C., Gong, S., & McOwan, P. W. (2005) · IEEE International Conference on Image Processing 2005 · doi:10.1109/icip.2005.1530069 · Cited by 414
Local binary patterns for multi-view facial expression recognition
Moore, S., & Bowden, R. (2011) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2010.12.001 · Cited by 351
Toward Transformer-Based Object DetectionPreprint
Beal, J., Kim, E. S., Tzeng, E., Park, D. H., Zhai, A., & Kislyuk, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09958 · Cited by 350
Attention mechanism-based CNN for facial expression recognition
Li, J., Jin, K., Zhou, D., Kubota, N., & Ju, Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.014 · Cited by 347
Facial Expression Recognition with Inconsistently Annotated Datasets
Zeng, J., Shan, S., & Chen, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_14 · Cited by 341
Joint Pose and Expression Modeling for Facial Expression Recognition
Zhang, F., Zhang, T., Mao, Q., & Xu, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00354 · Cited by 256
Visformer: The Vision-friendly Transformer
Chen, Z., Xie, L., Niu, J., Liu, X., Wei, L., & Tian, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00063 · Cited by 249
Facial Expression Recognition in Video with Multiple Feature Fusion
Chen, J., Chen, Z., Chi, Z., & Fu, H. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2593719 · Cited by 228
Three convolutional neural network models for facial expression recognition in the wild
Shao, J., & Qian, Y. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.005 · Cited by 186
Investigating Bias and Fairness in Facial Expression Recognition
Xu, T., White, J., Kalkan, S., & Gunes, H. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-65414-6_35 · Cited by 165
Patch-Gated CNN for Occlusion-aware Facial Expression Recognition
Li, Y., Zeng, J., Shan, S., & Chen, X. (2018) · 2018 24th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2018.8545853 · Cited by 158
Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression
Zheng, W. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2304712 · Cited by 145
Geometry Guided Pose-Invariant Facial Expression Recognition
Zhang, F., Zhang, T., Mao, Q., & Xu, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2972114 · Cited by 143
Multi-view facial expression recognition
Hu, Y., Zeng, Z., Yin, L., Wei, X., Zhou, X., & Huang, T. S. (2008) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2008.4813445 · Cited by 142
Facial Expression Recognition in the Wild Using Multi-Level Features and Attention Mechanisms
Li, Y., Lu, G., Li, J., Zhang, Z., & Zhang, D. D. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3031602 · Cited by 136
ConvTransformer: A Convolutional Transformer Network for Video Frame SynthesisPreprint
Liu, Z., Luo, S. N., Li, W., Lu, J., Wu, Y., Sun, S., Li, C., & Luxi, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10185 · Cited by 125
Lee, S. H., Plataniotis, K. N., & Ro, Y. M. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2346515 · Cited by 124
Lai, Y.-H., & Lai, S. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00046 · Cited by 113
Recognizing Facial Expressions Using a Shallow Convolutional Neural Network
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2921220 · Cited by 110
Facial Expression Recognition With Deeply-Supervised Attention Network
Fan, Y., Li, V. O. K., & Lam, J. C. K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2988264 · Cited by 107
Feature Selection Mechanism in CNNs for Facial Expression Recognition.
Zhao, S., Cai, H., Liu, H., Zhang, J., & Chen, S. (2018) · British Machine Vision Conference (BMVC) · Cited by 102
1 Facial Expression Recognition with Local Binary Patterns and Linear Programming1
Feng, X., Pietikäinen, M., & Hadid, A. (2008) · Cited by 101
Buciu, I., & Pitas, I. (2004) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2004.1334109 · Cited by 93
Recognition of Facial Expressions in the Presence of Occlusion
Bourel, F., Chibelushi, C. C., & Low, A. A. (2001) · British Machine Vision Conference (BMVC) · doi:10.5244/c.15.23 · Cited by 79
Multi-resolution feature fusion for face recognition
Pong, K.-H., & Lam, K. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.08.023 · Cited by 73
Combining convolutional neural networks for emotion recognition
Huang, C. (2017) · IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc.2017.8284175 · Cited by 65
Learning Informative and Discriminative Features for Facial Expression Recognition in the Wild
Li, Y., Lu, Y., Chen, B., Zhang, Z., Li, J., Lu, G., & Zhang, D. D. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3103760 · Cited by 64
Weakly-Supervised Facial Expression Recognition in the Wild With Noisy Data
Zhang, F., Xu, M., & Xu, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3072786 · Cited by 62
Relation-Aware Facial Expression Recognition
Xia, Y., Yu, H. L., Wang, X., Jian, M., & Wang, F. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3100131 · Cited by 59
Multi-Channel Pose-Aware Convolution Neural Networks for Multi-View Facial Expression Recognition
Liu, Y., Zeng, J., Shan, S., & Zheng, Z. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00074 · Cited by 55
Learning Discriminative Representation For Facial Expression Recognition From Uncertainties
Fan, X., Deng, Z., Wang, K., Peng, X., & Qiao, Y. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9190643 · Cited by 48
Deep Siamese Neural Networks for Facial Expression Recognition in the Wild
Hayale, W. S. A., Negi, P., & Mahoor, M. H. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3077248 · Cited by 48
Xie, W., Wu, H., Tian, Y., Bai, M., & Shen, L. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3063052 · Cited by 48
TransPose: Towards Explainable Human Pose Estimation by TransformerPreprint
Yang, S., Quan, Z., Nie, M., & Yang, W. (2020) · arXiv (Cornell University) · Cited by 40
Trajformer: Trajectory Prediction with Local Self-Attentive Contexts for Autonomous DrivingPreprint
Bhat, M., Francis, J., & Oh, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14910 · Cited by 32
Expression recognition with deep features extracted from holistic and part-based models
Happy, S. L., Dantcheva, A., & Brémond, F. (2020) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104038 · Cited by 20
Unsupervised Cross-View Facial Expression Image Generation and Recognition
Sun, N., Lu, Q., Zheng, W., Liu, J., & Han, G. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3029531 · Cited by 13
Cited by
A survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 451
Savchenko, A. V., Savchenko, L. V., & Makarov, I. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3188390 · Cited by 346
Adaptive Multilayer Perceptual Attention Network for Facial Expression Recognition
Liu, H., Cai, H., Lin, Q., Li, X., & Xiao, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3165321 · Cited by 129
PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition
Ngwe, J. L., Lim, K. M., Lee, C. P., Ong, T. S., & Alqahtani, A. (2024) · IEEE Access · doi:10.1109/access.2024.3407108 · Cited by 45
Metadata
- Authors
- Ma, F., Sun, B., & Li, S.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2023-04-01
- Volume
- 14
- Issue
- 2
- Pages
- 1236-1248
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W3209798173
- Cited by
- 327
- References
- 67
- FWCI
- 39.94
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.