Recognizing Facial Expressions Using a Shallow Convolutional Neural Network
Open accessAbstract
Generally, facial expressions could be classified into two categories: static facial expressions and micro-expressions. There are many promising applications of facial expression recognition, such as pain detection, lie detection, and babysitting. Traditional convolutional neural network (CNN)-based methods suffer from two critical problems when they are adopted to recognize micro-expressions. First, they are usually dependent on very deep architectures that overfit on small datasets. However, reliable expressions are relatively difficult to collect and relevant datasets are usually relatively small. Second, for micro-expressions, these methods usually neglect the temporal redundancy of micro-expressions which could be utilized to reduce the temporal complexity. In this paper, we propose a shallow CNN (SHCNN) architecture with only three layers to classify static expressions and micro-expressions simultaneously without big training datasets. To better explain the functionality of our SHCNN architecture, we improve the saliency maps by introducing a shrinkage factor after studying the vanishing gradient problem of existing saliency maps. Experiments are conducted on five open datasets: FER2013, FERPlus, CASME, CASME II, and SAMM. To the best of our knowledge, by comparing with other methods offering source code (or pseudo code), we believe that our method would be the best on FERPlus, CASME, and CASME II and competitive on FER2013 and SAMM.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
Nonlinear total variation based noise removal algorithms
Rudin, L., Osher, S., & Fatemi, E. (1992) · Physica D Nonlinear Phenomena · doi:10.1016/0167-2789(92)90242-f · Cited by 17,089 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · see citing works
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965 · see citing works
Learning realistic human actions from movies
Ivan Laptev, A., Marszałek, M., Schmid, C., & Rozenfeld, B. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587756 · Cited by 3,481 · see citing works
chambolle (2004) · doi:10.1023/b:jmiv.0000011321.19549.88 · Cited by 3,227 · see citing works
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.179 · Cited by 3,177 · see citing works
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846 · see citing works
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719 · see citing works
Eulerian video magnification for revealing subtle changes in the world
Wu, H., Rubinstein, M., Shih, E. I., Guttag, J. V., Durand, F., & Freeman, W. T. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185561 · Cited by 1,365 · see citing works
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477450 · Cited by 1,132 · see citing works
CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086041 · Cited by 993 · see citing works
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926 · see citing works
Deep Learning using Linear Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0239 · Cited by 891 · see citing works
Beyond pixels: exploring new representations and applications for motion analysis
Freeman, W. T., Adelson, E. H., & Liu, C. (2009) · Cited by 830 · see citing works
A Biologically Inspired System for Action Recognition
Jhuang, H., Serre, T., Wolf, L., & Poggio, T. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408988 · Cited by 721 · see citing works
SAMM: A Spontaneous Micro-Facial Movement Dataset
Davison, A. K., Lansley, C., Costen, N. P., Tan, K., & Yap, M. H. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2573832 · Cited by 715 · see citing works
Robust facial expression recognition using local binary patterns
Shan, C., Gong, S., & McOwan, P. W. (2005) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2005.1530069 · Cited by 418 · see citing works
Less is more: Micro-expression recognition from video using apex frame
Liong, S., See, J., Wong, K., & Phan, R. C. ‐. (2017) · Signal Processing Image Communication · doi:10.1016/j.image.2017.11.006 · Cited by 399 · see citing works
Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., & Pietikainen, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2667642 · Cited by 384 · see citing works
CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces
Yan, W., Wu, Q., Liu, Y., Wang, S., & Fu, X. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553799 · Cited by 363 · see citing works
Wang, Y., See, J., Phan, R. C. ‐., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16865-4_34 · Cited by 309 · see citing works
Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns
Huang, X., Zhao, G., Hong, X., Zheng, W., & Pietikäinen, M. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.10.096 · Cited by 304 · see citing works
Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions
Xia, Z., Hong, X., Gao, X., Feng, X., & Zhao, G. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2931351 · Cited by 291 · see citing works
Microexpression Identification and Categorization Using a Facial Dynamics Map
Xu, F., Zhang, J., & Wang, J. Z. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2518162 · Cited by 284 · see citing works
Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · see citing works
Enriched Long-Term Recurrent Convolutional Network for Facial Micro-Expression Recognition
Khor, H.-Q., See, J., Phan, R. C. W., & Lin, W. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00105 · Cited by 244 · see citing works
Yang, B., Cao, J., Ni, R., & Zhang, Y. (2017) · IEEE Access · doi:10.1109/access.2017.2784096 · Cited by 206 · see citing works
Fuzzy Histogram of Optical Flow Orientations for Micro-Expression Recognition
Happy, S. L., & Routray, A. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2723386 · Cited by 175 · see citing works
Micro-expression recognition based on 3D flow convolutional neural network
Li, J., Wang, Y., See, J., & Liu, W. (2018) · Pattern Analysis and Applications · doi:10.1007/s10044-018-0757-5 · Cited by 173 · see citing works
Towards a practical lipreading system
Zhou, Z., Zhao, G., & Pietikäinen, M. (2011) · doi:10.1109/cvpr.2011.5995345 · Cited by 161 · see citing works
A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges
Oh, Y.-H., See, J., Ngo, A. C. L., Phan, R. C. ‐., & Baskaran, V. M. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01128 · Cited by 161 · see citing works
From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning
Peng, M., Wu, Z., Zhang, Z., & Chen, T. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00103 · Cited by 154 · see citing works
Learning From Hierarchical Spatiotemporal Descriptors for Micro-Expression Recognition
Zong, Y., Huang, X., Zheng, W., Cui, Z., & Zhao, G. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2820321 · Cited by 153 · see citing works
Wang, S., Li, B., Liu, Y., Yan, W., Ou, X., Huang, X., Xu, F., & Fu, X. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.107 · Cited by 149 · see citing works
Can Micro-Expression be Recognized Based on Single Apex Frame?
Li, Y., Huang, X., & Zhao, G. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451376 · Cited by 132 · see citing works
Facial Expression Recognition with Identity and Emotion Joint Learning
Li, M., Xu, H., Huang, X., Song, Z., Liu, X., & Li, X. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2880201 · Cited by 128 · see citing works
Self-citationZhang, Z., Chen, T., Meng, H., Liu, G., & Fu, X. (2018) · IEEE Access · doi:10.1109/access.2018.2879485 · Cited by 94 · see citing works
For micro-expression recognition: Database and suggestions
Yan, W., Wang, S., Liu, Y., Wu, Q., & Fu, X. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.01.029 · Cited by 66 · see citing works
Hand-Crafted Feature Guided Deep Learning for Facial Expression Recognition
Zeng, G., Zhou, J., Jia, X., Xie, W., & Shen, L. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00068 · Cited by 65 · see citing works
Tang, Y. D., Zhang, X. M., & Wang, H. (2018) · IEEE Access · doi:10.1109/access.2018.2858278 · Cited by 45 · see citing works
Facial expression recognition using ensemble of classifiers
Zavaschi, T. H. H., Koerich, A. L., & Oliveira, L. S. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946775 · Cited by 31 · see citing works
Ming, Z., Chazalon, J., Luqman, M. M., Visani, M., & Burie, J.-C. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545274 · Cited by 30 · see citing works
Multi-task Micro-expression Recognition Combining Deep and Handcrafted Features
Hu, C., Jiang, D., Zou, H., Zuo, X., & Shu, Y. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545555 · Cited by 20 · see citing works
Huang, X., Wang, S., Liu, X., Zhao, G., Feng, X., & Pietikäinen, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.02255 · Cited by 17 · see citing works
Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressionsPreprint
Xia, Z., Hong, X., Gao, X., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04656 · Cited by 17 · see citing works
Region Based Robust Facial Expression Analysis
Lian, Z., Li, Y., Tao, J., Huang, J., & Niu, M. (2018) · 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia) · doi:10.1109/aciiasia.2018.8470391 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [1]1. R. W. Picard, Affective computing. Cambridge, MA, USA: MIT Press, 1995.
- [26]26. M. Richter, T. Gehrig, and H. K. Ekenel, ''Facial expression classifica- tion on Web images,'' in Proc. 21st Int. Conf. Pattern Recognit. (ICPR), Nov. 2012, pp. 3517-3520.
- [37]37. A. L. Maas, A. Y. Hannun, and A. Y. Ng, ''Rectifier nonlinearities improve neural network acoustic models,'' in Proc. Int. Conf. Mach. Learn. (ICML), vol. 30, 2013, pp. 1-3.
- [41]41. S. T. Liong, Y. S. Gan, J. See, and H. Q. Khor, ''A shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recog- nition system,'' 2019, arXiv:1902.03634. [Online]. Available: http:// arxiv.org/abs/1902.03634 · arXiv
- [51]51. F. Xu, J. Zhang, and J. Z. Wang, ''Micro-expression identification and cat- egorization using a facial dynamics map,'' IEEE Trans. Affective Comput., vol. 8, no. 2, pp. 254-267, Apr./Jun. 2017.
Cited by
Sorted by citations
Kim, J., Kang, J.-K., & Kim, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3099075 · Cited by 50 · see citing works
Wikanningrum, A., Rachmadi, R. F., & Ogata, K. (2019) · 2019 International Conference on Computer Engineering, Network, and Intelligent Multimedia (CENIM) · doi:10.1109/cenim48368.2019.8973312 · Cited by 17 · see citing works
- Authors
- Miao, S., Xu, H., Han, Z., & Zhu, Y.
- Affiliations
- Chinese Academy of Sciences (CN)
- Lenovo (China) (CN)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 78000-78011
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2948217907
- Cited by
- 110
- References
- 52
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 19 · 2023: 28 · 2022: 25 · 2021: 20 · 2020: 8 · 2019: 1
- FWCI
- 15.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Science and Technology Commission of Shanghai Municipality (17511106400, 17511106402)
- Aerospace System Department (30508020301)
- National Natural Science Foundation of China (61201059)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, micro-expressions, static facial expressions, shallow convolutional neural network, saliency maps, gradient vanishing, temporal redundancy, limited training data, overfitting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
