The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions
Open accessAbstract
Recently, the recognition task of spontaneous facial micro-expressions has attracted much attention with its various real-world applications. Plenty of handcrafted or learned features have been employed for a variety of classifiers and achieved promising performances for recognizing micro-expressions. However, the micro-expression recognition is still challenging due to the subtle spatiotemporal changes of micro-expressions. To exploit the merits of deep learning, we propose a novel deep recurrent convolutional networks based micro-expression recognition approach, capturing the spatiotemporal deformations of micro-expression sequence. Specifically, the proposed deep model is constituted of several recurrent convolutional layers for extracting visual features and a classificatory layer for recognition. It is optimized by an end-to-end manner and obviates manual feature design. To handle sequential data, we exploit two ways to extend the connectivity of convolutional networks across temporal domain, in which the spatiotemporal deformations are modeled in views of facial appearance and geometry separately. Besides, to overcome the shortcomings of limited and imbalanced training samples, two temporal data augmentation strategies as well as a balanced loss are jointly used for our deep network. By performing the experiments on three spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression recognition approach compared to the state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 5×
“To model spatiotemporal variations, several deep approaches have exploited various architectures to be adaptive for video/sequence based data in many fields [48], such as video classification [49]–[51] and action recognition [52]–[54], to cite only few.”
BackgroundMethodInfluential3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_2 · Cited by 3,958
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
Secrets of optical flow estimation and their principles
Sun, D., Roth, S., & Black, M. J. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539939 · Cited by 1,389
Eulerian video magnification for revealing subtle changes in the world
Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., & Freeman, W. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185561 · Cited by 1,357
CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086041 · Cited by 993 · cited 5×
“Besides, no more than 250 original samples in CASME II [26] can be used to train deep RCNs.”
BackgroundMethodInfluentialSelf-citationRecurrent convolutional neural network for object recognition
Liang, M., & Hu, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298958 · Cited by 976
SAMM: A Spontaneous Micro-Facial Movement Dataset
Davison, A. K., Lansley, C., Costen, N., Tan, K., & Yap, M. H. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2573832 · Cited by 708
A Spontaneous Micro-expression Database: Inducement, collection and baseline
Li, X., Pfister, T., Huang, X., Zhao, G., & Pietikainen, M. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553717 · Cited by 665
Self-citationGoing deeper into action recognition: A survey
Herath, S., Harandi, M., & Porikli, F. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.010 · Cited by 634
A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition
Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., & Fu, X. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2485205 · Cited by 520
Self-citationECO: Efficient Convolutional Network for Online Video Understanding
Zolfaghari, M., Singh, K., & Brox, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_43 · Cited by 505
Recognising spontaneous facial micro-expressions
Pfister, T., Li, X., Zhao, G., & Pietikainen, M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126401 · Cited by 450
Self-citationZhang, S., Zhang, S., Huang, T., & Gao, W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 441
Less is more: Micro-expression recognition from video using apex frame
Liong, S.-T., See, J., Wong, K., & Phan, R. C.-W. (2018) · Signal Processing: Image Communication · doi:10.1016/j.image.2017.11.006 · Cited by 394
CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces
Yan, W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., & Fu, X. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553799 · Cited by 338
Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., & Pietikainen, M. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2667642 · Cited by 323
Self-citationA Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition
Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., & Yan, K. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2598092 · Cited by 314
Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor
Polikovsky, S., Kameda, Y., & Ohta, Y. (2009) · 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009) · doi:10.1049/ic.2009.0244 · Cited by 312 · cited 3×
“Among them, the Polikovskys dataset [28] and USF-HD dataset [29] were constructed by collecting acted micro-expressions of subjects with high-speed cameras and are not available publicly.”
BackgroundMethodSpontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns
Huang, X., Zhao, G., Hong, X., Zheng, W., & Pietikäinen, M. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.10.096 · Cited by 303
Self-citationWang, Y., See, J., Phan, R. C.-W., & Oh, Y.-H. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16865-4_34 · Cited by 295
CAS(ME): A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition
Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., & Fu, X. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2654440 · Cited by 293
Microexpression Identification and Categorization Using a Facial Dynamics Map
Xu, F., Zhang, J., & Wang, J. Z. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2518162 · Cited by 283
Macro- and micro-expression spotting in long videos using spatio-temporal strain
Shreve, M., Godavarthy, S., Goldgof, D., & Sarkar, S. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771451 · Cited by 218 · cited 1×
“For the detection task, the geometric features [34]–[36] and local textures [37], [38] have been proposed to capture micro-expression frames from videos.”
BackgroundMultimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network
Li, H., Sun, J., Xu, Z., & Chen, L. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2713408 · Cited by 211
Huang, X., Wang, S.-J., Zhao, G., & Piteikainen, M. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.10 · Cited by 203
Self-citationHuang, X., Wang, S.-J., Liu, X., Zhao, G., Feng, X., & Pietikainen, M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2713359 · Cited by 194
Self-citationMicro-Expression Recognition Using Color Spaces
Wang, S.-J., Yan, W.-J., Li, X., Zhao, G., Zhou, C.-G., Fu, X., Yang, M., & Tao, J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2496314 · Cited by 193
Self-citationLearning-Based Video Motion Magnification
Oh, T.-H., Jaroensri, R., Kim, C., Elgharib, M., Durand, F., Freeman, W. T., & Matusik, W. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_39 · Cited by 167
Learning From Hierarchical Spatiotemporal Descriptors for Micro-Expression Recognition
Zong, Y., Huang, X., Zheng, W., Cui, Z., & Zhao, G. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2820321 · Cited by 155
Self-citationMicro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space
Wang, S., Yan, W., Li, X., Zhao, G., & Fu, X. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.800 · Cited by 147
Self-citationRecurrent Convolutional Neural Network Regression for Continuous Pain Intensity Estimation in Video
Zhou, J. L., Hong, X., Su, F., & Zhao, G. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.191 · Cited by 142
Self-citationEffects of the duration of expressions on the recognition of microexpressions
Shen, X., Wu, Q., & Fu, X. (2012) · Journal of Zhejiang University SCIENCE B · doi:10.1631/jzus.b1100063 · Cited by 141
A survey: facial micro-expression recognition
Takalkar, M. A., Xu, M. G., Wu, Q., & Chaczko, Z. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5317-2 · Cited by 133
Spotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis
Moilanen, A., Zhao, G., & Pietikäinen, M. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.303 · Cited by 125
Self-citationTowards macro- and micro-expression spotting in video using strain patterns
Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., & Sarkar, S. (2009) · 2009 Workshop on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2009.5403044 · Cited by 124 · cited 3×
“Among them, the Polikovskys dataset [28] and USF-HD dataset [29] were constructed by collecting acted micro-expressions of subjects with high-speed cameras and are not available publicly.”
BackgroundMethodWang, S., Yan, W., Zhao, G., Fu, X., & Zhou, C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_23 · Cited by 119
Self-citationAdaptive weighted imbalance learning with application to abnormal activity recognition
Gao, X., Chen, Z., Tang, S., Zhang, Y., & Li, J. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.09.064 · Cited by 112
Self-citationSOML: Sparse Online Metric Learning with Application to Image Retrieval
Gao, X., Hoi, S. C. H., Zhang, Y., Wan, J., & Li, J. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8911 · Cited by 101
Self-citationA main directional maximal difference analysis for spotting facial movements from long-term videos
Wang, S., Wu, S., Qian, X., Li, J., & Fu, X. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.12.034 · Cited by 99
Learning Compact Hash Codes for Multimodal Representations Using Orthogonal Deep Structure
Wang, D., Cui, P., Ou, M., & Zhu, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2455415 · Cited by 97
Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets
Takalkar, M. A., & Xu, M. G. (2017) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2017.8227443 · Cited by 97
Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis and Application
Ngo, A. C. L., See, J., & Phan, R. C. ‐. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2523996 · Cited by 81
Spontaneous micro-expression spotting via geometric deformation modeling
Xia, Z., Feng, X., Peng, J., Peng, X., & Zhao, G. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.12.006 · Cited by 76
Self-citationSpontaneous subtle expression detection and recognition based on facial strain
Liong, S., See, J., Phan, R. C. ‐., Oh, Y.-H., Ngo, A. C. L., Wong, K., & Tan, S. (2016) · Signal Processing Image Communication · doi:10.1016/j.image.2016.06.004 · Cited by 76
Sparse Online Learning of Image Similarity
Gao, X., Hoi, S. C. H., Zhang, Y., Zhou, J., Wan, J., Chen, Z., Li, J., & Zhu, J. (2017) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3065950 · Cited by 54
Self-citationMicro-Facial Movements: An Investigation on Spatio-Temporal Descriptors
Davison, A. K., Yap, M. H., Costen, N. P., Tan, K., Lansley, C., & Leightley, D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16181-5_8 · Cited by 51
A Delaunay-Based Temporal Coding Model for Micro-expression Recognition
Lu, Z., Luo, Z., Zheng, H., Chen, J., & Li, W. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16631-5_51 · Cited by 49
Natural affect data — Collection & annotation in a learning context
Afzal, S., & Robinson, P. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349537 · Cited by 48
Encoding Local Binary Patterns using the re-parametrization of the second order Gaussian jet
Ruiz-Hernandez, J. A., & Pietikäinen, M. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553709 · Cited by 39
Spontaneous Facial Micro-expression Recognition via Deep Convolutional Network
Xia, Z., Feng, X., Hong, X., & Zhao, G. (2018) · International Conference on Image Processing Theory, Tools and Applications (IPTA) · doi:10.1109/ipta.2018.8608119 · Cited by 37
Self-citationRecognizing spontaneous micro-expression from eye region
Duan, X., Dai, Q., Wang, X., Wang, Y., & Hua, Z. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.03.090 · Cited by 32
Exploiting Web Images for Video Highlight Detection With Triplet Deep Ranking
Kim, H., Mei, T., Byun, H., & Yao, T. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2806224 · Cited by 25
Micro-expression recognition by feature points tracking
Yao, S., He, N., Zhang, H., et al. (2014) · International Conference on Communications (COMM) · doi:10.1109/iccomm.2014.6866671 · Cited by 17
An accurate eye localization approach for smart embedded system
Xia, Z., Zhang, W., Tan, F., Feng, X., & Hadid, A. (2016) · International Conference on Image Processing Theory, Tools and Applications (IPTA) · doi:10.1109/ipta.2016.7821006 · Cited by 7
Self-citation
Cited by
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., & Liu, Y.-J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3067464 · Cited by 306
Recognizing Facial Expressions Using a Shallow Convolutional Neural Network
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2921220 · Cited by 110
“STRCN [42] employs 3D convolution and face alignments.”
MethodAn Overview of Emotion in Artificial Intelligence
Assunção, G., Patrão, B., Castelo-Branco, M., & Menezes, P. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3159614 · Cited by 103
Saffaryazdi, N., Wasim, S. T., Dileep, K., Nia, A. F., Nanayakkara, S., Broadbent, E., & Billinghurst, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.864047 · Cited by 69
Metadata
- Authors
- Xia, Z., Hong, X., Gao, X., Feng, X., & Zhao, G.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2020-03
- Volume
- 22
- Issue
- 3
- Pages
- 626-640
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2964606879
- Cited by
- 290
- References
- 66
- FWCI
- 38.05
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.