The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods
Open accessAbstract
Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Self-citationPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Active Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikainen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,584
Self-citationDynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Self-citationA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
Emotion elicitation using films
Gross, J. J., & Levenson, R. W. (1995) · Cognition & Emotion · doi:10.1080/02699939508408966 · Cited by 2,179
Nonverbal Leakage and Clues to Deception †
Ekman, P., & Friesen, W. V. (1969) · Psychiatry · doi:10.1080/00332747.1969.11023575 · Cited by 1,854
Eulerian video magnification for revealing subtle changes in the world
Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., & Freeman, W. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185561 · Cited by 1,357
Web-Based Database for Facial Expression Analysis
Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005) · 2005 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2005.1521424 · Cited by 1,039
CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086041 · Cited by 993
Self-citationBeyond pixels: exploring new representations and applications for motion analysis
Freeman, W. T., Adelson, E. H., & Liu, C. (2009) · Cited by 805
Facial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 778
Self-citationA Spontaneous Micro-expression Database: Inducement, collection and baseline
Li, X., Pfister, T., Huang, X., Zhao, G., & Pietikainen, M. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553717 · Cited by 665
Self-citationDarwin, Deception, and Facial Expression
EKMAN, P. (2003) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1280.010 · Cited by 624
Face recognition using Histograms of Oriented Gradients
Déniz, O., Bueno, G., Salido, J., & De la Torre, F. (2011) · Pattern Recognition Letters · doi:10.1016/j.patrec.2011.01.004 · Cited by 619
Porter, S., & ten Brinke, L. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02116.x · Cited by 463
Recognising spontaneous facial micro-expressions
Pfister, T., Li, X., Zhao, G., & Pietikainen, M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126401 · Cited by 450
Self-citationHow Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions
Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., & Fu, X. (2013) · Journal of Nonverbal Behavior · doi:10.1007/s10919-013-0159-8 · Cited by 445
Face Recognition Under Varying Illumination Using Gradientfaces
Zhang, T., Tang, Y. Y., Fang, B., Shang, Z., & Liu, X. (2009) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2028255 · Cited by 412
The ability to detect deceit generalizes across different types of high-stake lies.
Frank, M. G., & Ekman, P. (1997) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.72.6.1429 · Cited by 403
Ekman, P., O'Sullivan, M., & Frank, M. G. (1999) · Psychological Science · doi:10.1111/1467-9280.00147 · Cited by 388
Handbook of Emotion Elicitation and Assessment
(2007) · doi:10.1093/oso/9780195169157.001.0001 · Cited by 348
Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy
Haggard, E. A., & Isaacs, K. S. (1966) · Methods of Research in Psychotherapy · doi:10.1007/978-1-4684-6045-2_14 · Cited by 343 · cited 1×
“The phenomenon was first discovered by Haggard and Isaacs [5] in 1966, who called them micromomentary facial expressions.”
BackgroundLie Catching and Microexpressions
Ekman, P. (2009) · The Philosophy of Deception · doi:10.1093/acprof:oso/9780195327939.003.0008 · Cited by 332
Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor
Polikovsky, S., Kameda, Y., & Ohta, Y. (2009) · 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009) · doi:10.1049/ic.2009.0244 · Cited by 312
Image registration by local approximation methods
Goshtasby, A. (1988) · Image and Vision Computing · doi:10.1016/0262-8856(88)90016-9 · Cited by 291
Evidence for training the ability to read microexpressions of emotion
Matsumoto, D., & Hwang, H. S. (2011) · Motivation and Emotion · doi:10.1007/s11031-011-9212-2 · Cited by 245
Eye-blink detection system for human–computer interaction
Królak, A., & Strumiłło, P. (2012) · Universal Access in the Information Society · doi:10.1007/s10209-011-0256-6 · Cited by 237
Macro- and micro-expression spotting in long videos using spatio-temporal strain
Shreve, M., Godavarthy, S., Goldgof, D., & Sarkar, S. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771451 · Cited by 218
Micro-Expression Recognition Using Color Spaces
Wang, S.-J., Yan, W.-J., Li, X., Zhao, G., Zhou, C.-G., Fu, X., Yang, M., & Tao, J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2496314 · Cited by 193
Self-citationDetecting Deception from Emotional and Unemotional Cues
Warren, G., Schertler, E., & Bull, P. (2009) · Journal of Nonverbal Behavior · doi:10.1007/s10919-008-0057-7 · Cited by 189
Emotion recognition in the wild challenge 2013
Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531739 · Cited by 178
Towards a practical lipreading system
Zhou, Z., Zhao, G., & Pietikainen, M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995345 · Cited by 158
Self-citationMicro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space
Wang, S., Yan, W., Li, X., Zhao, G., & Fu, X. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.800 · Cited by 147
Self-citationSpotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis
Moilanen, A., Zhao, G., & Pietikäinen, M. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.303 · Cited by 125
Self-citationTowards macro- and micro-expression spotting in video using strain patterns
Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., & Sarkar, S. (2009) · 2009 Workshop on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2009.5403044 · Cited by 124
The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System
Wu, Q., Shen, X., & Fu, X. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24571-8_16 · Cited by 104
Spontaneous Emotional Facial Expression Detection
Zeng, Z., Fu, Y., Roisman, G. I., Wen, Z., Hu, Y., & Huang, T. S. (2006) · Journal of Multimedia · doi:10.4304/jmm.1.5.1-8 · Cited by 99
Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System (FACS)
Polikovsky, S., Kameda, Y., & Ohta, Y. (2012) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e96.d.81 · Cited by 74
Facial-component-based bag of words and PHOG descriptor for facial expression recognition
Li, Z., Imai, J., & Kaneko, M. (2009) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2009.5346254 · Cited by 70
Optical strain based recognition of subtle emotions
Liong, S., Phan, R. C. ‐., See, J., Oh, Y.-H., & Wong, K. (2014) · International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS) · doi:10.1109/ispacs.2014.7024448 · Cited by 67
Encoding Local Binary Patterns using the re-parametrization of the second order Gaussian jet
Ruiz-Hernandez, J. A., & Pietikäinen, M. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553709 · Cited by 35
Learning a sparse codebook of facial and body microexpressions for emotion recognition
Song, Y., Morency, L., & Davis, R. (2013) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2522848.2522851 · Cited by 35
Incremental Slow Feature Analysis with Indefinite Kernel for Online Temporal Video Segmentation
Liwicki, S., Zafeiriou, S. P., & Pantić, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37444-9_13 · Cited by 17
Micro-expression recognition by feature points tracking
Yao, S., He, N., Zhang, H., et al. (2014) · International Conference on Communications (COMM) · doi:10.1109/iccomm.2014.6866671 · Cited by 17
Cited by
Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., & Liu, Y.-J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3067464 · Cited by 306
Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions
Xia, Z., Hong, X., Gao, X., Feng, X., & Zhao, G. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2931351 · Cited by 290
Self-citationShallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition
Liong, S.-T., Gan, Y. S., See, J., Khor, H.-Q., & Huang, Y.-C. (2019) · 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) · doi:10.1109/fg.2019.8756567 · Cited by 253
Multimodal data to design visual learning analytics for understanding regulation of learning
Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.019 · Cited by 140
15 challenges for AI: Or what AI (currently) can’t do
Hagendorff, T., & Wezel, K. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00886-y · Cited by 113
Metadata
- Authors
- Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., & Pietikainen, M.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2018-10-01
- Volume
- 9
- Issue
- 4
- Pages
- 563-577
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2587585332
- Cited by
- 323
- References
- 49
- FWCI
- 28.19
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.