Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Abstract
Dynamic texture (DT) is an extension of texture to the temporal domain. Description and recognition of DTs have attracted growing attention. In this paper, a novel approach for recognizing DTs is proposed and its simplifications and extensions to facial image analysis are also considered. First, the textures are modeled with volume local binary patterns (VLBP), which are an extension of the LBP operator widely used in ordinary texture analysis, combining motion and appearance. To make the approach computationally simple and easy to extend, only the co-occurrences of the local binary patterns on three orthogonal planes (LBP-TOP) are then considered. A block-based method is also proposed to deal with specific dynamic events such as facial expressions in which local information and its spatial locations should also be taken into account. In experiments with two DT databases, DynTex and Massachusetts Institute of Technology (MIT), both the VLBP and LBP-TOP clearly outperformed the earlier approaches. The proposed block-based method was evaluated with the Cohn-Kanade facial expression database with excellent results. The advantages of our approach include local processing, robustness to monotonic gray-scale changes, and simple computation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LBP-TOP is a widely used feature in facial expression analysis from video.
“Local binary patterns from three orthogonal planes (LBP-TOP) is a widely used feature in facial expression analysis from video52,53.”
From A Multimodal Dataset for Mixed Emotion Recognition (2024) · cites this paper as [52]
The importance of facial dynamics in FER has been established in many vision experiments.
“The importance of facial dynamics in FER has been established in many vision experiments [16,19].”
From Multimodal learning for facial expression recognition (2015) · cites this paper as [16]
Cites
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · see citing works
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · see citing works
Automatic Facial Expression Recognition Using Facial Animation Parameters and Multistream HMMs
Aleksic, P. S., & Katsaggelos, A. K. (2006) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2005.863510 · Cited by 235 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300 · see citing works
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · see citing works
BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database
Zhang, X., Yin, L., Cohn, J. F., Canavan, S., Reale, M., Horowitz, A., Liu, P., & Girard, J. M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.002 · Cited by 745 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
The first facial expression recognition and analysis challenge
Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771374 · Cited by 385 · see citing works
Local binary features for texture classification: Taxonomy and experimental study
Liu, L., Fieguth, P., Guo, Y., Wang, X., & Pietikäinen, M. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.08.032 · Cited by 369 · see citing works
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · see citing works
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305 · see citing works
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Khorrami, P., Paine, T. L., & Huang, T. S. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.12 · Cited by 285 · see citing works
Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · see citing works
Facial Expression Recognition Using Facial Movement Features
Zhang, L., & Tjondronegoro, D. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.13 · Cited by 253 · see citing works
Video-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235 · see citing works
From individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218 · see citing works
Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network
Li, H., Sun, J., Xu, Z., & Chen, L. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2713408 · Cited by 211 · see citing works
Feature Extraction Based on Co-occurrence of Adjacent Local Binary Patterns
Nosaka, R., Ohkawa, Y., & Fukui, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25346-1_8 · Cited by 192 · see citing works
Generalized Completed Local Binary Patterns for Time-Efficient Steel Surface Defect Classification
Luo, Q., Sun, Y., Li, P., Simpson, O., Tian, L., & He, Y. (2019) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2018.2852918 · Cited by 178 · see citing works
Audio-visual emotion fusion (AVEF): A deep efficient weighted approach
Ma, Y., Hao, Y., Chen, M., Chen, J., Lu, P., & Košir, A. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.06.003 · Cited by 155 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Facial expression recognition using Gabor motion energy filters
Wu, T., Bartlett, M. S., & Movellan, J. R. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543267 · Cited by 126 · see citing works
Multiple kernel learning for emotion recognition in the wild
Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531741 · Cited by 119 · see citing works
Recognizing Facial Expressions Using a Shallow Convolutional Neural Network
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2921220 · Cited by 110 · see citing works
Multimodal learning for facial expression recognition
Zhang, W., Zhang, Y., Ma, L., Guan, J., & Gong, S. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.04.012 · Cited by 100 · see citing works
Chu, M., Gong, R., Gao, S., & Zhao, J. (2017) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2017.10.020 · Cited by 97 · see citing works
FIVR: Fine-Grained Incident Video Retrieval
Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., & Kompatsiaris, I. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2905741 · Cited by 78 · see citing works
Facial expression recognition in the wild based on multimodal texture features
Sun, B., Li, L., Zhou, G., & He, J. (2016) · Journal of Electronic Imaging · doi:10.1117/1.jei.25.6.061407 · Cited by 74 · see citing works
Chen, S., Tian, Y., Liu, Q., & Metaxas, D. N. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.014 · Cited by 72 · see citing works
Partial least squares regression on grassmannian manifold for emotion recognition
Liu, M., Wang, R., Huang, Z., Shan, S., & Chen, X. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531738 · Cited by 62 · see citing works
Anti-spoofing in Action: Joint Operation with a Verification System
Chingovska, I., Anjos, A., & Marcel, S. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.22 · Cited by 62 · see citing works
Local normal binary patterns for 3D facial action unit detection
Sandbach, G., Zafeiriou, S., & Pantic, M. (2012) · 2012 19th IEEE International Conference on Image Processing · doi:10.1109/icip.2012.6467234 · Cited by 56 · see citing works
A Multimodal Dataset for Mixed Emotion Recognition
Yang, P., Liu, N., Liu, X., Shu, Y., Ji, W., Ren, Z., Sheng, J., Yu, M., Yi, R., Zhang, D., et al. (2024) · Scientific Data · doi:10.1038/s41597-024-03676-4 · Cited by 37 · see citing works
Dynamic facial expression recognition by joint static and multi-time gap transition classification
Dapogny, A., Bailly, K., & Dubuisson, S. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163111 · Cited by 24 · see citing works
Sharifnejad, M., Shahbahrami, A., Akoushideh, A., & Hassanpour, R. Z. (2021) · IET Image Processing · doi:10.1049/ipr2.12037 · Cited by 19 · see citing works
Multi-scale Facial Expression Recognition Based on Multi-Direction Local Binary Pattern
Xu, S., Xiao, S., Wang, S., Zhang, W., Zeng, F., Yu, Y., & Jiang, S. (2025) · 2025 5th International Conference on Communication Technology and Information Technology (ICCTIT) · doi:10.1109/icctit68197.2025.11406676
- Authors
- Zhao, G., & Pietikainen, M.
- Affiliations
- University of Oulu (FI)
- ORCID
- Zhao, Guoying
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2007-06
- Volume
- 29
- Issue
- 6
- Pages
- 915-928
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2139916508
- Cited by
- 2,846
- References
- 3
- Citations by year
- 2026: 73 · 2025: 131 · 2024: 128 · 2023: 134 · 2022: 177 · 2021: 198 · 2020: 199 · 2019: 212 · 2018: 212 · 2017: 197 · 2016: 232 · 2015: 242 · 2014: 188 · 2013: 164 · 2012: 99
- FWCI
- 71.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Massachusetts Institute of Technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- dynamic texture recognition, local binary pattern, facial expressions, Cohn-Kanade dataset
- MeSH terms
- Algorithms, Artificial Intelligence, Biometry, Face, Facial Expression, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Pattern Recognition, Automated, Sensitivity and Specificity, Signal Processing, Computer-Assisted, Video Recording, Reproducibility of Results, Computer Security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review