Robust Face Recognition via Multimodal Deep Face Representation
Open accessAbstract
Face images appearing in multimedia applications, e.g., social networks and digital entertainment, usually exhibit dramatic pose, illumination, and expression variations, resulting in considerable performance degradation for traditional face recognition algorithms. This paper proposes a comprehensive deep learning framework to jointly learn face representation using multimodal information. The proposed deep learning structure is composed of a set of elaborately designed convolutional neural networks (CNNs) and a three-layer stacked auto-encoder (SAE). The set of CNNs extracts complementary facial features from multimodal data. Then, the extracted features are concatenated to form a high-dimensional feature vector, whose dimension is compressed by SAE. All of the CNNs are trained using a subset of 9,000 subjects from the publicly available CASIA-WebFace database, which ensures the reproducibility of this work. Using the proposed single CNN architecture and limited training data, 98.43% verification rate is achieved on the LFW database. Benefitting from the complementary information contained in multimodal data, our small ensemble system achieves higher than 99.0% recognition rate on LFW using publicly available training set.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ding and colleagues introduced a late-fusion scheme that uses CNNs to extract multimodal facial features and stacked autoencoders to reduce dimensionality.
“In [51], Ding et al. proposed a new late fusion policy using CNNs for multimodal facial feature extraction and SAEs for dimensional reduction.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · see citing works
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikäinen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,592 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,748 · see citing works
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,445 · see citing works
Deep Learning Face Representation by Joint Identification-Verification
Sun, Y., Wang, X., & Tang, X. (2014) · arXiv · doi:10.48550/arxiv.1406.4773 · Cited by 1,906 · see citing works
Learning Face Representation from ScratchPreprint
Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7923 · Cited by 1,629 · see citing works
Facial Landmark Detection by Deep Multi-task Learning
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_7 · Cited by 1,410 · see citing works
A 3D Face Model for Pose and Illumination Invariant Face Recognition
Paysan, P., Knothe, R., Amberg, B., Romdhani, S., & Vetter, T. R. (2009) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2009.58 · Cited by 1,325 · see citing works
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · see citing works
Blur Insensitive Texture Classification Using Local Phase Quantization
Ojansivu, V., & Heikkilä, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69905-7_27 · Cited by 1,043 · see citing works
Probabilistic Linear Discriminant Analysis for Inferences About Identity
Prince, S. J. D., & Elder, J. H. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4409052 · Cited by 988 · see citing works
Deeply learned face representations are sparse, selective, and robust
Sun, Y. J., Wang, X., & Tang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298907 · Cited by 952 · see citing works
Classification with Noisy Labels by Importance Reweighting
Liu, T., & Tao, D. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2456899 · Cited by 796 · see citing works
Self-citationBsif: Binarized statistical image features
Kannala, J., & Rahtu, E. (2012) · Cited by 562 · see citing works
Correlative multi-label video annotation
Qi, G.-J., Hua, X., Rui, Y., et al. (2007) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1291233.1291245 · Cited by 478 · see citing works
Bayesian Face Revisited: A Joint Formulation
Chen, D., Cao, X., Wang, L., Wen, F., & Sun, J. Z. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33712-3_41 · Cited by 437 · see citing works
Multi-Directional Multi-Level Dual-Cross Patterns for Robust Face Recognition
Ding, C., Choi, J., Tao, D., & Davis, L. S. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2462338 · Cited by 383 · see citing works
Self-citationLearning Discriminant Face Descriptor
Lei, Z., Pietikäinen, M., & Li, S. Z. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.112 · Cited by 347 · see citing works
A Comprehensive Survey on Pose-Invariant Face Recognition
Ding, C., & Tao, D. (2016) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2845089 · Cited by 329 · see citing works
Self-citationRecognition of blurred faces using Local Phase Quantization
Ahonen, T., Rahtu, E., Ojansivu, V., & Heikkilä, J. (2008) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2008.4761847 · Cited by 308 · see citing works
Multi-Task Pose-Invariant Face Recognition
Ding, C., Xu, C., & Tao, D. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2390959 · Cited by 269 · see citing works
Self-citationLarge-Margin Multi-ViewInformation Bottleneck
Xu, C., Tao, D., & Xu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.2296528 · Cited by 255 · see citing works
Self-citationWeb-scale training for face identification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298891 · Cited by 232 · see citing works
Towards Pose Robust Face Recognition
Yi, D., Lei, Z., & Li, S. Z. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.454 · Cited by 206 · see citing works
Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not?Preprint
Zhou, E., Cao, Z., & Yin, Q. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.04690 · Cited by 192 · see citing works
Exploring Context and Content Links in Social Media: A Latent Space Method
Qi, G.-J., Aggarwal, C. C., Tian, Q., et al. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.191 · Cited by 151 · see citing works
Two-Dimensional Active Learning for image classification
Qi, G.-J., Hua, X., Rui, Y., Tang, J., & Zhang, H.-J. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587383 · Cited by 117 · see citing works
Kernel Optimization in Discriminant Analysis
You, D., Hamsici, O. C., & Martı́nez, A. M. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.173 · Cited by 96 · see citing works
Choi, J. Y., Neve, W. D., Plataniotis, K. N., et al. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2087320 · Cited by 84 · see citing works
Dynamic Texture Recognition Using Multiscale Binarized Statistical Image Features
Arashloo, S. R., & Kittler, J. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2362855 · Cited by 81 · see citing works
Face Search at Scale: 80 Million GalleryPreprint
Wang, D., Otto, C., & Jain, A. K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.07242 · Cited by 79 · see citing works
Robust Multitask Multiview Tracking in Videos
Mei, X., Hong, Z., Prokhorov, D. V., & Tao, D. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2399233 · Cited by 78 · see citing works
Self-citationQi, G.-J., Hua, X., Rui, Y., et al. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.218 · Cited by 78 · see citing works
Learning semantic distance from community-tagged media collection
Qi, G.-J., Hua, X., & Zhang, H.-J. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631307 · Cited by 60 · see citing works
The 2013 speaker recognition evaluation in mobile environment
Khoury, E., Vesnicer, B., Franco-Pedroso, J., et al. (2013) · International Conference on Biometrics · doi:10.1109/icb.2013.6613025 · Cited by 56 · see citing works
Report on the FG 2015 Video Person Recognition Evaluation
Beveridge, J. R., Zhang, H., Draper, B. A., et al. (2015) · IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163156 · Cited by 51 · see citing works
Expression-Invariant Face Recognition With Constrained Optical Flow Warping
Hsieh, C.-K., Lai, S., & Chen, Y. (2009) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2009.2017606 · Cited by 37 · see citing works
A Comprehensive Survey on Pose-Invariant Face RecognitionPreprint
Ding, C., & Tao, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04383 · Cited by 35 · see citing works
Self-citationThe 2013 face recognition evaluation in mobile environment
Günther, M., Costa‐Pazo, A., Ding, C., et al. (2013) · International Conference on Biometrics · doi:10.1109/icb.2013.6613024 · Cited by 35 · see citing works
Self-citationMulti-view constrained clustering with an incomplete mapping between views
Eaton, E., desJardins, M., & Jacob, S. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-012-0577-7 · Cited by 32 · see citing works
Co-clustering of multi-view datasets
Hussain, S. F., & Bashir, S. (2015) · Knowledge and Information Systems · doi:10.1007/s10115-015-0861-4 · Cited by 31 · see citing works
Multi-View Constrained Clustering with an Incomplete Mapping Between Views
Eaton, E., Desjardins, M. E., & Jacob, S. (2012) · Cited by 29 · see citing works
Web-Scale Training for Face IdentificationPreprint
Taigman, Y., Yang, M., Ranzato, M., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5266 · Cited by 13 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
DehazeNet: An End-to-End System for Single Image Haze Removal
Cai, B., Xu, X., Jia, K., Qing, C., & Tao, D. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2598681 · Cited by 3,367 · see citing works
Self-citationDeep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · see citing works
Face Recognition Systems: A Survey
Kortli, Y., Jridi, M., Al Falou, A., & Atri, M. (2020) · Sensors · doi:10.3390/s20020342 · Cited by 623 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Attractive-and-repulsive center-symmetric local binary patterns for texture classification
El merabet, Y., Ruichek, Y., & El idrissi, A. (2019) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2018.11.011 · Cited by 86 · see citing works
A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding
Montesinos-López, O. A., Chavira-Flores, M., Kiasmiantini, Crespo-Herrera, L., Saint Piere, C., Li, H., Fritsche-Neto, R., Al-Nowibet, K., Montesinos-López, A., & Crossa, J. (2024) · GENETICS · doi:10.1093/genetics/iyae161 · Cited by 47 · see citing works
Multimodal Recurrent Neural Networks With Information Transfer Layers for Indoor Scene Labeling
Abdulnabi, A. H., Shuai, B., Zuo, Z., Chau, L.-P., & Wang, G. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2774007 · Cited by 33 · see citing works
- Authors
- Ding, C., & Tao, D.
- Affiliations
- University of Technology Sydney (AU)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2015-11
- Volume
- 17
- Issue
- 11
- Pages
- 2049-2058
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W1951319388
- Cited by
- 449
- References
- 47
- Citations by year
- 2026: 7 · 2025: 29 · 2024: 23 · 2023: 36 · 2022: 44 · 2021: 58 · 2020: 53 · 2019: 67 · 2018: 52 · 2017: 50 · 2016: 27 · 2015: 3
- FWCI
- 38.17
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UTS ePRESS (University of Technology Sydney) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Australian Research Council (FT-130101457, DP-120103730, DP-140102164)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Keywords
- face recognition, stacked autoencoder, convolutional neural networks, pose variation, illumination variation, facial expression variation, CASIA-WebFace, Labeled Faces in the Wild, robust face recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
