CoChat
Sign Up
PreprintCited by 3Face recognition and analysisAdversarial Robustness in Machine LearningBiometric Identification and Security

The DOI matches its DataCite record.

Masked Faces with Faced Masks

Open access
Zhu, Jiayi, Guo, Qing, Juefei-Xu, Felix, Huang, Yihao, Liu, Yang, Pu, Geguang·2022·arXiv
DOI 10.48550/arxiv.2201.06427arXiv 2201.06427
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern face recognition systems (FRS) still fall short when the subjects are wearing facial masks, a common theme in the age of respiratory pandemics. An intuitive partial remedy is to add a mask detector to flag any masked faces so that the FRS can act accordingly for those low-confidence masked faces. In this work, we set out to investigate the potential vulnerability of such FRS equipped with a mask detector, on large-scale masked faces, which might trigger a serious risk, e.g., letting a suspect evade the FRS where both facial identity and mask are undetected. As existing face recognizers and mask detectors have high performance in their respective tasks, it is significantly challenging to simultaneously fool them and preserve the transferability of the attack. We formulate the new task as the generation of realistic & adversarial-faced mask and make three main contributions: First, we study the naive Delanunay-based masking method (DM) to simulate the process of wearing a faced mask that is cropped from a template image, which reveals the main challenges of this new task. Second, we further equip the DM with the adversarial noise attack and propose the adversarial noise Delaunay-based masking method (AdvNoise-DM) that can fool the face recognition and mask detection effectively but make the face less natural. Third, we propose the adversarial filtering Delaunay-based masking method denoted as MF2M by employing the adversarial filtering for AdvNoise-DM and obtain more natural faces. With the above efforts, the final version not only leads to significant performance deterioration of the state-of-the-art (SOTA) deep learning-based FRS, but also remains undetected by the SOTA facial mask detector, thus successfully fooling both systems at the same time.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2201.06427

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SOTA detectors may fall short when faces are under occlusions such as facial masks, heavy makeups, heavy facial hairs, etc.

    “In addition, the SOTA detectors may fall short when faces are under occlusions such as facial masks (Zhu et al. 2022) where only the eye region is visible (Juefei-Xu and Savvides 2016; Juefei-Xu et al. 2015), heavy makeups, heavy facial hairs, etc.”

    From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Zhu et al. 2022

    The Introduction (p. 2) says existing mask detectors can be confused by specially patterned ‘faced masks,’ supporting the facial-mask part of the claim. It does not establish that face detectors generally fall short with ordinary masks; the cited paper’s other findings concern face recognition systems. The makeup and facial-hair examples are not attributed to Zhu et al. in the citing sentence. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 45

  1. [1][1] Borut Batagelj, Peter Peer, Vitomir Štruc, and Simon Dobrišek. 2021. How to Correctly Detect Face-Masks for COVID-19 from Visual Information? Applied Sciences 11, 5 (2021), 2070. -
  2. [2][2] Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. 2021. Unmasking Face Embeddings by Self-restrained Triplet Loss for Accurate Masked Face Recognition. arXiv preprint arXiv:2103.01716 (2021). - · arXiv
  3. [3][3] Fadi Boutros, Naser Damer, Jan Niklas Kolf, Kiran Raja, Florian Kirchbuchner, Raghavendra Ramachandra, Arjan Kuijper, Pengcheng Fang, Chao Zhang, Fei Wang, et al. 2021. MFR 2021: Masked Face Recognition Competition. In 2021 IEEE International Joint Conference on Biometrics (IJCB) . IEEE, 1-10. -
  4. [4][4] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. 2018. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) . IEEE, 67-74. -
  5. [5][5] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. 2019. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR . -
  6. [9][9] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. Boosting adversarial attacks with momentum. In CVPR . 9185-9193. -
  7. [10][10] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. 2019. Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR . 4312-4321. -
  8. [11][11] Ruijun Gao, Qing Guo, Qian Zhang, Felix Juefei-Xu, Hongkai Yu, and Wei Feng. 2021. Adversarial Relighting against Face Recognition. arXiv preprint arXiv:2108.07920 (2021). - · arXiv
  9. [12][12] Shiming Ge, Jia Li, Qiting Ye, and Zhao Luo. 2017. Detecting masked faces in the wild with lle-cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition . 2682-2690. -
  10. [13][13] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014). - · arXiv
  11. [14][14] Qing Guo, Ziyi Cheng, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yang Liu, and Jianjun Zhao. 2021. Learning to Adversarially Blur Visual Object Tracking. In ICCV . 10839-10848. -
  12. [15][15] Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Jian Wang, Bing Yu, Wei Feng, and Yang Liu. 2020. Watch out! Motion is Blurring the Vision of Your Deep Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS) . -
  13. [16][16] Felix Juefei-Xu. 2018. Unconstrained Periocular Face Recognition: From Reconstructive Dictionary Learning to Generative Deep Learning and Beyond. PhD Dissertation, Carnegie Mellon University (2018). -
  14. [17][17] Felix Juefei-Xu, Khoa Luu, and Marios Savvides. 2015. Spartans: Single-sample periocular-based alignment-robust recognition technique applied to non-frontal scenarios. IEEE Transactions on Image Processing 24, 12 (2015), 4780-4795. -
  15. [18][18] Felix Juefei-Xu, Dipan K Pal, and Marios Savvides. 2014. Hallucinating the full face from the periocular region via dimensionally weighted K-SVD. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops . 1-8. -
  16. [19][19] Felix Juefei-Xu and Marios Savvides. 2014. Subspace-based discrete transform encoded local binary patterns representations for robust periocular matching on NIST's face recognition grand challenge. IEEE transactions on image processing 23, 8 (2014), 3490-3505. -
  17. [20][20] Felix Juefei-Xu and Marios Savvides. 2016. Fastfood dictionary learning for periocular-based full face hallucination. In 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS) . IEEE, 1-8. -
  18. [21][21] Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 4401-4410. -
  19. [22][22] Ira Kemelmacher-Shlizerman, Steven M Seitz, Daniel Miller, and Evan Brossard. 2016. The megaface benchmark: 1 million faces for recognition at scale. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4873-4882. -
  20. [23][23] Stepan Komkov and Aleksandr Petiushko. 2021. Advhat: Real-world adversarial attack on arcface face id system. In 2020 25th International Conference on Pattern Recognition (ICPR) . IEEE, 819-826. -
  21. [24][24] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236 (2016). - · arXiv
  22. [25][25] Yande Li, Kun Guo, Yonggang Lu, and Li Liu. 2021. Cropping and attention based approach for masked face recognition. Applied Intelligence 51, 5 (2021), 3012-3025. -
  23. [26][26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision . 2980-2988. -
  24. [27][27] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 212-220. -
  25. [28][28] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. 2016. Large-margin softmax loss for convolutional neural networks.. In ICML , Vol. 2. 7. -
  26. [29][29] Mohamed Loey, Gunasekaran Manogaran, Mohamed Hamed N Taha, and Nour Eldeen M Khalifa. 2021. A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic. Measurement 167 (2021), 108288. -
  27. [30][30] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017). - · arXiv
  28. [32][32] David Montero, Marcos Nieto, Peter Leskovsky, and Naiara Aginako. 2021. Boosting Masked Face Recognition with Multi-Task ArcFace. arXiv preprint arXiv:2104.09874 (2021). - · arXiv
  29. [33][33] Mei Ngan, Patrick Grother, and Kayee Hanaoka. 2020. Ongoing Face Recognition Vendor Test (FRVT) Part 6B: Face recognition accuracy with face masks using post-COVID-19 algorithms. https://doi . org/10 . 6028/NIST . IR . 8331 -
  30. [34][34] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep Face Recognition. In Proceedings of the British Machine Vision Conference (BMVC) . BMVA Press, Article 41, 12 pages. -
  31. [35][35] Bosheng Qin and Dongxiao Li. 2020. Identifying facemask-wearing condition using image super-resolution with classification network to prevent COVID-19. Sensors 20, 18 (2020), 5236. -
  32. [36][36] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. 2016. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communications security . 1528-1540. -
  33. [38][38] Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2014. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1891-1898. -
  34. [39][39] Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1701-1708. -
  35. [40][40] Binyu Tian, Felix Juefei-Xu, Qing Guo, Xiaofei Xie, Xiaohong Li, and Yang Liu. 2021. AVA: Adversarial Vignetting Attack against Visual Recognition. arXiv preprint arXiv:2105.05558 (2021). - · arXiv
  36. [41][41] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. 2018. Additive margin softmax for face verification. IEEE Signal Processing Letters 25, 7 (2018), 926-930. -
  37. [42][42] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. 2018. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 5265-5274. -
  38. [43][43] Run Wang, Felix Juefei-Xu, Qing Guo, Yihao Huang, Xiaofei Xie, Lei Ma, and Yang Liu. 2020. Amora: Black-box adversarial morphing attack. In ACM-MM . 1376-1385. -
  39. [44][44] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600-612. -
  40. [45][45] Zhanxiong Wang, Keke He, Yanwei Fu, Rui Feng, Yu-Gang Jiang, and Xiangyang Xue. 2017. Multi-task deep neural network for joint face recognition and facial attribute prediction. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval . 365-374. -
  41. [46][46] Zhongyuan Wang, Guangcheng Wang, Baojin Huang, Zhangyang Xiong, Qi Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, et al. 2020. Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020). - · arXiv
  42. [47][47] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In European conference on computer vision . Springer, 499-515. -
  43. [48][48] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. 2019. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2730-2739. -
  44. [49][49] Bangjie Yin, Wenxuan Wang, Taiping Yao, Junfeng Guo, Zelun Kong, Shouhong Ding, Jilin Li, and Cong Liu. 2021. Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognition. arXiv preprint arXiv:2105.03162 (2021). - · arXiv
  45. [50][50] Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. 2017. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision . 5409-5418.

Cited by

Sorted by citations

Metadata

Authors
Zhu, J., Guo, Q., Juefei-Xu, F., Huang, Y., Liu, Y., & Pu, G.
Affiliations
  • East China Normal University (CN)
  • Nanyang Technological University (SG)
  • Alibaba Group (United States) (US)
Corresponding author
  • Guo, Qing
  • Pu, Geguang
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
360-377
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2201.06427
arXiv
2201.06427
OpenAlex
W4223419855
Cited by
3
References
0
Citations by year
2025: 1 · 2023: 1 · 2022: 1
FWCI
0.20
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
Keywords
face recognition, facial mask, face mask detection, adversarial attacks, face spoofing, adversarial perturbations, evasion attacks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review