Semantic Alignment: Finding Semantically Consistent Ground-Truth for Facial Landmark Detection
Open accessAbstract
Recently, deep learning based facial landmark detection has achieved great success. Despite this, we notice that the semantic ambiguity greatly degrades the detection performance. Specifically, the semantic ambiguity means that some landmarks (e.g. those evenly distributed along the face contour) do not have clear and accurate definition, causing inconsistent annotations by annotators. Accordingly, these inconsistent annotations, which are usually provided by public databases, commonly work as the ground-truth to supervise network training, leading to the degraded accuracy. To our knowledge, little research has investigated this problem. In this paper, we propose a novel probabilistic model which introduces a latent variable, i.e. the `real' ground-truth which is semantically consistent, to optimize. This framework couples two parts (1) training landmark detection CNN and (2) searching the `real' ground-truth. These two parts are alternatively optimized: the searched `real' ground-truth supervises the CNN training; and the trained CNN assists the searching of `real' ground-truth. In addition, to recover the unconfidently predicted landmarks due to occlusion and low quality, we propose a global heatmap correction unit (GHCU) to correct outliers by considering the global face shape as a constraint. Extensive experiments on both image-based (300W and AFLW) and video-based (300-VW) databases demonstrate that our method effectively improves the landmark detection accuracy and achieves the state of the art performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The alignment task can be divided into attentional alignment and semantic alignment.
“Generally, the alignment task can be divided into two subtasks: the attentional alignment task [170, 171] and the semantic alignment task [172, 173].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [172]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
(2017) · Cited by 11,625 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
Face detection, pose estimation, and landmark localization in the wild
Zhu, X., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248014 · Cited by 2,171 · see citing works
Self-citationSupervised Descent Method and Its Applications to Face Alignment
Xiong, X., & la Torre, F. D. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.75 · Cited by 1,917 · see citing works
Bulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464 · see citing works
Facial Landmark Detection by Deep Multi-task Learning
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_7 · Cited by 1,410 · see citing works
Deep Convolutional Network Cascade for Facial Point Detection
Sun, Y. J., Wang, X., & Tang, X. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.446 · Cited by 1,405 · see citing works
Bulat, A., & Tzimiropoulos, G. (2017) · Nottingham ePrints (University of Nottingham) · Cited by 1,277 · see citing works
Face Alignment Across Large Poses: A 3D Solution
Zhu, X., Lei, Z., Liu, X., Shi, H., & Li, S. Z. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.23 · Cited by 1,137 · see citing works
Self-citationKöstinger, M., Wohlhart, P., Roth, P. M., & Bischof, H. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130513 · Cited by 1,004 · see citing works
Interactive Facial Feature Localization
Le, V., Brandt, J., Lin, Z., Bourdev, L. D., & Huang, T. S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33712-3_49 · Cited by 957 · see citing works
Localizing parts of faces using a consensus of exemplars
Belhumeur, P. N., Jacobs, D. W., Kriegman, D., et al. (2011) · doi:10.1109/cvpr.2011.5995602 · Cited by 536 · see citing works
Look at Boundary: A Boundary-Aware Face Alignment Algorithm
Wu, W., Qian, C., Yang, S., Wang, Q., Cai, Y., & Zhou, Q. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00227 · Cited by 517 · see citing works
Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks
Feng, Z., Kittler, J., Awais, M., Huber, P., & Wu, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00238 · Cited by 489 · see citing works
Face alignment by coarse-to-fine shape searching
Zhu, S., Li, C., Loy, C. C., & Tang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299134 · Cited by 488 · see citing works
Learning Deep Representation for Face Alignment with Auxiliary Attributes
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2469286 · Cited by 425 · see citing works
A Semi-automatic Methodology for Facial Landmark Annotation
Sagonas, C., Tzimiropoulos, G., Zafeiriou, S. P., & Pantić, M. (2013) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2013.132 · Cited by 392 · see citing works
Face Alignment in Full Pose Range: A 3D Total Solution
Zhu, X., Liu, X., Lei, Z., & Li, S. Z. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2778152 · Cited by 391 · see citing works
Self-citationHu, G., Yang, Y., Yi, D., Kittler, J., Christmas, W. J., Li, S. Z., & Hospedales, T. M. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.58 · Cited by 337 · see citing works
Self-citationAutomatic feature localisation with constrained local models
Cristinacce, D., & Cootes, T. F. (2008) · Pattern Recognition · doi:10.1016/j.patcog.2008.01.024 · Cited by 329 · see citing works
The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results
Shen, J., Zafeiriou, S. P., Chrysos, G. G., Kossaifi, J., Tzimiropoulos, G., & Pantić, M. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.132 · Cited by 289 · see citing works
Stacked Hourglass Network for Robust Facial Landmark Localisation
Yang, J., Liu, Q., & Zhang, K. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.253 · Cited by 281 · see citing works
Lv, J.-J., Shao, X., Xing, J., Cheng, C., & Zhou, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.393 · Cited by 244 · see citing works
Project-Out Cascaded Regression with an application to face alignment
Tzimiropoulos, G. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298989 · Cited by 219 · see citing works
Dong, X., Yu, S.-I., Weng, X., Wei, S.-E., Yang, Y., & Sheikh, Y. A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00045 · Cited by 216 · see citing works
Robust Facial Landmark Detection via Recurrent Attentive-Refinement Networks
Xiao, S., Feng, J., Xing, J., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_4 · Cited by 214 · see citing works
Unconstrained Face Alignment via Cascaded Compositional Learning
Zhu, S., Li, C., Loy, C. C., & Tang, X. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.371 · Cited by 174 · see citing works
Disentangling 3D Pose in a Dendritic CNN for Unconstrained 2D Face Alignment
Kumar, A., & Chellappa, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00052 · Cited by 152 · see citing works
Frankenstein: Learning Deep Face Representations Using Small Data
Hu, G., Peng, X., Yang, Y., Hospedales, T. M., & Verbeek, J. J. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2756450 · Cited by 137 · see citing works
Self-citationDirect Shape Regression Networks for End-to-End Face Alignment
Miao, X., Zhen, X., Liu, X., Deng, C. D., Athitsos, V., & Huang, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00529 · Cited by 111 · see citing works
Offline Deformable Face Tracking in Arbitrary Videos
Chrysos, G. G., Antonakos, E., Zafeiriou, S. P., et al. (2015) · doi:10.1109/iccvw.2015.126 · Cited by 107 · see citing works
A Deeply-Initialized Coarse-to-fine Ensemble of Regression Trees for Face Alignment
Valle, R., Buenaposada, J. M., Valdés, A., & Baumela, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_36 · Cited by 106 · see citing works
Face Alignment via Regressing Local Binary Features
Ren, S., Cao, X., Wei, Y.-C., & Sun, J. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2518867 · Cited by 96 · see citing works
Robust Facial Landmark Detection via a Fully-Convolutional Local-Global Context Network
Merget, D., Rock, M., & Rigoll, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00088 · Cited by 96 · see citing works
Two-Stream Transformer Networks for Video-Based Face Alignment
Liu, H., Lu, J., Feng, J., & Zhou, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2734779 · Cited by 94 · see citing works
Ketkar, N. S. (2017) · Apress eBooks · doi:10.1007/978-1-4842-2766-4_9 · Cited by 74 · see citing works
Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural NetworksPreprint
Feng, Z., Kittler, J., Awais, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06753 · Cited by 37 · see citing works
Joint Multi-view Face Alignment in the WildPreprint
Deng, J., Trigeorgis, G., Zhou, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06023 · Cited by 32 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- K. Messer, J. Matas, J. Kittler, and K. Jonsson. Xm2vts: the extended m2vts database. In Proc. Second International Conference on Audio-and Video-Based Biometric Person Authentication, pages 72-77, 2000.
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
- Authors
- Liu, Z., Zhu, X., Hu, G., Guo, H., Tang, M., Lei, Z., Robertson, N. M., & Wang, J.
- Affiliations
- Chinese Academy of Sciences (CN)
- Queen's University Belfast (GB)
- ORCID
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 3462-3471
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962708168
- Cited by
- 73
- References
- 39
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 7 · 2023: 7 · 2022: 13 · 2021: 25 · 2020: 11 · 2019: 5
- FWCI
- 4.40
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Research Portal (Queen's University Belfast) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- facial landmark detection, semantic ambiguity, probabilistic models, latent variables, semantic consistency, occlusion handling, AFLW
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
