CoChat
Sign Up
PreprintCited by 111Face recognition and analysisFace and Expression RecognitionDomain Adaptation and Few-Shot Learning

Self-supervised learning of a facial attribute embedding from video

Open access
Wiles, Olivia, Koepke, A. Sophia, Zisserman, Andrew·2018·arXiv
DOI 10.48550/arxiv.1808.06882arXiv 1808.06882
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1808.06882

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Ensuring data diversity, complex scene variations, environmental disturbances, and individual expression differences contribute to the difficulty and expense of data collection and labelling.

    “Ensuring data diversity is one aspect of this challenge, but it is also necessary to take into account complex scene variations, environmental disturbances, and individual expression differences. These factors contribute to the difficulty and expense of data collection and labelling [7].”

    From Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition (2025) · cites this paper as [7]

    The Introduction (p. 1) says supervised facial-analysis methods need large amounts of labelled data that may be expensive or difficult to obtain in realistic, unconstrained settings; it also notes disagreement among annotators of emotions. It does not identify data diversity, scene variations, environmental disturbances, or differences between individuals’ expressions as causes of that difficulty and expense. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Self-supervised learning techniques can discern salient features from unlabelled data and subsequently be fine-tuned for specific tasks.

    “The capacity of SSL techniques to discern salient features from unlabelled data, subsequently fine-tuned for specific tasks, is a salient benefit.”

    From Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition (2025) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 63

  1. [1][1] P. Agrawal, J. Carreira, and J. Malik. Learning to see by moving. In Proc. ICCV , 2015. -
  2. [2][2] E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction , pages 279-283. ACM, 2016. -
  3. [3][3] A. Bas, P. Huber, W. A. P. Smith, M. Awais, and J. Kittler. 3D morphable models as spatial transformer networks. In Proc. ICCV Workshop on Geometry Meets Deep Learning , 2017. -
  4. [4][4] P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman, and N. Kumar. Localizing parts of faces using a consensus of exemplars. IEEE PAMI , 2013. -
  5. [6][6] C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez. EmotioNet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In Proc. CVPR , 2016. -
  6. [7][7] V. Blanz, S. Romdhani, and T. Vetter. Face identification across different poses and illumination with a 3D morphable model. In Proc. AFGR , 2002. -
  7. [8][8] J. S. Chung, A. Jamaludin, and A. Zisserman. You said that? In Proc. BMVC. , 2017. -
  8. [10][10] F. Cole, D. Belanger, D. Krishnan, A. Sarna, I. Mosseri, and W. T. Freeman. Synthesizing normalized faces from facial identity features. In Proc. CVPR , 2017. -
  9. [11][11] E. L. Denton and V. Birodkar. Unsupervised learning of disentangled representations from video. In NIPS , 2017. -
  10. [12][12] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proc. ICCV , 2015. -
  11. [13][13] A. Dosovitskiy, J. T. Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. In Proc. CVPR , 2015. -
  12. [14][14] P. Ekman and H. Oster. Facial expressions of emotion. Annual review of psychology , 30(1):527-554, 1979. -
  13. [15][15] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. Proc. ACM SIGGRAPH , 2018. -
  14. [16][16] B. Fernando, H. Bilen, E. Gavves, and S. Gould. Self-supervised video representation learning with odd-one-out networks. In Proc. CVPR , 2017. -
  15. [17][17] C. Gan, B. Gong, K. Liu, H. Su, and L. J. Guibas. Geometry guided convolutional neural networks for self-supervised video representation learning. In Proc. CVPR , 2018. -
  16. [19][19] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In Proc. ICLR , 2018. -
  17. [20][20] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML , 2015. -
  18. [21][21] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi. Conditional image generation for learning the structure of visual objects. arXiv preprint arXiv:1806.07823 , 2018. - · arXiv
  19. [22][22] X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool. Dynamic filter networks. In NIPS , 2016. -
  20. [23][23] D. Kingma and J. Ba. Adam: A method for stochastic optimization. Proc. ICLR , 2015. -
  21. [24][24] M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof. Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization. In Proc. First IEEE International Workshop on Benchmarking Facial Image Analysis Technologies , 2011. -
  22. [25][25] T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. Deep convolutional inverse graphics network. In NIPS , 2015. -
  23. [26][26] A. Kumar, A. Alavi, and R. Chellappa. KEPLER: keypoint and pose estimation of unconstrained faces by learning efficient H-CNN regressors. In Proc. Int. Conf. Autom. Face and Gesture Recog. , 2017. -
  24. [27][27] M. P. Kumar, B. Packer, and D. Koller. Self-paced learning for latent variable models. In NIPS . 2010. -
  25. [28][28] N. Kumar, A. Berg, P. Belhumeur, and S. Nayar. Describable visual attributes for face verification and image search. IEEE PAMI , 33(10):1962-1977, 2011. -
  26. [29][29] D. Lee, H. Park, and C. D. Yoo. Face alignment using cascade gaussian process regression trees. In Proc. CVPR , 2015. -
  27. [30][30] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang. Unsupervised representation learning by sorting sequences. In Proc. ICCV , 2017. -
  28. [31][31] Y. Li, R. Wang, H. Liu, H. Jiang, S. Shan, and X. Chen. Two birds, one stone: Jointly learning binary code for large-scale face image retrieval and attributes prediction. In Proc. ICCV , 2015. -
  29. [32][32] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proc. ICCV , 2015. -
  30. [33][33] I. Loshchilov and F. Hutter. Online batch selection for faster training of neural networks. ICLR Workshops , 2016. -
  31. [34][34] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Proc. ECCV , 2016. -
  32. [35][35] A. Mollahosseini, B. Hasani, and M. H. Mahoor. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing , 2017. -
  33. [36][36] A. Nagrani, J. S. Chung, and A. Zisserman. VoxCeleb: a large-scale speaker identification dataset. In INTERSPEECH , 2017. -
  34. [37][37] A. Nagrani, S. Albanie, and A. Zisserman. Learnable pins: Cross-modal embeddings for person identity. In Proc. ECCV , 2018. -
  35. [38][38] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proc. ECCV , 2016. -
  36. [39][39] D. Novotny, D. Larlus, and A. Vedaldi. Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching. In Proc. CVPR , 2017. -
  37. [40][40] D. Novotny, S. Albanie, D. Larlus, and A. Vedaldi. Self-supervised learning of geometrically stable features through probabilistic introspection. In Proc. CVPR , 2018. -
  38. [41][41] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In Proc. BMVC. , 2015. -
  39. [43][43] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proc. CVPR , 2016. -
  40. [44][44] V. P˘ atr˘ aucean, A. Handa, and R. Cipolla. Spatio-temporal video autoencoder with differentiable memory. In NIPS , 2016. -
  41. [45][45] S. Ren, X. Cao, Y. Wei, and J. Sun. Face alignment at 3000 fps via regressing local binary features. In Proc. CVPR , 2014. -
  42. [46][46] E. M. Rudd, M. Günther, and T. E. Boult. Moon: A mixed objective optimization network for the recognition of facial attributes. In Proc. ECCV , 2016. -
  43. [48][48] A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In Proc. CVPR , 2016. -
  44. [49][49] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors. In Proc. ICCV , 2015. -
  45. [50][50] A. Tewari, M. Zollhöfer, H. Kim, P. Garrido, F. Bernard, P. Perez, and C. Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proc. ICCV , 2017. -
  46. [51][51] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks by factorized spatial embeddings. In Proc. ICCV , 2017. -
  47. [52][52] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object frames by dense equivariant image labelling. In NIPS , 2017. -
  48. [53][53] L. Tran, X. Yin, and X. Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proc. CVPR , 2017. -
  49. [54][54] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy. Tracking emerges by colorizing videos. In Proc. ECCV , 2018. -
  50. [56][56] O. Wiles, A. S. Koepke, and A. Zisserman. X2Face: A network for controlling face generation using images, audio, and pose codes. In Proc. ECCV , 2018. -
  51. [57][57] S. Xiao, J. Feng, J. Xing, H. Lai, S. Yan, and A. Kassim. Robust facial landmark detection via recurrent attentive-refinement networks. In Proc. ECCV , 2016. -
  52. [58][58] T. Xue, J. Wu, K. Bouman, and B. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In NIPS , 2016. -
  53. [59][59] J. Yang, P. Ren, D. Chen, F. Wen, H. Li, and G. Hua. Neural aggregation network for video face recognition. In Proc. CVPR , 2017. -
  54. [60][60] X. Yu, F. Zhou, and M. Chandraker. Deep deformation network for object landmark localization. In Proc. ECCV , 2016. -
  55. [61][61] A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and S. Savarese. Generic 3d representation via pose estimation and matching. In Proc. ECCV , 2016. -
  56. [62][62] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In Proc. ECCV , 2016. -
  57. [63][63] R. Zhang, P. Isola, and A. A. Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proc. CVPR , 2017. -
  58. [64][64] Y. Zhang, Y. Guo, Y. Jin, Y. Luo, Z. He, and H. Lee. Unsupervised discovery of object landmarks as structural representations. In Proc. CVPR , 2018. -
  59. [65][65] Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Facial landmark detection by deep multi-task learning. In Proc. ECCV , 2014. -
  60. [66][66] Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Learning deep representation for face alignment with auxiliary attributes. IEEE PAMI , 2016. -
  61. [68][68] T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros. View synthesis by appearance flow. In Proc. ECCV , 2016. -
  62. [69][69] S. Zhu, C. Li, C. C. Loy, and X. Tang. Face alignment by coarse-to-fine shape searching. In Proc. CVPR , 2015. -
  63. [70][70] X. Zhu and D. Ramanan. Face detection, pose estimation and landmark localization in the wild. Proc. CVPR , 2012.

Cited by

Sorted by citations

Authors
Wiles, O., Koepke, A. S., & Zisserman, A.
Affiliations
University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
302
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1808.06882
arXiv
1808.06882
OpenAlex
W2887997593
Cited by
111
References
4
Citations by year
2025: 5 · 2024: 14 · 2023: 15 · 2022: 15 · 2021: 23 · 2020: 20 · 2019: 15 · 2018: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Oxford University Research Archive (ORA) (University of Oxford) (submittedVersion)
  • arXiv (Cornell University)
Funders
Engineering and Physical Sciences Research Council (EP/M013774/1, EP/M013774/)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
self-supervised learning, facial expression recognition, head pose estimation, facial landmark detection, facial video, curriculum learning, attention masks, face tracking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review