CoChat

A survey on face data augmentation for the training of deep neural networks

Open access
Wang, Xiang, Wang, Kai, Lian, Shiguo · 2020 · Neural Computing and Applications

The DOI matches its Crossref record; 115 of the 115 references it cites were found in a registry or an index.

Abstract

The quality and size of training set have a great impact on the results of deep learning-based face-related tasks. However, collecting and labeling adequate samples with high-quality and balanced distributions still remains a laborious and expensive work, and various data augmentation techniques have thus been widely used to enrich the training dataset. In this paper, we review the existing works of face data augmentation from the perspectives of the transformation types and methods, with the state-of-the-art approaches involved. Among all these approaches, we put the emphasis on the deep learning-based works, especially the generative adversarial networks which have been recognized as more powerful and effective tools in recent years. We present their principles, discuss the results and show their applications as well as limitations. Different evaluation metrics for evaluating these approaches are also introduced. We point out the challenges and opportunities in the field of face data augmentation and provide brief yet insightful discussions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00521-020-04748-3

Cites

115 of 115 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 49

  1. Alhaija HA, Mustikovela SK, Mescheder L, Geiger A, Rother C (2017) Augmented reality meets deep learning for car instance segmentation in urban scenes. In: British machine vision conference, vol 1, p 2
  2. Antoniou A, Storkey A, Edwards H (2017) Data augmentation generative adversarial networks. ArXiv preprint arXiv:1711.04340 · arXiv
  3. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: International conference on machine learning, pp 214–223
  4. Banerjee S, Scheirer WJ, Bowyer KW, Flynn, PJ (2018) On hallucinating context and background pixels from a face mask using multi-scale gans. ArXiv preprint arXiv:1811.07104 · arXiv
  5. Blanz V, Vetter T et al (1999) A morphable model for the synthesis of 3d faces. Siggraph 99:187–194
  6. Cao J, Hu Y, Yu B, He R, Sun Z (2018) Load balanced gans for multi-view face image synthesis. ArXiv preprint arXiv:1802.07447 · arXiv
  7. Cao J, Hu Y, Zhang H, He R, Sun Z (2018) Learning a high fidelity pose invariant model for high-resolution face frontalization. In: Advances in neural information processing systems, pp 2872–2882
  8. Chen W, Xie X, Jia X, Shen L (2018) Texture deformation based generative adversarial networks for face editing. ArXiv preprint arXiv:1812.09832 · arXiv
  9. Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P (2016) Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In: Advances in neural information processing systems, pp 2172–2180
  10. Crispell D, Biris O, Crosswhite N, Byrne J, Mundy JL (2017) Dataset augmentation for pose and lighting invariant face recognition. ArXiv preprint arXiv:1704.04326 · arXiv
  11. Das D, Lee CSG (2018) Graph matching and pseudo-label guided deep unsupervised domain adaptation. In: International conference on artificial neural networks, pp 342–352
  12. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680
  13. Gu G, Kim ST, Kim K, Baddar WJ, Ro YM (2017) Differential generative adversarial networks: synthesizing non-linear facial variations with limited number of training data. ArXiv preprint arXiv:1711.10267 · arXiv
  14. Gulrajani I, Kumar K, Ahmed F, Taiga AA, Visin F, Vazquez D, Courville A (2016) Pixelvae: a latent variable model for natural images. ArXiv preprint arXiv:1611.05013 · arXiv
  15. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems, pp 6626–6637
  16. Huang H, He R, Sun Z, Tan T et al (2018) Introvae: introspective variational autoencoders for photographic image synthesis. In: Advances in neural information processing systems, pp 52–63
  17. Huang H, Yu PS, Wang C (2018) An introduction to image synthesis with generative adversarial nets. ArXiv preprint arXiv:1803.04469 · arXiv
  18. Juefei-Xu F, Dey R, Bodetti V, Savvides M (2018) Rankgan: a maximum margin ranking gan for generating faces. In: Proceedings of the Asian conference on computer vision (ACCV), vol 4
  19. Kim T, Kim B, Cha M, Kim J (2017) Unsupervised visual attribute transfer with reconfigurable generative adversarial networks. ArXiv preprint arXiv:1707.09798 · arXiv
  20. Kingma DP, Dhariwal P (2018) Glow: Generative flow with invertible 1x1 convolutions. In: Advances in neural information processing systems, pp 10236–10245
  21. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: Proceedings of the 2nd international conference on learning representations (ICLR)
  22. Kortylewski A, Schneider A, Gerig T, Egger B, Morel-Forster A, Vetter T (2018) Training deep face recognition systems with synthetic data. ArXiv preprint arXiv:1802.05891 · arXiv
  23. Kulkarni TD, Whitney WF, Kohli P, Tenenbaum J (2015) Deep convolutional inverse graphics network. In: Advances in neural information processing systems, pp 2539–2547
  24. Lample G, Zeghidour N, Usunier N, Bordes A, Denoyer L et al (2017) Fader networks: manipulating images by sliding attributes. In: Advances in neural information processing systems, pp 5967–5976
  25. Larsen ABL, Sønderby SK, Larochelle H, Winther O (2015) Autoencoding beyond pixels using a learned similarity metric. In: 33rd International conference on machine learning
  26. Li M, Zuo W, Zhang D (2016) Deep identity-aware transfer of facial attributes. ArXiv preprint arXiv:1610.05586 · arXiv
  27. Liu L, Xing J, Liu S, Xu H, Zhou X, Yan S (2014) Wow! you are so beautiful today!. ACM Trans Multimed Comput Commun Appl (TOMM) 11(1s):20
  28. Liu MY, Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. In: Advances in neural information processing systems, pp 700–708
  29. Liu MY, Tuzel O (2016) Coupled generative adversarial networks. In: Advances in neural information processing systems, pp 469–477
  30. Makhzani A, Shlens J, Jaitly N, Goodfellow I, Frey B (2015) Adversarial autoencoders. ArXiv preprint arXiv:1511.05644 · arXiv
  31. Masi I, Trãn AT, Hassner T, Leksut JT, Medioni G (2016) Do we really need to collect millions of faces for effective face recognition? In: European conference on computer vision. Springer, pp 579–596
  32. Moniz JRA, Beckham C, Rajotte S, Honari S, Pal C (2018) Unsupervised depth estimation, 3d face rotation and replacement. In: Advances in neural information processing systems, pp 9759–9769
  33. Pham HX, Wang Y, Pavlovic V (2018) Generative adversarial talking head: Bringing portraits to life with a weakly supervised neural network. ArXiv preprint arXiv:1803.07716 · arXiv
  34. Qiao F, Yao N, Jiao Z, Li Z, Chen H, Wang H (2018) Geometry-contrastive gan for facial expression transfer. ArXiv preprint arXiv:1802.01822 · arXiv
  35. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. ArXiv preprint arXiv:1511.06434 · arXiv
  36. Sanchez E, Valstar M (2018) Triple consistency loss for pairing distributions in gan-based face synthesis. ArXiv preprint arXiv:1811.03492 · arXiv
  37. Shen Y, Zhou B, Luo P, Tang X (2018) Facefeat-gan: a two-stage approach for identity-preserving face synthesis. ArXiv preprint arXiv:1812.01288 · arXiv
  38. Sohn K, Lee H, Yan X (2015) Learning structured output representation using deep conditional generative models. In: Advances in neural information processing systems, pp 3483–3491
  39. Van den Oord A, Kalchbrenner N, Espeholt L, Vinyals O, Graves A et al (2016) Conditional image generation with pixelcnn decoders. In: Advances in neural information processing systems, pp 4790–4798
  40. Van Den Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. In: Proceedings of the 33rd international conference on international conference on machine learning-volume 48, ICML’16. JMLR.org, pp 1747–1756
  41. Volpi R, Namkoong H, Sener O, Duchi JC, Murino V, Savarese S (2018) Generalizing to unseen domains via adversarial data augmentation. In: Advances in neural information processing systems, pp 5334–5344
  42. Wang Z, Tang X, Luo W, Gao S (2018) Face aging with identity-preserved conditional generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7939–7947
  43. Xiao T, Hong J, Ma J (2018) Dna-gan: learning disentangled representations from multi-attribute images. In: International conference on learning representations workshop 2018
  44. Yeh R, Liu Z, Goldman DB, Agarwala A (2016) Semantic facial expression editing using autoencoded flow. ArXiv preprint arXiv:1611.09961 · arXiv
  45. Zhao J, Cheng Y, Cheng Y, Yang Y, Zhao F, Li J, Liu H, Yan S, Feng J (2019) Look across elapse: disentangled representation learning and photorealistic cross-age face synthesis for age-invariant face recognition. Proc AAAI Conf Artif Intell 33:9251–9258
  46. Zhao J, Xiong L, Jayashree PK, Li J, Zhao F, Wang Z, Pranata PS, Shen PS, Yan S, Feng J (2017) Dual-agent gans for photorealistic and identity preserving profile face synthesis. In: Advances in neural information processing systems, pp 66–76
  47. Zhu H, Zhou Q, Zhang J, Wang JZ (2018) Facial aging and rejuvenation by conditional multi-adversarial autoencoder with ordinal regression. ArXiv preprint arXiv:1804.02740 · arXiv
  48. Zhu J.Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232
  49. Zhu X, Lei Z, Yan J, Yi D, Li SZ (2015) High-fidelity pose and expression normalization for face recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 787–796

Cited by

See all 142

Metadata

Authors
Wang, X., Wang, K., & Lian, S.
Affiliations
China United Network Communications Group (China) (CN)
Corresponding author
Wang, Xiang
Venue
Neural Computing and Applications
Published
2020-03-17
Volume
32
Issue
19
Pages
15503-15531
ISSN
  • 0941-0643 (ISSN-L)
  • 1433-3058
OpenAlex
W2942077508
Cited by
141
References
104
Citations by year
2026: 9 · 2025: 13 · 2024: 26 · 2023: 31 · 2022: 20 · 2021: 28 · 2020: 12
FWCI
8.84
Type
journal article
Language
en
Open access
Licence
Locations
  • Neural Computing and Applications (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
generative adversarial networks, deep neural network, training data quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.