CoChat
Sign Up
Cited by 259Generative Adversarial Networks and Image SynthesisAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Open access
Zhang, Yuxuan, Ling, Huan, Gao, Jun, Yin, Kangxue, Lafleche, Jean-Francois, Barriuso, Adela, Torralba, Antonio, Fidler, Sanja·2021·2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr46437.2021.01001
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort. Current deep networks are extremely data-hungry, benefiting from training on large-scale datasets, which are time consuming to annotate. Our method relies on the power of recent GANs to generate realistic images. We show how the GAN latent code can be decoded to produce a semantic segmentation of the image. Training the decoder only needs a few labeled examples to generalize to the rest of the latent space, resulting in an infinite annotated dataset generator! These generated datasets can then be used for training any computer vision architecture just as real datasets are. As only a few images need to be manually segmented, it becomes possible to annotate images in extreme detail and generate datasets with rich object and part segmentations. To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts. Our approach outperforms all semi-supervised baselines significantly and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as our method.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr46437.2021.01001

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [71]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15535-15545, 2019.
  2. David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019. 1, 2
  3. Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Gen- erating visual representations for zero-shot classification. In Proceedings of the IEEE International Conference on Com- puter Vision Workshops, pages 2666-2673, 2017. 2
  4. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic im- age segmentation with deep convolutional nets, atrous con- volution, and fully connected crfs. IEEE trans. on pattern analysis and machine intelligence, 40(4):834-848, 2017. 5, 7
  5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020. 2 · arXiv
  6. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised mod- els are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33, 2020. 1, 2
  7. Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learn- ing to predict 3d objects with an interpolation-based differ- entiable renderer. In Advances In Neural Information Pro- cessing Systems, 2019. 7, 8
  8. Jaehoon Choi, Taekyung Kim, and Changick Kim. Self- ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In ICCV, pages 6830- 6840, 2019. 2
  9. Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3223, 2016. 4
  10. Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta- sim2: Unsupervised learning of scene structure for synthetic data generation. In ECCV, 2020. 2
  11. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) chal- lenge. International Journal of Computer Vision, 88(2):303- 338, June 2010. 6
  12. Rafael Felix, Vijay BG Kumar, Ian Reid, and Gustavo Carneiro. Multi-modal cycle-consistent generalized zero- shot learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 21-37, 2018. 2
  13. Danil Galeev, Konstantin Sofiiuk, Danila Rukhovich, Mikhail Romanov, Olga Barinova, and Anton Konushin. Learning high-resolution domain-specific representations with a gan generator. arXiv preprint arXiv:2006.10451, 2020. 2 · arXiv
  14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, pages 2672-2680, 2014. 2
  15. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020. 2
  16. Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensional- ity reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pat- tern Recognition (CVPR'06), volume 2, pages 1735-1742. IEEE, 2006. 2
  17. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020. 2
  18. Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceed- ings of the IEEE International Conference on Computer Vi- sion, pages 1501-1510, 2017. 3
  19. Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang. Adversarial learning for semi-supervised semantic segmentation. arXiv preprint arXiv:1802.07934, 2018. 2 · arXiv
  20. Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 9865-9874, 2019. 1, 2
  21. Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. In ICCV, 2019. 2
  22. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. 2, 3 · arXiv
  23. Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Ryn- son WH Lau. Guided collaborative training for pixel-wise semi-supervised learning. arXiv preprint arXiv:2008.05258, 2020. 2 · arXiv
  24. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013. 6
  25. Weicheng Kuo, Christian Häne, E. Yuh, P. Mukherjee, and Jitendra Malik. Cost-sensitive active learning for intracranial hemorrhage detection. In MICCAI, 2018. 4, 7
  26. Daiqing Li, Amlan Kar, Nishant Ravikumar, Alejandro F Frangi, and Sanja Fidler. Federated simulation for medical imaging. In MICCAI, 2020. 2
  27. Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In CVPR, 2021. 2
  28. Tsung-Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. ArXiv, abs/1405.0312, 2014. 5 · arXiv
  29. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 4
  30. Huan Ling, David Acuna, Karsten Kreis, Seung Kim, and Sanja Fidler. Variational amodal object completion for inter- active scene editing. In NeurIPS, 2020. 2
  31. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), De- cember 2015. 6
  32. Yang Long, Li Liu, Fumin Shen, Ling Shao, and Xuelong Li. Zero-shot learning using synthesised unseen visual data with diffusion regularisation. IEEE transactions on pattern analysis and machine intelligence, 40(10):2498-2512, 2017. 2
  33. Prem Melville, Stewart M. Yang, Maytal Saar-Tsechansky, and Raymond Mooney. Active learning for probability esti- mation using jensen-shannon divergence. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo, editors, Machine Learning: ECML 2005, pages 268- 279, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. 4
  34. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6707-6717, 2020. 1, 2
  35. Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmentation with high-and low- level consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. 1, 2, 5, 6
  36. Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ra- mamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 4500-4509, 2018. 2
  37. Bryan Russell, Antonio Torralba, Kevin Murphy, and William Freeman. Labelme: A database and web-based tool for image annotation. International Journal of Computer Vi- sion, 77(05), 2008. 4
  38. Mert Bulent Sariyildiz and Ramazan Gokberk Cinbis. Gra- dient matching generative networks for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2168-2178, 2019. 2
  39. O. Sener and S. Savarese. A geometric approach to ac- tive learning for convolutional neural networks. ArXiv, abs/1708.00489, 2017. 7 · arXiv
  40. Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In NeurIPS, 2020. 1, 2
  41. Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative ad- versarial network. In Proceedings of the IEEE International Conference on Computer Vision, pages 5688-5696, 2017. 2
  42. Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pages 1195-1204, 2017. 1, 2
  43. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. arXiv preprint arXiv:1906.05849, 2019. 1, 2 · arXiv
  44. G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 595-604, 2015. 5
  45. Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2517-2526, 2019. 2
  46. Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detec- tion -how to effectively exploit shape and texture features. In David Forsyth, Philip Torr, and Andrew Zisserman, edi- tors, Computer Vision -ECCV 2008, pages 802-816, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. 6
  47. Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Im- age gans meet differentiable rendering for inverse graph- ics and interpretable 3d neural rendering. arXiv preprint arXiv:2010.09125, 2020. 2, 3, 7, 8 · arXiv
  48. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understand- ing of scenes through the ade20k dataset. arXiv preprint arXiv:1608.05442, 2016. 6 · arXiv
  49. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 4, 6
  50. Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic seg- mentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289-305, 2018.

Cited by

Sorted by citations

Authors
Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.-F., Barriuso, A., Torralba, A., & Fidler, S.
Affiliations
  • Nvidia (United Kingdom) (GB)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2021-06
Pages
10140-10150
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr46437.2021.01001
OpenAlex
W3167788848
Cited by
259
References
6
Citations by year
2026: 20 · 2025: 33 · 2024: 79 · 2023: 67 · 2022: 48 · 2021: 12
FWCI
16.84
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
synthetic image generation, generative adversarial networks, semantic segmentation, semi-supervised learning, data augmentation, object segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review