CoChat
Sign Up
Cited by 83Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningGenerative Adversarial Networks and Image Synthesis

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

Open access
Li, Daiqing, Ling, Huan, Kim, Seung Wook, Kreis, Karsten, Fidler, Sanja, Torralba, Antonio·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52688.2022.02064
Open this paper in CoChat — free accountOpen PDF

Abstract

Annotating images with pixel-wise labels is a time consuming and costly process. Recently, DatasetGAN [78] showcased a promising alternative - to synthesize a large labeled dataset via a generative adversarial network (GAN) by exploiting a small set of manually labeled, GAN generated images. Here, we scale DatasetGAN to ImageNet scale of class diversity. We take image samples from the class-conditional generative model BigGAN [5] trained on ImageNet, and manually annotate only 5 images per class, for all 1k classes. By training an effective feature segmentation architecture on top of BigGAN, we turn Big GAN into a labeled dataset generator. We further show that VQGAN [18] can similarly serve as a dataset generator, leveraging the already annotated data. We create a new ImageNet benchmark by labeling an additional set of real images and evaluate segmentation performance in a variety of settings. Through an extensive ablation study, we show big gains in leveraging a large generated dataset to train different supervised and self-supervised backbone models on pixel-wise tasks. Furthermore, we demonstrate that using our synthesized datasets for pre-training leads to improvements over standard ImageNet pre-training on several downstream datasets, such as PASCAL-VOC, MS-COCO, Cityscapes and chest X-ray, as well as tasks (detection, segmentation). Our benchmark will be made public and maintain a leaderboard for this challenging task. Project Page: https://nv-tlabs.github.io/big-datasetgan/

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.02064

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [72]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. [1][1] David Acuna, Amlan Kar, and Sanja Fidler. Devil is in the edges: Learning semantic boundaries from noisy annota- tions. In CVPR, 2019.
  2. [2][2] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Ef- ficient annotation of segmentation datasets with polygon- rnn++. In CVPR, 2018.
  3. [3][3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. · arXiv
  4. [5][5] Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In Interna- tional Conference on Learning Representations, 2018.
  5. [8][8] Liang-Chieh Chen, Sanja Fidler, Alan Yuille, and Raquel Ur- tasun. Beat the mturkers: Automatic image labeling from weak 3d supervision. In CVPR, 2014.
  6. [9][9] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587, 2017. 11 Figure 13. Examples from the Real-annotated dataset. We visualize both the segmentation masks as well as the boundary polygons. 12 Figure 14. Examples from the Synthetic-annotated dataset. We visualize both the segmentation masks as well as the boundary polygons. 13 Figure 15. BigGAN-sim random samples. We visualize both the segmentation masks as well as the boundary polygons. 14 Figure 16. VQGAN-sim random samples. We visualize both the segmentation masks as well as the boundary polygons. 15 Figure 17. BigGAN-sim per-class samples. We visualize both the segmentation masks as well as the boundary polygons. 16 Figure 18. VQGAN-sim per-class samples. We visualize both the segmentation masks as well as the boundary polygons. 17 Figure 19. BigGAN-sim mean shapes. Mean shapes are calculated using k-means clustering over normalized segmentation masks. 18 · arXiv
  7. [15][15] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta- sim2: Unsupervised learning of scene structure for synthetic data generation. In ECCV, 2020.
  8. [18][18] David H Douglas and Thomas K Peucker. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: the interna- tional journal for geographic information and geovisualiza- tion, 10(2):112–122, 1973.
  9. [20][20] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010.
  10. [21][21] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) chal- lenge. International Journal of Computer Vision, 88(2):303– 338, June 2010.
  11. [23][23] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-object tracking anal- ysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340–4349, 2016.
  12. [25][25] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea- ture hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580–587, 2014.
  13. [27][27] Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmentation propagation. International Journal of Computer Vision, 110(3):328–348, 2014.
  14. [29][29] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir- shick. Mask r-cnn. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, 2017.
  15. [33][33] Xun Huang, Ming-Yu Liu, Serge J. Belongie, and Jan Kautz. Multimodal unsupervised image-to-image transla- tion. CoRR, abs/1804.04732, 2018. · arXiv
  16. [36][36] Stefan Jaeger, Sema Candemir, Sameer Antani, Yı̀-Xiáng J Wáng, Pu-Xuan Lu, and George Thoma. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery, 4(6):475, 2014.
  17. [37][37] Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. In ICCV, 2019.
  18. [45][45] Viveka Kulharia, Siddhartha Chandra, Amit Agrawal, Philip Torr, and Ambrish Tyagi. Box2seg: Attention weighted loss and discriminative feature learning for weakly supervised segmentation. In ECCV, 2020.
  19. [46][46] Weicheng Kuo, Christian Häne, Esther Yuh, Pratik Mukher- jee, and Jitendra Malik. Cost-sensitive active learning for intracranial hemorrhage detection. In MICCAI, 2018.
  20. [47][47] Shiyi Lan, Zhiding Yu, Christopher Bongsoo Choy, Sub- hashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry Davis, and Anima Anandkumar. Discobox: Weakly supervised in- stance segmentation and semantic correspondence from box supervision. ArXiv, abs/2105.06464, 2021. · arXiv
  21. [48][48] Daiqing Li, Amlan Kar, Nishant Ravikumar, Alejandro F Frangi, and Sanja Fidler. Federated simulation for medical imaging. In MICCAI, 2020.
  22. [50][50] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, 2016.
  23. [53][53] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler. Fast interactive object annotation with curve-gcn. In CVPR, 2019.
  24. [54][54] Ce Liu, Jenny Yuen, and Antonio Torralba. Nonparamet- ric scene parsing: Label transfer via dense scene alignment. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1972–1979. IEEE, 2009.
  25. [56][56] K.K. Maninis, S. Caelles, J. Pont-Tuset, and L. Van Gool. Deep extreme cut: From extreme points to object segmenta- tion. In Computer Vision and Pattern Recognition (CVPR), 2018.
  26. [57][57] Prem Melville, Stewart M. Yang, Maytal Saar-Tsechansky, and Raymond Mooney. Active learning for probability esti- mation using jensen-shannon divergence. In João Gama, Rui Camacho, Pavel B. Brazdil, Alı́pio Mário Jorge, and Luı́s Torgo, editors, Machine Learning: ECML 2005, pages 268– 279, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.
  27. [59][59] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017. · arXiv
  28. [61][61] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018.
  29. [64][64] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pages 102–118. Springer, 2016.
  30. [66][66] A. Rosenfeld and D. Weinshall. Extracting foreground masks towards object recognition. In ICCV, 2021.
  31. [67][67] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. " grabcut" interactive foreground extraction using iter- ated graph cuts. ACM transactions on graphics (TOG), 23(3):309–314, 2004.
  32. [69][69] Junji Shiraishi, Shigehiko Katsuragawa, Junpei Ikezoe, Tsu- neo Matsumoto, Takeshi Kobayashi, Ken-ichi Komatsu, Mitate Matsui, Hiroshi Fujita, Yoshie Kodera, and Kunio Doi. Development of a digital image database for chest radiographs with and without a lung nodule: receiver op- erating characteristic analysis of radiologists' detection of pulmonary nodules. American Journal of Roentgenology, 174(1):71–74, 2000.
  33. [70][70] Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
  34. [71][71] Sergii Stirenko, Yuriy Kochura, Oleg Alienin, Oleksandr Rokovyi, Yuri Gordienko, Peng Gang, and Wei Zeng. Chest x-ray analysis of tuberculosis by deep learning with segmen- tation and augmentation. In 2018 IEEE 38th International Conference on Electronics and Nanotechnology (ELNANO), pages 422–428. IEEE, 2018.
  35. [72][72] Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Normalized cut loss for weakly-supervised cnn segmentation. In CVPR, 2018. 20
  36. [73][73] J. Tighe and S. Lazebnik. Superparsing: Scalable nonpara- metric image parsing with superpixels. In ECCV, 2010.
  37. [75][75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
  38. [77][77] Zian Wang, David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Object instance annotation with deep extreme level set evolution. In CVPR, 2019.
  39. [80][80] Jianjin Xu and Changxi Zheng. Linear semantics in genera- tive adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9351–9360, 2021.
  40. [81][81] Matthew D. Zeiler and Rob Fergus. Visualizing and under- standing convolutional networks. arxiv, 1311.2901, 2013. · arXiv
  41. [82][82] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021. 21 https://github.com/facebookresearch/detectron2 https://github.com/facebookresearch/detectron2

Cited by

Sorted by citations

Authors
Li, D., Ling, H., Kim, S. W., Kreis, K., Fidler, S., & Torralba, A.
Affiliations
  • Nvidia (United Kingdom) (GB)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
21298-21308
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52688.2022.02064
OpenAlex
W4313186194
Cited by
83
References
13
Citations by year
2026: 6 · 2025: 23 · 2024: 35 · 2023: 18 · 2022: 1
FWCI
6.98
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
BigGAN, VQGAN, pixel-wise annotation, image segmentation, ImageNet, self-supervised pre-training, generative adversarial networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review