CoChat
Sign Up
Cited by 21Generative Adversarial Networks and Image SynthesisComputer Graphics and Visualization Techniques3D Shape Modeling and Analysis

Meta-Sim: Learning to Generate Synthetic Datasets

Open access
Kar, Amlan, Prakash, Aayush, Liu, Ming-Yu, Cameracci, Eric, Yuan, Justin, Rusiniak, Matt, Acuna, David, Torralba, Antonio +1 more·2019·2019 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2019.00465
Open this paper in CoChat — free accountOpen PDF

Abstract

Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our work is to automatically synthesize labeled datasets that are relevant for a downstream task. We propose Meta-Sim, which learns a generative model of synthetic scenes, and obtain images as well as its corresponding ground-truth via a graphics engine. We parametrize our dataset generator with a neural network, which learns to modify attributes of scene graphs obtained from probabilistic scene grammars, so as to minimize the distribution gap between its rendered outputs and target data. If the real dataset comes with a small labeled validation set, we additionally aim to optimize a meta-objective, i.e. downstream task performance. Experiments show that the proposed method can greatly improve content generation quality over a human-engineered probabilistic scene grammar, both qualitatively and quantitatively as measured by performance on a downstream task.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv.2019.00465

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [73]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [2][2] M. Bi´ nkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying mmd gans. ICLR , 2018. 4 -
  2. [3][3] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018. 3 - · arXiv
  3. [4][4] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. In arXiv:1606.01540 , 2016. 2 - · arXiv
  4. [6][6] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv:1810.05687 , 2018. 2 - · arXiv
  5. [8][8] G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In UAI , 2015. 4 -
  6. [9][9] G. French, M. Mackiewicz, and M. Fisher. Self-ensembling for visual domain adaptation. In ICLR , 2018. 1, 2 -
  7. [11][11] Y. Ganin, T. Kulkarni, I. Babuschkin, S. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118 , 2018. 2 - · arXiv
  8. [12][12] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In CVPR , 2012. 1, 7 -
  9. [13][13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS , 2014. 2 -
  10. [14][14] E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. JMLR , 5(Nov):1471-1530, 2004. 5 -
  11. [15][15] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch¨ olkopf, and A. Smola. A kernel two-sample test. JMLR , 2012. 2, 4 -
  12. [16][16] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017. 7, 11 -
  13. [17][17] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isol, K. S. A. A. Efros, and T. Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML , 2018. 1, 2 -
  14. [18][18] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image-to-image translation. In ECCV , 2018. 1, 2, 7 -
  15. [19][19] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948 , 2018. 3 - · arXiv
  16. [20][20] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In CVPR , 2018. 4 -
  17. [22][22] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017. 3 -
  18. [24][24] T. D. Kulkarni, P. Kohli, J. B. Tenenbaum, and V. Mansinghka. Picture: A probabilistic programming language for scene perception. In Proceedings of the ieee conference on computer vision and pattern recognition , pages 4390-4399, 2015. 2 -
  19. [26][26] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. P´ oczos. Mmdgan: Towards deeper understanding of moment matching network. In NIPS , 2017. 4 -
  20. [27][27] P. Li, X. Liang, D. Jia, and E. P. Xing. Semantic-aware gradgan for virtual-to-real urban scene adaption. In BMVC , 2018. 1, 2 -
  21. [28][28] T.-M. Li, M. Aittala, F. Durand, and J. Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) , 2018. 4 -
  22. [29][29] Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In ICML , 2015. 4 -
  23. [30][30] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-toimage translation networks. In NIPS , 2017. 1, 2 -
  24. [31][31] G. Louppe and K. Cranmer. Adversarial variational optimization of non-differentiable simulators. arXiv preprint arXiv:1707.07113 , 2017. 2 - · arXiv
  25. [32][32] V. K. Mansinghka, T. D. Kulkarni, Y. N. Perov, and J. Tenenbaum. Approximate bayesian image interpretation using generative probabilistic graphics programs. In Advances in Neural Information Processing Systems , pages 1520-1528, 2013. 2 -
  26. [33][33] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield. Structured domain randomization: Bridging the reality gap by contextaware synthetic data. In arXiv:1810.10093 , 2018. 1, 2, 7, 9 - · arXiv
  27. [34][34] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In CVPR , 2018. 2 -
  28. [35][35] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In ECCV , 2016. 1, 2 -
  29. [38][38] N. Ruiz, S. Schulter, and M. Chandraker. Learning to simulate. arXiv preprint arXiv:1810.02513 , 2018. 2 - · arXiv
  30. [39][39] F. Sadeghi and S. Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 , 2016. 2 - · arXiv
  31. [41][41] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016. 4, 10 -
  32. [43][43] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In Intl. Conf. on Intelligent Robots and Systems , 2012. 2 -
  33. [44][44] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured output space for semantic segmentation. In CVPR , 2018. 1 -
  34. [45][45] M. Wrenninge and J. Unger. Synscapes: A photorealistic synthetic dataset for street scene parsing. In arXiv:1810.08705 , 2018. 2, 7 - · arXiv
  35. [46][46] Y. Wu, Y. Wu, G. Gkioxari, and Y. Tiani. Building generalizable agents with a realistic and rich 3d environment. In arXiv:1801.02209 , 2018. 1, 2 - · arXiv
  36. [47][47] T. Yao, Y. Pan, Y. Li, and T. Mei. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 684699, 2018. 3, 9 -
  37. [48][48] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networkss. In ICCV , 2017. 1, 2 -
  38. [49][49] S.-C. Zhu, D. Mumford, et al. A stochastic grammar of images. Foundations and Trends R © in Computer Graphics and Vision , 2(4):259-362, 2007. 3 -
  39. [50][50] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In ICLR , 2017. 3 -
  40. [51][51] Y. Zou, Z. Yu, B. V. K. V . Kumar, and J. Wang. Domain adaptation for semantic segmentation via class-balanced selftraining. In ECCV , 2018. 1, 2

Cited by

Sorted by citations

Authors
Kar, A., Prakash, A., Liu, M.-Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., & Fidler, S.
Affiliations
  • Nvidia (United Kingdom) (GB)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2019-10
Pages
4550-4559
DOI
10.1109/iccv.2019.00465
OpenAlex
W2940868402
Cited by
21
References
8
Citations by year
2023: 2 · 2021: 9 · 2020: 5 · 2019: 5
FWCI
1.55
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
Keywords
synthetic data generation, synthetic data, generative models, scene graph, downstream task performance, computer graphics rendering, distribution gap, content creation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review