CoChat
Sign Up
Cited by 73Generative Adversarial Networks and Image SynthesisComputer Graphics and Visualization TechniquesArtificial Intelligence in Games

Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation

Open access
Devaranjan, Jeevan, Kar, Amlan, Fidler, Sanja·2020·Lecture Notes in Computer Science
DOI 10.1007/978-3-030-58520-4_42
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-030-58520-4_42

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [74]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [1]1. Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al.: Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 (2019) 14 · arXiv
  2. [2]2. Alhaija, H.A., Mustikovela, S.K., Mescheder, L., Geiger, A., Rother, C.: Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal of Computer Vision 126 (9), 961-972 (2018) 3
  3. [4]4. Armeni, I., He, Z.Y., Gwak, J., Zamir, A.R., Fischer, M., Malik, J., Savarese, S.: 3d scene graph: A structure for unified semantics, 3d space, and camera. In: Proceedings of the IEEE International Conference on Computer Vision (2019) 3
  4. [5]5. Bi´ nkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying mmd gans. ICLR (2018) 12, 14
  5. [6]6. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. In: arXiv:1606.01540 (2016) 3 · arXiv
  6. [8]8. Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., Fox, D.: Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv:1810.05687 (2018) 4 · arXiv
  7. [9]9. Chen, X., Liu, C., Song, D.: Tree-to-tree neural networks for program translation. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 2547-2557. Curran Associates, Inc. (2018), http://papers.nips.cc/paper/ 7521-tree-to-tree-neural-networks-for-program-translation.pdf 4
  8. [10]10. Chu, H., Li, D., Acuna, D., Kar, A., Shugrina, M., Wei, X., Liu, M.Y., Torralba, A., Fidler, S.: Neural turtle graphics for modeling city road layouts. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4522-4530 (2019) 4
  9. [11]11. Cranmer, K., Brehmer, J., Louppe, G.: The frontier of simulation-based inference. arXiv preprint arXiv:1911.01429 (2019) 4 · arXiv
  10. [12]12. Dai, H., Tian, Y., Dai, B., Skiena, S., Song, L.: Syntax-directed variational autoencoder for structured data. arXiv preprint arXiv:1802.08786 (2018) 2, 4 · arXiv
  11. [13]13. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A largescale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. Ieee (2009) 12
  12. [14]14. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: CORL. pp. 1-16 (2017) 3
  13. [15]15. Dziugaite, G.K., Roy, D.M., Ghahramani, Z.: Training generative neural networks via maximum mean discrepancy optimization. In: UAI (2015) 7
  14. [16]16. Eslami, S.A., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Hinton, G.E., et al.: Attend, infer, repeat: Fast scene understanding with generative models. In: Advances in Neural Information Processing Systems. pp. 3225-3233 (2016) 3
  15. [17]17. Fan, S., Huang, B.: Labeled graph generative adversarial networks. CoRR abs/1906.03220 (2019), http://arxiv.org/abs/1906.03220 2, 4 · arXiv
  16. [19]19. Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118 (2018) 4, 7 · arXiv
  17. [20]20. Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., Zhu, S.: Vrkitchen: an interactive 3d virtual environment for task-oriented learning. arXiv abs/1903.05757 (2019) 3 · arXiv
  18. [21]21. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In: CVPR (2012) 9, 11, 19
  19. [22]22. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014) 7
  20. [23]23. Gretton, A., Borgwardt, K.M., Rasch, M.J., Sch¨ olkopf, B., Smola, A.: A kernel two-sample test. JMLR (2012) 7, 9, 10
  21. [24]24. Handa, A., Patraucean, V., Badrinarayanan, V., Stent, S., Cipolla, R.: Understanding real world indoor scenes with synthetic data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4077-4085 (2016) 3
  22. [25]25. He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961-2969 (2017) 12, 13, 19
  23. [29]29. Jyothi, A.A., Durand, T., He, J., Sigal, L., Mori, G.: Layoutvae: Stochastic scene layout generation from a label set. In: The IEEE International Conference on Computer Vision (ICCV) (October 2019) 3
  24. [30]30. Kar, A., Prakash, A., Liu, M.Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., Fidler, S.: Meta-sim: Learning to generate synthetic datasets. In: ICCV (2019) 2, 3, 5, 6, 7, 9, 11, 12, 13, 14, 18, 19
  25. [32]32. Kim, Y., Dyer, C., Rush, A.M.: Compound probabilistic context-free grammars for grammar induction. vol. abs/1906.10225 (2019), http://arxiv.org/abs/1906. 10225 4 · arXiv
  26. [33]33. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013) 7 · arXiv
  27. [35]35. Kulkarni, T.D., Kohli, P., Tenenbaum, J.B., Mansinghka, V.: Picture: A probabilistic programming language for scene perception. In: Proceedings of the ieee conference on computer vision and pattern recognition. pp. 4390-4399 (2015) 4
  28. [36]36. Kulkarni, T.D., Whitney, W.F., Kohli, P., Tenenbaum, J.: Deep convolutional inverse graphics network. In: NIPS. pp. 2539-2547 (2015) 7
  29. [37]37. Kusner, M.J., Paige, B., Hern´ andez-Lobato, J.M.: Grammar variational autoencoder. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70. pp. 1945-1954. ICML'17, JMLR.org (2017), http://dl.acm.org/ citation.cfm?id=3305381.3305582 2, 4, 6, 11
  30. [39]39. Li, C.L., Chang, W.C., Cheng, Y., Yang, Y., P´ oczos, B.: Mmd gan: Towards deeper understanding of moment matching network. In: NIPS (2017) 7
  31. [40]40. Li, M., Patil, A.G., Xu, K., Chaudhuri, S., Khan, O., Shamir, A., Tu, C., Chen, B., Cohen-Or, D., Zhang, H.: Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphics (TOG) 38 (2), 12 (2019) 3
  32. [41]41. Li, Y., Swersky, K., Zemel, R.: Generative moment matching networks. In: ICML (2015) 7
  33. [42]42. Li, Y., Vinyals, O., Dyer, C., Pascanu, R., Battaglia, P.: Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324 (2018) 2, 4 · arXiv
  34. [43]43. Liao, R., Li, Y., Song, Y., Wang, S., Nash, C., Hamilton, W.L., Duvenaud, D., Urtasun, R., Zemel, R.S.: Efficient graph generation with graph recurrent attention networks. arXiv preprint arXiv:1910.00760 (2019) 4 · arXiv
  35. [44]44. Louppe, G., Cranmer, K.: Adversarial variational optimization of non-differentiable simulators. arXiv preprint arXiv:1707.07113 (2017) 4 · arXiv
  36. [45]45. Mansinghka, V.K., Kulkarni, T.D., Perov, Y.N., Tenenbaum, J.: Approximate bayesian image interpretation using generative probabilistic graphics programs. In: Advances in Neural Information Processing Systems. pp. 1520-1528 (2013) 4
  37. [46]46. McCormac, J., Handa, A., Leutenegger, S., Davison, A.J.: Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth. arXiv preprint arXiv:1612.05079 (2016) 1, 3 · arXiv
  38. [48]48. Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., Shapira, O., Birchfield, S.: Structured domain randomization: Bridging the reality gap by context-aware synthetic data. In: arXiv:1810.10093 (2018) 3, 6, 12, 13, 14, 17 · arXiv
  39. [49]49. Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., Torralba, A.: Virtualhome: Simulating household activities via programs. In: CVPR (2018) 3
  40. [50]50. Qi, S., Zhu, Y., Huang, S., Jiang, C., Zhu, S.C.: Human-centric indoor scene synthesis using stochastic grammar. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5899-5908 (2018) 3
  41. [51]51. Razavi, A., Oord, A.v.d., Vinyals, O.: Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv:1906.00446 (2019) 5 · arXiv
  42. [52]52. Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082 (2014) 7 · arXiv
  43. [53]53. Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: ECCV (2016) 1, 3
  44. [54]54. Ritchie, D., Wang, K., Lin, Y.A.: Fast and flexible indoor scene synthesis via deep convolutional generative models. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019) 3
  45. [56]56. Sadeghi, F., Levine, S.: Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 (2016) 3 · arXiv
  46. [57]57. Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al.: Habitat: A platform for embodied ai research. arXiv preprint arXiv:1904.01201 (2019) 1, 3 · arXiv
  47. [58]58. Shugrina, M., Liang, Z., Kar, A., Li, J., Singh, A., Singh, K., Fidler, S.: Creative flow+ dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5384-5393 (2019) 3
  48. [59]59. Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition (2017) 3
  49. [60]60. Such, F.P., Rawal, A., Lehman, J., Stanley, K.O., Clune, J.: Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. arXiv preprint arXiv:1912.07768 (2019) 3 · arXiv
  50. [61]61. Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., Riedmiller, M.: DeepMind control suite. Tech. rep., DeepMind (Jan 2018), https://arxiv.org/abs/1801. 00690 1, 3
  51. [63]63. Todorov, E., Erez, T., Tassa, Y.: Mujoco: A physics engine for model-based control. In: Intl. Conf. on Intelligent Robots and Systems (2012) 3
  52. [64]64. Wang, K., Lin, Y.A., Weissmann, B., Savva, M., Chang, A.X., Ritchie, D.: Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics (TOG) 38 (4), 132 (2019) 3
  53. [65]65. Wang, K., Savva, M., Chang, A.X., Ritchie, D.: Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphics (TOG) 37 (4), 70 (2018) 3
  54. [68]68. Wrenninge, M., Unger, J.: Synscapes: A photorealistic synthetic dataset for street scene parsing. In: arXiv:1810.08705 (2018) 3 · arXiv
  55. [69]69. Wu, J., Tenenbaum, J.B., Kohli, P.: Neural scene de-rendering. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 4
  56. [70]70. Wu, Y., Wu, Y., Gkioxari, G., Tiani, Y.: Building generalizable agents with a realistic and rich 3d environment. In: arXiv:1801.02209 (2018) 3 · arXiv
  57. [71]71. Yin, P., Neubig, G.: A syntactic neural model for general-purpose code generation. CoRR abs/1704.01696 (2017), http://arxiv.org/abs/1704.01696 4 · arXiv
  58. [72]72. Yin, P., Zhou, C., He, J., Neubig, G.: Structvae: Tree-structured latent variable models for semi-supervised semantic parsing. CoRR abs/1806.07832 (2018), http://arxiv.org/abs/1806.07832 4 · arXiv
  59. [73]73. You, J., Ying, R., Ren, X., Hamilton, W., Leskovec, J.: Graphrnn: Generating realistic graphs with deep auto-regressive models. In: International Conference on Machine Learning. pp. 5694-5703 (2018) 4
  60. [74]74. Yu, L.F., Yeung, S.K., Tang, C.K., Terzopoulos, D., Chan, T.F., Osher, S.: Make it home: automatic optimization of furniture arrangement. ACM Trans. Graph. 30 (4), 86 (2011) 3
  61. [75]75. Zhang, Y., Song, S., Yumer, E., Savva, M., Lee, J.Y., Jin, H., Funkhouser, T.: Physically-based rendering for indoor scene understanding using convolutional neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017) 3
  62. [76]76. Zhou, Y., While, Z., Kalogerakis, E.: Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In: The IEEE International Conference on Computer Vision (ICCV) (October 2019) 3

Cited by

Sorted by citations

Authors
Devaranjan, J., Kar, A., & Fidler, S.
Affiliations
  • University of Waterloo (CA)
  • University of Toronto (CA)
Corresponding author
Kar, Amlan
ORCID
Fidler, Sanja
Venue
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Published
2020
Pages
715-733
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-030-58520-4_42
OpenAlex
W3050584442
Cited by
73
References
10
Citations by year
2026: 1 · 2025: 8 · 2024: 11 · 2023: 20 · 2022: 18 · 2021: 15
FWCI
4.09
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
procedural scene generation, unsupervised learning, synthetic data generation, reinforcement learning, parameter tuning, object detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review