CoChat
Sign Up
PreprintCited by 143D Shape Modeling and AnalysisGenerative Adversarial Networks and Image SynthesisComputer Graphics and Visualization Techniques

Deep Generative Models on 3D Representations: A Survey

Open access
Shi, Zifan, Peng, Sida, Xu, Yinghao, Geiger, Andreas, Liao, Yiyi, Shen, Yujun·2022·arXiv
DOI 10.48550/arxiv.2210.15663arXiv 2210.15663
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative models aim to learn the distribution of observed data by generating new instances. With the advent of neural networks, deep generative models, including variational autoencoders (VAEs), generative adversarial networks (GANs), and diffusion models (DMs), have progressed remarkably in synthesizing 2D images. Recently, researchers started to shift focus from 2D to 3D space, considering that 3D data is more closely aligned with our physical world and holds immense practical potential. However, unlike 2D images, which possess an inherent and efficient representation (\textit{i.e.}, a pixel grid), representing 3D data poses significantly greater challenges. Ideally, a robust 3D representation should be capable of accurately modeling complex shapes and appearances while being highly efficient in handling high-resolution data with high processing speeds and low memory requirements. Regrettably, existing 3D representations, such as point clouds, meshes, and neural fields, often fail to satisfy all of these requirements simultaneously. In this survey, we thoroughly review the ongoing developments of 3D generative models, including methods that employ 2D and 3D supervision. Our analysis centers on generative models, with a particular focus on the representations utilized in this context. We believe our survey will help the community to track the field's evolution and to spark innovative ideas to propel progress towards solving this challenging task.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.15663

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 201

  1. [2][2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' in NeurIPS , 2012. -
  2. [3][3] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
  3. [4][4] R. Girshick, 'Fast r-cnn,' in ICCV , 2015. -
  4. [5][5] S. Ren, K. He, R. Girshick, and J. Sun, 'Faster r-cnn: Towards realtime object detection with region proposal networks,' NeurIPS , 2015. -
  5. [6][6] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick, 'Mask r-cnn,' in ICCV , 2017. -
  6. [7][7] M. Oechsle, S. Peng, and A. Geiger, 'Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction,' in ICCV , 2021. -
  7. [8][8] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, 'Volume rendering of neural implicit surfaces,' NeurIPS , 2021. -
  8. [9][9] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, 'Nerf: Representing scenes as neural radiance fields for view synthesis,' in ECCV , 2020. -
  9. [10][10] D. Xu, D. Anguelov, and A. Jain, 'Pointfusion: Deep sensor fusion for 3d bounding box estimation,' in CVPR , 2018. -
  10. [11][11] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan et al. , 'Argoverse: 3d tracking and forecasting with rich maps,' in CVPR , 2019. -
  11. [13][13] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, 'Zero-shot text-to-image generation,' in ICML , 2021. -
  12. [14][14] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , 2022. - · arXiv
  13. [15][15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in NeurIPS , 2014. -
  14. [16][16] D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' in ICLR , 2014. -
  15. [17][17] J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' NeurIPS , 2020. -
  16. [19][19] Y. Xu, Y. Shen, J. Zhu, C. Yang, and B. Zhou, 'Generative hierarchical features from synthesizing images,' in CVPR , 2021. -
  17. [20][20] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, 'Image-to-image translation with conditional adversarial networks,' in CVPR , 2017. -
  18. [21][21] Y. Shen, C. Yang, X. Tang, and B. Zhou, 'Interfacegan: Interpreting the disentangled face representation learned by gans,' TPAMI , 2020. -
  19. [22][22] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes et al. , 'Photorealistic text-to-image diffusion models with deep language understanding,' arXiv preprint arXiv:2205.11487 , 2022. - · arXiv
  20. [23][23] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan et al. , 'Scaling autoregressive models for content-rich text-to-image generation,' arXiv preprint arXiv:2206.10789 , 2022. - · arXiv
  21. [24][24] Y. Bengio, R. Ducharme, and P. Vincent, 'A neural probabilistic language model,' NeurIPS , 2000. -
  22. [25][25] D. Rezende and S. Mohamed, 'Variational inference with normalizing flows,' in ICML , 2015. -
  23. [26][26] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum, 'Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling,' in NeurIPS , 2016. -
  24. [27][27] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, 'Learning representations and generative models for 3d point clouds,' in ICML , 2018. -
  25. [28][28] Z. Chen and H. Zhang, 'Learning implicit fields for generative shape modeling,' in CVPR , 2019. -
  26. [29][29] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, 'Graf: Generative radiance fields for 3d-aware image synthesis,' NeurIPS , 2020. -
  27. [30][30] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, 'pi-gan: Periodic implicit generative adversarial networks for 3daware image synthesis,' in CVPR , 2021. -
  28. [31][31] M. Niemeyer and A. Geiger, 'Giraffe: Representing scenes as compositional generative neural feature fields,' in CVPR , 2021. -
  29. [32][32] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, 'Pointnet: Deep learning on point sets for 3d classification and segmentation,' in CVPR , 2017. -
  30. [33][33] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, 'Pointnet++: Deep hierarchical feature learning on point sets in a metric space,' NeurIPS , 2017. -
  31. [34][34] A. Sinha, A. Unmesh, Q. Huang, and K. Ramani, 'Surfnet: Generating 3d shape surfaces using deep residual networks,' in CVPR , 2017. -
  32. [35][35] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, 'Pixel2mesh: Generating 3d mesh models from single rgb images,' in ECCV , 2018. -
  33. [36][36] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, '3d shapenets: A deep representation for volumetric shapes,' in CVPR , 2015. -
  34. [37][37] D. Maturana and S. Scherer, 'Voxnet: A 3d convolutional neural network for real-time object recognition,' in IROS , 2015. -
  35. [38][38] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, 'Stereo magnification: Learning view synthesis using multiplane images,' in SIGGRAPH , 2018. -
  36. [40][40] S. Bond-Taylor, A. Leach, Y. Long, and C. G. Willcocks, 'Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models,' arXiv preprint arXiv:2103.04922 , 2021. - · arXiv
  37. [41][41] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi et al. , 'Advances in neural rendering,' in CGF , 2022. -
  38. [42][42] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, 'Deep learning for 3d point clouds: A survey,' TPAMI , 2020. -
  39. [43][43] J. Lahoud, J. Cao, F. S. Khan, H. Cholakkal, R. M. Anwer, S. Khan, and M.-H. Yang, '3d vision with transformers: A survey,' arXiv preprint arXiv:2208.04309 , 2022. - · arXiv
  40. [48][48] A. Jain, B. Mildenhall, J. T. Barron, P. Abbeel, and B. Poole, 'Zeroshot text-guided object generation with dream fields,' in CVPR , 2022. -
  41. [49][49] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, 'Dreamfusion: Text-to-3d using 2d diffusion,' arXiv preprint arXiv:2209.14988 , 2022. - · arXiv
  42. [50][50] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, 'Magic3d: Highresolution text-to-3d content creation,' in CVPR , 2023. -
  43. [51][51] H. Kato, D. Beker, M. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon, 'Differentiable rendering: A survey,' arXiv preprint arXiv:2006.12057 , 2020. - · arXiv
  44. [53][53] A. Radford, L. Metz, and S. Chintala, 'Unsupervised representation learning with deep convolutional generative adversarial networks,' in ICLR , 2016. -
  45. [54][54] T. Karras, T. Aila, S. Laine, and J. Lehtinen, 'Progressive growing of gans for improved quality, stability, and variation,' in ICLR , 2018. -
  46. [55][55] T. Karras, S. Laine, and T. Aila, 'A style-based generator architecture for generative adversarial networks,' in CVPR , 2019. -
  47. [56][56] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, 'Analyzing and improving the image quality of StyleGAN,' in CVPR , 2020. -
  48. [57][57] T. Karras, M. Aittala, S. Laine, E. H¨ ark¨ onen, J. Hellsten, J. Lehtinen, and T. Aila, 'Alias-free generative adversarial networks,' in NeurIPS , 2021. -
  49. [58][58] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, 'Variational inference: A review for statisticians,' J. of the American statis. Associat. , 2017. -
  50. [59][59] Y. Xu, S. Peng, C. Yang, Y. Shen, and B. Zhou, '3d-aware image synthesis via learning structural and textural representations,' in CVPR , 2022. -
  51. [60][60] K. Schwarz, A. Sauer, M. Niemeyer, Y. Liao, and A. Geiger, 'Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids,' in NeurIPS , 2022. -
  52. [61][61] P. Mittal, Y.-C. Cheng, M. Singh, and S. Tulsiani, 'Autosdf: Shape priors for 3d completion, reconstruction and generation,' in CVPR , 2022. -
  53. [62][62] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese, '3dr2n2: A unified approach for single and multi-view 3d object reconstruction,' in ECCV , 2016. -
  54. [64][64] M. Tatarchenko, A. Dosovitskiy, and T. Brox, 'Octree generating networks: Efficient convolutional architectures for highresolution 3d outputs,' in ICCV , 2017. -
  55. [65][65] G. Riegler, A. Osman Ulusoy, and A. Geiger, 'Octnet: Learning deep 3d representations at high resolutions,' in CVPR , 2017. -
  56. [67][67] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, 'Deepvoxels: Learning persistent 3d feature embeddings,' in CVPR , 2019. -
  57. [68][68] T. Nguyen-Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang, 'Hologan: Unsupervised learning of 3d representations from natural images,' in ICCV , 2019. -
  58. [70][70] J. T. Kajiya and B. P. Von Herzen, 'Ray tracing volume densities,' SIGGRAPH , 1984. -
  59. [71][71] H. Fan, H. Su, and L. J. Guibas, 'A point set generation network for 3d object reconstruction from a single image,' in CVPR , 2017. -
  60. [72][72] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, 'Pointcnn: Convolution on x-transformed points,' NeurIPS , 2018. -
  61. [74][74] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, 'Pvrcnn: Point-voxel feature set abstraction for 3d object detection,' in CVPR , 2020. -
  62. [75][75] M. Meshry, D. B. Goldman, S. Khamis, H. Hoppe, R. Pandey, N. Snavely, and R. Martin-Brualla, 'Neural rerendering in the wild,' in CVPR , 2019. -
  63. [77][77] C. Lassner and M. Zollhofer, 'Pulsar: Efficient sphere-based neural rendering,' in CVPR , 2021. -
  64. [78][78] M. Wu, Y. Wang, Q. Hu, and J. Yu, 'Multi-view neural human rendering,' in CVPR , 2020. -
  65. [79][79] K.-A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, and V. Lempitsky, 'Neural point-based graphics,' in ECCV , 2020. -
  66. [80][80] O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson, 'SynSin: Endto-end view synthesis from a single image,' in CVPR , 2020. -
  67. [82][82] J. Huang, H. Zhang, L. Yi, T. Funkhouser, M. Nießner, and L. J. Guibas, 'Texturenet: Consistent local parametrizations for learning from high-resolution signals on meshes,' in CVPR , 2019. -
  68. [83][83] N. Verma, E. Boyer, and J. Verbeek, 'Feastnet: Feature-steered graph convolutions for 3d shape analysis,' in CVPR , 2018. -
  69. [85][85] Y. Zhou, C. Wu, Z. Li, C. Cao, Y. Ye, J. Saragih, H. Li, and Y. Sheikh, 'Fully convolutional mesh autoencoder using efficient spatially varying kernels,' NeurIPS , 2020. -
  70. [86][86] M. Fey, J. E. Lenssen, F. Weichert, and H. M¨ uller, 'Splinecnn: Fast geometric deep learning with continuous b-spline kernels,' in CVPR , 2018. -
  71. [87][87] C. Wen, Y. Zhang, Z. Li, and Y. Fu, 'Pixel2mesh++: Multi-view 3d mesh generation via deformation,' in ICCV , 2019. -
  72. [88][88] S. Goel, A. Kanazawa, and J. Malik, 'Shape and viewpoint without keypoints,' in ECCV , 2020. -
  73. [89][89] A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik, 'Learning category-specific mesh reconstruction from image collections,' in ECCV , 2018. -
  74. [90][90] M. M. Loper and M. J. Black, 'Opendr: An approximate differentiable renderer,' in ECCV , 2014. -
  75. [91][91] H. Kato, Y. Ushiku, and T. Harada, 'Neural 3d mesh renderer,' in CVPR , 2018. -
  76. [92][92] S. Liu, T. Li, W. Chen, and H. Li, 'Soft rasterizer: A differentiable renderer for image-based 3d reasoning,' in ICCV , 2019. -
  77. [94][94] A. Raj, J. Tanke, J. Hays, M. Vo, C. Stoll, and C. Lassner, 'Anr: Articulated neural rendering for virtual avatars,' in CVPR , 2021. -
  78. [95][95] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, 'Pytorch: An imperative style, high-performance deep learning library,' in NeurIPS , 2019. -
  79. [96][96] R. Wu, Y. Zhuang, K. Xu, H. Zhang, and B. Chen, 'Pq-net: A generative part seq2seq network for 3d shapes,' in CVPR , 2020. -
  80. [97][97] G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan, 'Pointflow: 3d point cloud generation with continuous normalizing flows,' in ICCV , 2019. -
  81. [98][98] L. Hui, R. Xu, J. Xie, J. Qian, and J. Yang, 'Progressive point cloud deconvolution generation network,' in ECCV , 2020. -
  82. [99][99] R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. Belongie, N. Snavely, and B. Hariharan, 'Learning gradient fields for shape generation,' in ECCV , 2020. -
  83. [100][100] H. Kim, H. Lee, W. H. Kang, J. Y. Lee, and N. S. Kim, 'Softflow: Probabilistic framework for normalizing flow on manifolds,' in NeurIPS , 2020. -
  84. [102][102] J. Xie, Y. Xu, Z. Zheng, S.-C. Zhu, and Y. N. Wu, 'Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification,' in CVPR , 2021. -
  85. [103][103] S. Luo and W. Hu, 'Diffusion probabilistic models for 3d point cloud generation,' in CVPR , 2021. -
  86. [104][104] Y. Deng, J. Yang, and X. Tong, 'Deformed implicit field: Modeling 3d shapes with learned dense correspondence,' in CVPR , 2021. -
  87. [105][105] X. Chen, T. Jiang, J. Song, J. Yang, M. J. Black, A. Geiger, and O. Hilliges, 'gdna: Towards generative detailed neural avatars,' in CVPR , 2022. -
  88. [107][107] Z. Shi, Y. Shen, J. Zhu, D.-Y. Yeung, and Q. Chen, '3d-aware indoor scene synthesis with depth priors,' in ECCV , 2022. -
  89. [108][108] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai, 'A shadingguided generative implicit model for shape-accurate 3d-aware image synthesis,' NeurIPS , 2021. -
  90. [110][110] X. Xu, X. Pan, D. Lin, and B. Dai, 'Generative occupancy fields for 3d surface-aware image synthesis,' NeurIPS , 2021. -
  91. [111][111] J. Sun, X. Wang, Y. Zhang, X. Li, Q. Zhang, Y. Liu, and J. Wang, 'Fenerf: Face editing in neural radiance fields,' in CVPR , 2022. -
  92. [112][112] J. Gu, L. Liu, P. Wang, and C. Theobalt, 'Stylenerf: A stylebased 3d-aware generator for high-resolution image synthesis,' in ICLR , 2022. -
  93. [113][113] Y. Deng, J. Yang, J. Xiang, and X. Tong, 'Gram: Generative radiance manifolds for 3d-aware image generation,' in CVPR , 2022. -
  94. [114][114] Y. Xue, Y. Li, K. K. Singh, and Y. J. Lee, 'Giraffe hd: A highresolution 3d-aware generative model,' in CVPR , 2022. -
  95. [115][115] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis et al. , 'Efficient geometry-aware 3d generative adversarial networks,' in CVPR , 2022. -
  96. [116][116] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, 'Occupancy networks: Learning 3d reconstruction in function space,' in CVPR , 2019. -
  97. [117][117] J. J. Park, P . Florence, J. Straub, R. Newcombe, and S. Lovegrove, 'Deepsdf: Learning continuous signed distance functions for shape representation,' in CVPR , 2019. -
  98. [118][118] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, 'Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,' NeurIPS , 2021. -
  99. [119][119] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, 'Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision,' in CVPR , 2020. -
  100. [121][121] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler, 'Neural geometric level of detail: Real-time rendering with implicit 3d shapes,' in CVPR , 2021. -
  101. [122][122] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, 'Neural sparse voxel fields,' NeurIPS , 2020. -
  102. [123][123] D. B. Lindell, J. N. Martel, and G. Wetzstein, 'Autoint: Automatic integration for fast neural volume rendering,' in CVPR , 2021. -
  103. [124][124] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, 'Plenoctrees for real-time rendering of neural radiance fields,' in ICCV , 2021. -
  104. [125][125] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec, 'Baking neural radiance fields for real-time view synthesis,' in ICCV , 2021. -
  105. [126][126] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin, 'Fastnerf: High-fidelity neural rendering at 200fps,' in ICCV , 2021. -
  106. [127][127] C. Reiser, S. Peng, Y. Liao, and A. Geiger, 'Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps,' in ICCV , 2021. -
  107. [128][128] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, 'Plenoxels: Radiance fields without neural networks,' in CVPR , 2022. -
  108. [129][129] J. Li, Z. Feng, Q. She, H. Ding, C. Wang, and G. H. Lee, 'Mine: Towards continuous depth mpi with nerf for novel view synthesis,' in ICCV , 2021. -
  109. [130][130] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann, 'Point-nerf: Point-based neural radiance fields,' in CVPR , 2022. -
  110. [131][131] D. Zimny, T. Trzci´ nski, and P. Spurek, 'Points2nerf: Generating neural radiance fields from 3d point cloud,' arXiv preprint arXiv:2206.01290 , 2022. - · arXiv
  111. [132][132] B. Yang, C. Bao, J. Zeng, H. Bao, Y. Zhang, Z. Cui, and G. Zhang, 'Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing,' in ECCV , 2022. -
  112. [133][133] J. Gao, W. Chen, T. Xiang, A. Jacobson, M. McGuire, and S. Fidler, 'Learning deformable tetrahedral meshes for 3d reconstruction,' NeurIPS , 2020. -
  113. [136][136] S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou, 'Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans,' in CVPR , 2021. -
  114. [137][137] J. Gao, T. Shen, Z. Wang, W. Chen, K. Yin, D. Li, O. Litany, Z. Gojcic, and S. Fidler, 'GET3D: A generative model of high quality 3d textured shapes learned from images,' in NeurIPS , 2022. -
  115. [138][138] A. Brock, T. Lim, J. M. Ritchie, and N. Weston, 'Generative and discriminative voxel modeling with convolutional neural networks,' arXiv preprint arXiv:1608.04236 , 2016. - · arXiv
  116. [139][139] Y. Zhou and O. Tuzel, 'Voxelnet: End-to-end learning for point cloud based 3d object detection,' in CVPR , 2018. -
  117. [140][140] J. Mao, Y. Xue, M. Niu, H. Bai, J. Feng, X. Liang, H. Xu, and C. Xu, 'Voxel transformer for 3d object detection,' in ICCV , 2021. -
  118. [142][142] H.-Y. Meng, L. Gao, Y.-K. Lai, and D. Manocha, 'Vv-net: Voxel vae net with group convolutions for point cloud segmentation,' in ICCV , 2019. -
  119. [143][143] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther, 'Autoencoding beyond pixels using a learned similarity metric,' in ICML , 2016. -
  120. [145][145] J. Li, C. Niu, and K. Xu, 'Learning part generation and assembly for structure-aware shape synthesis,' in AAAI , 2020. -
  121. [146][146] M. Ibing, G. Kobsik, and L. Kobbelt, 'Octree transformer: Autoregressive 3d shape generation on hierarchically structured sequences,' arXiv preprint arXiv:2111.12480 , 2021. - · arXiv
  122. [147][147] C.-L. Li, M. Zaheer, Y. Zhang, B. Poczos, and R. Salakhutdinov, 'Point cloud gan,' arXiv preprint arXiv:1810.05795 , 2018. - · arXiv
  123. [148][148] C. Xie, C. Wang, B. Zhang, H. Yang, D. Chen, and F. Wen, 'Stylebased point generator with adversarial rendering for point cloud completion,' in CVPR , 2021. -
  124. [149][149] D. Valsesia, G. Fracastoro, and E. Magli, 'Learning localized generative models for 3d point clouds via graph convolution,' in ICLR , 2018. -
  125. [150][150] D. W. Shu, S. W. Park, and J. Kwon, '3d point cloud generative adversarial network based on tree structured graph convolutions,' in ICCV , 2019. -
  126. [152][152] S. Ramasinghe, S. Khan, N. Barnes, and S. Gould, 'Spectral-gans for high-resolution 3d point-cloud generation,' in IROS , 2020. -
  127. [154][154] R. Klokov, E. Boyer, and J. Verbeek, 'Discrete point flow networks for efficient point cloud generation,' in ECCV , 2020. -
  128. [155][155] L. Zhou, Y. Du, and J. Wu, '3d shape generation and completion through point-voxel diffusion,' in ICCV , 2021. -
  129. [156][156] X. Zeng, A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, and K. Kreis, 'Lion: Latent point diffusion models for 3d shape generation,' in NeurIPS , 2022. -
  130. [157][157] L. Wu, D. Wang, C. Gong, X. Liu, Y. Xiong, R. Ranjan, R. Krishnamoorthi, V. Chandra, and Q. Liu, 'Fast point cloud generation with straight flows,' arXiv preprint arXiv:2212.01747 , 2022. - · arXiv
  131. [158][158] Y. Sun, Y. Wang, Z. Liu, J. Siegel, and S. Sarma, 'Pointgrow: Autoregressively learned point cloud generation with selfattention,' in WACV , 2020. -
  132. [160][160] A. Zadeh, Y.-C. Lim, P. P. Liang, and L.-P. Morency, 'Variational auto-decoder: A method for neural generative modeling from incomplete data,' arXiv preprint arXiv:1903.00840 , 2019. - · arXiv
  133. [161][161] X. Chen, T. Jiang, J. Song, J. Yang, M. J. Black, A. Geiger, and O. Hilliges, 'gdna: Towards generative detailed neural avatars,' in CVPR , 2022. -
  134. [162][162] Z. Hao, H. Averbuch-Elor, N. Snavely, and S. Belongie, 'Dualsdf: Semantic shape manipulation using a two-level representation,' in CVPR , 2020. -
  135. [163][163] M. Arjovsky and L. Bottou, 'Towards principled methods for training generative adversarial networks,' arXiv preprint arXiv:1701.04862 , 2017. - · arXiv
  136. [164][164] M. Ibing, I. Lim, and L. Kobbelt, '3d shape generation with gridbased implicit functions,' in CVPR , 2021. -
  137. [165][165] M. Kleineberg, M. Fey, and F. Weichert, 'Adversarial generation of continuous implicit shape representations,' arXiv preprint arXiv:2002.00349 , 2020. - · arXiv
  138. [166][166] A. Luo, T. Li, W.-H. Zhang, and T. S. Lee, 'Surfgen: Adversarial 3d shape synthesis with explicit surface discriminators,' in ICCV , 2021. -
  139. [167][167] W. E. Lorensen and H. E. Cline, 'Marching cubes: A high resolution 3d surface construction algorithm,' SIGGRAPH , 1987. -
  140. [168][168] E. Dupont, H. Kim, S. Eslami, D. Rezende, and D. Rosenbaum, 'From data to functa: Your data point is a function and you should treat it like one,' in ICML , 2022. -
  141. [169][169] G. Chou, Y. Bahat, and F. Heide, 'Diffusionsdf: Conditional generative modeling of signed distance functions,' arXiv preprint arXiv:2211.13757 , 2022. - · arXiv
  142. [170][170] G. Nam, M. Khlifi, A. Rodriguez, A. Tono, L. Zhou, and P. Guerrero, '3d-ldm: Neural implicit 3d shape generation with latent diffusion models,' arXiv preprint arXiv:2212.00842 , 2022. - · arXiv
  143. [171][171] T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen et al. , 'Rodin: A generative model for sculpting 3d digital avatars using diffusion,' arXiv preprint arXiv:2212.06135 , 2022. - · arXiv
  144. [172][172] J. R. Shue, E. R. Chan, R. Po, Z. Ankner, J. Wu, and G. Wetzstein, '3d neural field generation using triplane diffusion,' arXiv preprint arXiv:2211.16677 , 2022. - · arXiv
  145. [173][173] N. M¨ uller, Y. Siddiqui, L. Porzi, S. R. Bulo, P. Kontschieder, and M. Nießner, 'Diffrf: Rendering-guided 3d radiance field diffusion,' in CVPR , 2023. -
  146. [176][176] Z. Liu, Y. Feng, M. J. Black, D. Nowrouzezahrai, L. Paull, and W. Liu, 'Meshdiffusion: Score-based generative 3d mesh modeling,' in ICLR , 2023. -
  147. [177][177] Z. Lyu, J. Wang, Y. An, Y. Zhang, D. Lin, and B. Dai, 'Controllable mesh generation through sparse latent point diffusion models,' in CVPR , 2023. -
  148. [178][178] X. Wang and A. Gupta, 'Generative image modeling using style and structure adversarial networks,' in ECCV , 2016. -
  149. [179][179] A. Noguchi and T. Harada, 'Rgbd-gan: Unsupervised 3d representation learning from natural image datasets via rgbd image synthesis,' arXiv preprint arXiv:1909.12573 , 2019. - · arXiv
  150. [180][180] J.-Y. Zhu, Z. Zhang, C. Zhang, J. Wu, A. Torralba, J. Tenenbaum, and B. Freeman, 'Visual object networks: Image generation with disentangled 3d representations,' NeurIPS , 2018. -
  151. [182][182] P. Henzler, N. J. Mitra, and T. Ritschel, 'Escaping plato's cave: 3d shape from adversarial rendering,' in ICCV , 2019. -
  152. [183][183] T. H. Nguyen-Phuoc, C. Richardt, L. Mai, Y. Yang, and N. Mitra, 'Blockgan: Learning 3d object-aware scene representations from unlabelled images,' NeurIPS , 2020. -
  153. [184][184] P. Zhou, L. Xie, B. Ni, and Q. Tian, 'Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis,' arXiv preprint arXiv:2110.09788 , 2021. - · arXiv
  154. [185][185] R. Or-El, X. Luo, M. Shan, E. Shechtman, J. J. Park, and I. Kemelmacher-Shlizerman, 'Stylesdf: High-resolution 3dconsistent image and geometry generation,' in CVPR , 2022. -
  155. [186][186] X. Zhang, Z. Zheng, D. Gao, B. Zhang, P. Pan, and Y. Yang, 'Multi-view consistent generative adversarial networks for 3daware image synthesis,' in CVPR , 2022. -
  156. [187][187] Z. Shi, Y. Xu, Y. Shen, D. Zhao, Q. Chen, and D.-Y. Yeung, 'Improving 3d-aware image synthesis with a geometry-aware discriminator,' in NeurIPS , 2022. -
  157. [188][188] J. Xiang, J. Yang, Y. Deng, and X. Tong, 'Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds,' arXiv preprint arXiv:2206.07255 , 2022. - · arXiv
  158. [189][189] I. Skorokhodov, S. Tulyakov, Y. Wang, and P. Wonka, 'Epigraf: Rethinking training of 3d gans,' arXiv preprint arXiv:2206.10535 , 2022. - · arXiv
  159. [190][190] Z. Shi, Y. Shen, Y. Xu, S. Peng, Y. Liao, S. Guo, Q. Chen, and D.-Y. Yeung, 'Learning 3d-aware image synthesis with unknown pose distribution,' in CVPR , 2023. -
  160. [191][191] T. DeVries, M. A. Bautista, N. Srivastava, G. W. Taylor, and J. M. Susskind, 'Unconstrained scene generation with locally conditioned radiance fields,' in ICCV , 2021. -
  161. [192][192] S. Cai, A. Obukhov, D. Dai, and L. Van Gool, 'Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation,' in CVPR , 2022. -
  162. [193][193] A. Tewari, X. Pan, O. Fried, M. Agrawala, C. Theobalt et al. , 'Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images,' in CVPR , 2022. -
  163. [194][194] M. Lee, C. Chung, H. Cho, M. Kim, S. Jung, J. Choo, and M. Sung, '3d-gif: 3d-controllable object generation via implicit factorized representations,' arXiv preprint arXiv:2203.06457 , 2022. - · arXiv
  164. [195][195] A. R. Kosiorek, H. Strathmann, D. Zoran, P. Moreno, R. Schneider, S. Mokr´ a, and D. J. Rezende, 'Nerf-vae: A geometry aware 3d scene generative model,' in ICML , 2021. -
  165. [196][196] D. Rebain, M. J. Matthews, K. M. Yi, D. Lagun, and A. Tagliasacchi, 'Lolnerf: Learn from one look,' in CVPR , 2022. -
  166. [197][197] Y. Xu, M. Chai, Z. Shi, S. Peng, I. Skorokhodov, A. Siarohin, C. Yang, Y. Shen, H.-Y. Lee, B. Zhou et al. , 'Discoscene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis,' in CVPR , 2023. -
  167. [199][199] J. Zhang, E. Sangineto, H. Tang, A. Siarohin, Z. Zhong, N. Sebe, and W. Wang, '3d-aware semantic-guided generative model for human synthesis,' in ECCV , 2022. -
  168. [200][200] J. Sun, X. Wang, Y. Shi, L. Wang, J. Wang, and Y. Liu, 'Ide3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis,' in SIGGRAPH Asia , 2022. -
  169. [201][201] J. Sun, X. Wang, L. Wang, X. Li, Y. Zhang, H. Zhang, and Y. Liu, 'Next3d: Generative neural texture rasterization for 3d-aware head avatars,' in CVPR , 2023. -
  170. [202][202] M. A. Bautista, P. Guo, S. Abnar, W. Talbott, A. Toshev, Z. Chen, L. Dinh, S. Zhai, H. Goh, D. Ulbricht et al. , 'Gaudi: A neural architect for immersive 3d scene generation,' NeurIPS , 2022. -
  171. [203][203] T. Anciukeviˇ cius, Z. Xu, M. Fisher, P. Henderson, H. Bilen, N. J. Mitra, and P. Guerrero, 'Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation,' arXiv preprint arXiv:2211.09869 , 2022. - · arXiv
  172. [204][204] C. Nash, Y. Ganin, S. A. Eslami, and P. Battaglia, 'Polygen: An autoregressive generative model of 3d meshes,' in ICML , 2020. -
  173. [205][205] Y. Shen, J. Gu, X. Tang, and B. Zhou, 'Interpreting the latent space of gans for semantic face editing,' in CVPR , 2020. -
  174. [206][206] A. Tewari, M. Elgharib, G. Bharaj, F. Bernard, H.-P. Seidel, P. P´ erez, M. Zollhofer, and C. Theobalt, 'Stylerig: Rigging stylegan for 3d control over portrait images,' in CVPR , 2020. -
  175. [207][207] Y. Deng, J. Yang, D. Chen, F. Wen, and X. Tong, 'Disentangled and controllable face image generation via 3d imitative-contrastive learning,' in CVPR , 2020. -
  176. [208][208] C. Yang, Y. Shen, and B. Zhou, 'Semantic hierarchy emerges in deep generative representations for scene synthesis,' IJCV , 2021. -
  177. [209][209] T. Leimk¨ uhler and G. Drettakis, 'Freestylegan: Free-view editable portrait rendering with the camera manifold,' arXiv preprint arXiv:2109.09378 , 2021. - · arXiv
  178. [210][210] M. Mirza and S. Osindero, 'Conditional generative adversarial nets,' arXiv preprint arXiv:1411.1784 , 2014. - · arXiv
  179. [211][211] V. Sitzmann, M. Zollh¨ ofer, and G. Wetzstein, 'Scene representation networks: Continuous 3d-structure-aware neural scene representations,' in NeurIPS , 2019. -
  180. [212][212] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, 'Implicit neural representations with periodic activation functions,' NeurIPS , 2020. -
  181. [213][213] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, 'Film: Visual reasoning with a general conditioning layer,' in AAAI , 2018. -
  182. [215][215] Y. Yang, Y. Yang, H. Guo, R. Xiong, Y. Wang, and Y. Liao, 'Urbangiraffe: Representing urban scenes as compositional generative neural feature fields,' arXiv preprint arXiv:2303.14167 , 2023. - · arXiv
  183. [216][216] R. Abdal, Y. Qin, and P. Wonka, 'Image2stylegan: How to embed images into the stylegan latent space?' in ICCV , 2019. -
  184. [217][217] S. Pidhorskyi, D. Adjeroh, and G. Doretto, 'Adversarial latent autoencoders,' in CVPR , 2020. -
  185. [218][218] Q. Wang, Z. Shi, K. Zheng, Y. Xu, S. Peng, and Y. Shen, 'Benchmarking and analyzing 3d-aware image synthesis with a modularized codebase,' arXiv preprint arXiv:2306.12423 , 2023. - · arXiv
  186. [219][219] J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng, 'Avatargen: a 3d generative model for animatable human avatars,' arXiv preprint arXiv:2208.00561 , 2022. - · arXiv
  187. [220][220] A. W. Bergman, P. Kellnhofer, Y. Wang, E. R. Chan, D. B. Lindell, and G. Wetzstein, 'Generative neural articulated radiance fields,' arXiv preprint arXiv:2206.14314 , 2022. - · arXiv
  188. [223][223] Z. Zheng, T. Yu, Q. Dai, and Y. Liu, 'Deep implicit templates for 3d shape representation,' in CVPR , 2021. -
  189. [225][225] Z. Liu, Y. Wang, X. Qi, and C. Fu, 'Towards implicit text-guided 3d shape generation,' CVPR , 2022. -
  190. [226][226] R. Fu, X. Zhan, Y. Chen, D. Ritchie, and S. Sridhar, 'Shapecrafter: A recursive text-conditioned 3d shape generation model,' arXiv preprint arXiv:2207.09446 , 2022. - · arXiv
  191. [227][227] Y. Liao, K. Schwarz, L. M. Mescheder, and A. Geiger, 'Towards unsupervised learning of generative models for 3d controllable image synthesis,' in CVPR , 2020. -
  192. [228][228] A. Noguchi, X. Sun, S. Lin, and T. Harada, 'Unsupervised learning of efficient geometry-aware neural articulated representations,' in ECCV , 2022. -
  193. [229][229] C. Wang, M. Chai, M. He, D. Chen, and J. Liao, 'Clip-nerf: Textand-image driven manipulation of neural radiance fields,' CVPR , 2022. -
  194. [230][230] B. Zhang, M. Nießner, and P. Wonka, '3dilg: Irregular latent grids for 3d generative modeling,' arXiv preprint arXiv:2205.13914 , 2022. - · arXiv
  195. [231][231] S. Duggal, Z. Wang, W. Ma, S. Manivasagam, J. Liang, S. Wang, and R. Urtasun, 'Secrets of 3d implicit object shape reconstruction in the wild,' arXiv preprint arXiv:2101.06860 , 2021. - · arXiv
  196. [233][233] X. Chen, Z. Dong, J. Song, A. Geiger, and O. Hilliges, 'Category level object pose estimation via neural analysis-by-synthesis,' in ECCV , 2020. -
  197. [234][234] J. Guo, F. Zhong, R. Xiong, Y. Liu, Y. Wang, and Y. Liao, 'A visual navigation perspective for category-level object pose estimation,' in ECCV , 2022. -
  198. [235][235] B. Eckart, W. Yuan, C. Liu, and J. Kautz, 'Self-supervised learning on 3d point clouds by learning discrete generative models,' in CVPR , 2021. -
  199. [236][236] J. Xie, Z. Zheng, R. Gao, W. Wang, S.-C. Zhu, and Y. N. Wu, 'Generative voxelnet: learning energy-based models for 3d shape synthesis and analysis,' TPAMI , 2020. -
  200. [237][237] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with CLIP latents,' arXiv preprint arXiv:2204.06125 , 2022. - · arXiv

200 of 201 shown

Cited by

Sorted by citations

Authors
Shi, Z., Peng, S., Xu, Y., Geiger, A., Liao, Y., & Shen, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2210.15663
arXiv
2210.15663
OpenAlex
W4320166487
Cited by
14
References
0
Citations by year
2025: 3 · 2024: 6 · 2023: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
Keywords
3D generative models, three-dimensional representation, point clouds, 3D mesh, implicit neural representations, variational autoencoder, generative adversarial networks, diffusion models, 3D shape synthesis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review