The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Magic3D: High-Resolution Text-to-3D Content Creation
Open accessAbstract
DreamFusion [31] has recently demonstrated the utility of a pretrained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF) [23], achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2× faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7% raters to prefer our approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
Instant neural graphics primitives with a multiresolution hash encoding
Müller, T., Evans, A., Schied, C., & Keller, A. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530127 · Cited by 4,114
Occupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00459 · Cited by 2,708
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2×
“) to generate high-quality images of objects and scenes [2, 33, 34, 36].”
BackgroundDreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., & Hedman, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00539 · Cited by 1,935
Generative Modeling by Estimating Gradients of the Data Distribution
Song, Y., & Ermon, S. (2019) · arXiv · doi:10.48550/arxiv.1907.05600 · Cited by 1,453
Learning Implicit Fields for Generative Shape Modeling
Chen, Z., & Zhang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00609 · Cited by 1,443
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
Wu, J., Zhang, C., Xue, T., Freeman, W. T., & Tenenbaum, J. B. (2016) · arXiv · doi:10.48550/arxiv.1610.07584 · Cited by 1,393 · cited 1×
“There is a large body of work on 3D generative modeling, exploring different types of 3D representations such as 3D voxel grids [7, 12, 20, 40, 45], pointclouds [1, 21, 24, 46, 47, 49], meshes [9, 48], implicit [6, 22], or octree [15] representations.”
BackgroundGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357
Classifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126
Efficient Geometry-aware 3D Generative Adversarial Networks
Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., de Mello, S., Gallo, O., Guibas, L., Tremblay, J., Khamis, S., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01565 · Cited by 1,093
Self-citationHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
Niemeyer, M., & Geiger, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01129 · Cited by 731
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis
Chan, E. R., Monteiro, M., Kellnhofer, P., Wu, J., & Wetzstein, G. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00574 · Cited by 689
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape SynthesisPreprint
Shen, T., Gao, J., Yin, K., Liu, M.-Y., & Fidler, S. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2111.04276 · Cited by 673
Self-citationPointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows
Yang, G., Huang, X., Hao, Z., Liu, M.-Y., Belongie, S., & Hariharan, B. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00464 · Cited by 656
Self-citationDiffusion Probabilistic Models for 3D Point Cloud Generation
Luo, S., & Hu, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00286 · Cited by 628
DreamFusion: Text-to-3D using 2D Diffusion
Poole, B., Jain, A., Barron, J. T., & Mildenhall, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14988 · Cited by 474
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahrai, D., Jacobson, A., McGuire, M., & Fidler, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01120 · Cited by 449
Self-citation3D Shape Generation and Completion through Point-Voxel Diffusion
Zhou, L., Du, Y., & Wu, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00577 · Cited by 421
Zero-Shot Text-Guided Object Generation with Dream Fields
Jain, A., Mildenhall, B., Barron, J. T., Abbeel, P., & Poole, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00094 · Cited by 388
Extracting Triangular 3D Models, Materials, and Lighting From Images
Munkberg, J., Chen, W., Hasselgren, J., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00810 · Cited by 343
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images
Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., & Yang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00768 · Cited by 340
Compositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022) · arXiv · doi:10.48550/arxiv.2206.01714 · Cited by 295
3D Shape Induction from 2D Views of Multiple Objects
Gadelha, M., Maji, S., & Wang, R. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00053 · Cited by 260
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation
OrEl, R., Luo, X., Shan, M., Shechtman, E., Park, J. J., & Kemelmacher-Shlizerman, I. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01314 · Cited by 245
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Gao, J., Shen, T., Wang, Z., Chen, W., Yin, K., Li, D., Litany, O., Gojčič, Ž., & Fidler, S. (2022) · neural information processing systems · doi:10.52202/068431-2308 · Cited by 233
Self-citationStructureNet: hierarchical graph networks for 3D shape generation
Mo, K., Guerrero, P., Yi, L., Su, H., Wonka, P., Mitra, N. J., & Guibas, L. J. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356527 · Cited by 229
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., & Malekshan, K. R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01805 · Cited by 220
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.01324 · Cited by 219
Self-citationEscaping Plato’s Cave: 3D Shape From Adversarial Rendering
Henzler, P., Mitra, N., & Ritschel, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01008 · Cited by 194
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
Mohammad Khalid, N., Xie, T., Belilovsky, E., & Popa, T. (2022) · SIGGRAPH Asia 2022 Conference Papers · doi:10.1145/3550469.3555392 · Cited by 190
LION: Latent Point Diffusion Models for 3D Shape Generation
Zeng, X., Vahdat, A., Williams, F., Gojčič, Ž., Litany, O., Fidler, S., & Kreis, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06978 · Cited by 190
Self-citationStyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
Gu, J., Liu, L., Wang, P., & Theobalt, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08985 · Cited by 178
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
Hao, Z., Mallya, A., Belongie, S., & Liu, M.-Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01381 · Cited by 128
Self-citationLearning Deformable Tetrahedral Meshes for 3D ReconstructionPreprint
Gao, J., Chen, W., Xiang, T., Tsang, C. F., Jacobson, A., McGuire, M., & Fidler, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01437 · Cited by 123
Self-citationInverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D DataPreprint
Lunz, S., Li, Y., Fitzgibbon, A. W., & Kushman, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12674 · Cited by 56
Research works printed but not linked: 8
- ibing, 2021, Octree transformer: Autoregressive 3d shape generation on hierarchically structured sequences, ArXiv Preprint
- liu, 2020, Neural sparse voxel fields, Advances in neural information processing systems, 33, 15651
- 10.1145/3414685.3417861 · DOI
- zhang, 2021, Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering, International Conference on Learning Representations
- takikawa, 2022, Kaolin wisp A pytorch library and engine for neural fields research
- smith, 2017, Improved adversarial systems for 3d object generation and reconstruction, Conference on Robot Learning, 87
- achlioptas, 2018, Learning representations and generative models for 3d point clouds, International Conference on Machine Learning, 40
- schwarz, 0, Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids, Advances in Neural IInformation Processing Systems
Cited by
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704
“Multi-Modal Learning Text-to-Image Generation [7],[216],[231],[189],[87],[65],[229],[126],[269],[9],[299],[321] Scene Graph-to-Image Generation [306] Text-to-3D Generation [296],[155],[207] Text-to-Motion Generation [263, 322],[130] Text-to-Video Generation [245],[96],[283],[210] Text-to-Audio Generation [208],[301],[…”
BackgroundMethodInfluentialZero-1-to-3: Zero-shot One Image to 3D Object
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00853 · Cited by 841
Bandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023) · Future Internet · doi:10.3390/fi15080260 · Cited by 638
A Survey on Generative Diffusion Models
Cao, H., Tan, C., Gao, Z., Xu, Y., Chen, G., Heng, P.-A., & Li, S. Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3361474 · Cited by 538
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services
Xu, M., Du, H., Niyato, D., Kang, J., Xiong, Z., Mao, S., Han, Z., Jamalipour, A., Kim, D. I., Shen, X., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3353265 · Cited by 449
Metadata
- Authors
- Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., & Lin, T.-Y.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 300-309
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386065887
- Cited by
- 857
- References
- 44
- FWCI
- 187.60
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.