The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Open accessAbstract
Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape gener-ation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive compar-ative evaluations to better understand its behavior.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · cited 1×
“A interesting avenue of future research would be to investigate prompt engineering [60].”
BackgroundTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2020) · arXiv · doi:10.48550/arxiv.2012.09841 · Cited by 2,514
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
NICE: Non-linear Independent Components Estimation
Dinh, L., Krueger, D., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1410.8516 · Cited by 1,493
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
Wu, J., Zhang, C., Xue, T., Freeman, W. T., & Tenenbaum, J. B. (2016) · arXiv · doi:10.48550/arxiv.1610.07584 · Cited by 1,393 · cited 1×
“Recently, there has been tremendous progress in 3D shape generating in different data formats such as point cloud [3,34,59], voxel [58], implicit representation [8,37,44] and mesh [39].”
BackgroundPointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows
Yang, G., Huang, X., Hao, Z., Liu, M.-Y., Belongie, S., & Hariharan, B. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00464 · Cited by 656
Research works printed but not linked: 49
- van den oord, 2017, Neural discrete representation learning, ArXiv Preprint
- nash, 2020, Polygen: An autoregressive generative model of 3d meshes, ICML
- chun-liang, 2018, Point cloud gan, ar Xiv preprint
- lampert, 2009, Learning to Detect Unseen Object Classes by Betweenclass Attribute Transfer
- kumar, 2020, Videoflow A conditional flow-based model for stochastic video generation
- kingma, 2018, Glow Generative flow with invertible 1×1 convolutions
- mishra, 2019, Machine Learning in the AWS Cloud Add Intelligence to Applications with Amazon SageMaker and Amazon Rekognition
- 10.1109/CVPR.2019.00459 · DOI · Paper page
- liu, 2019, An acceleration framework for high resolution image synthesis, ar Xiv preprint
- lin, 2018, Generating animated videos of human activities from natural language descriptions
- kim, 2019, Flowavenet A generative flow for raw audio
- karpathy, 2014, Deep frag-ment embeddings for bidirectional image sentence mapping, ArXiv Preprint
- achlioptas, 2018, Learning representations and generative models for 3d point clouds, International Conference on Machine Learning, 40
- achlioptas, 0, Referit3d: N eurallisteners for fine-grained 3d object identification in real-world scenes, European Conference on Computer Vision, 422
- pin-hao, 2021, Text-guided graph neural networks for re-ferring 3d instance segmentation, Proceedings of the AAAI Conference on Artificial Intelligence, 35, 1610, 10.1609/aaai.v35i2.16253 · DOI
- ioffe, 2015, Batch normalization: Accelerating deep network training by reducing internal co-variate shift, International Conference on Machine Learning, 448
- huq, 2020, Holistic static and animated 3d scene generation from diverse text descriptions, CoRR abs/2010 01549
- armand, 0, Learning visual features from large weakly supervised data, European Conference on Computer Vision, 67
- rezende, 2015, Variational inference with normalizing flows, International Conference on Ma-chine Learning, 1530
- rothfuss, 2019, Noise regularization for conditional density estimation, ArXiv Preprint
- yang, 2018, Foldingnet: Point cloud auto-encoder via deep grid deformation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 206
- wang, 2018, Pixel2mesh: Generating 3d mesh models from single rgb images, ECCV
- tang, 2021, Part2word Learning joint embedding of point clouds and text by matching parts to words
- 10.1162/tacl_a_00177 · DOI · Paper page
- shen, 2021, How much can clip benefit vision-and-language tasks?, ArXiv Preprint
- sariyildiz, 0, Learning visual representations with caption annotations, Computer Vision-ECCV 2020 16th European Conference Glasgow UK August 23–28 2020 Proceedings Part VIII 16, 153
- 10.1109/CVPR46437.2021.01101 · DOI · Paper page
- pakhomov, 2021, Segmentation in style: Unsuper-vised semantic image segmentation with stylegan and clip, ArXiv Preprint
- dinh, 2016, Density estimation using real nvp, ar Xiv preprint
- 10.1109/CVPR46437.2021.01268 · DOI · Paper page
- fang, 2021, Clip2video: Mastering video-text retrieval via image clip, ArXiv Preprint
- 10.7551/mitpress/7287.001.0001 · DOI · Paper page
- frans, 2021, Clipdraw Exploring text-to-drawing synthesis through language-image encoders
- frome, 2013, DeViSE A deep visual-semantic embedding model [J]
- 10.1609/aaai.v33i01.3301126 · DOI
- chang, 2015, ShapeNet An Information-Rich 3D Model Repository
- achlioptas, 2018, Learning to refer to 3D objects with natural language
- 10.1007/978-3-030-20893-6_7 · DOI · Paper page
- chen, 0, Scanrefer: 3d object localization in rgb-d scans using natu-ral language, Computer Vision-ECCV 2020 16th Euro-pean Conference Glasgow UK August 23–28 2020 Pro-ceedings Part XX 16, 202
- 10.1109/CVPR42600.2020.00700 · DOI · Paper page
- 10.1109/CVPR.2019.00609 · DOI · Paper page
- choy, 0, 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction, European Conference on Computer Vision, 628
- pumarola, 2020, C-flow Conditional generative flow models for images and 3d point clouds
- peng, 0, Convolutional occupancy networks, Computer Vision-ECCV 2020 16th European Conference Glasgow UK August 23–28 2020 Proceedings Part III 16, 523
- qi, 2017, Pointnet: Deep learning on point sets for 3d classification and segmentation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 652
- papamakarios, 2017, Masked autoregressive flow for density estimation, ar Xiv preprint
- palatucci, 2009, Zero-shot learning with semantic out-put codes, Advances in Neural Infor-Mation Processing Systems, 22
- 10.1109/ICCV48922.2021.00209 · DOI · Paper page
- 10.1109/CVPR.2019.00025 · DOI · Paper page
Cited by
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Magic3D: High-Resolution Text-to-3D Content Creation
Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., & Lin, T.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00037 · Cited by 857
Zero-Shot Text-Guided Object Generation with Dream Fields
Jain, A., Mildenhall, B., Barron, J. T., Abbeel, P., & Poole, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00094 · Cited by 388
MotionCLIP: Exposing Human Motion Generation to CLIP Space
Tevet, G., Gordon, B., Hertz, A., Bermano, A. H., & Cohen-Or, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20047-2_21 · Cited by 317
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
Höllein, L., Cao, A., Owens, A., Johnson, J., & Nießner, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00727 · Cited by 167
Metadata
- Authors
- Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., & Malekshan, K. R.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 18582-18592
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3201927751
- Cited by
- 220
- References
- 12
- FWCI
- 19.41
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.