The DOI matches its DataCite record; 69 of the 69 references it cites were found in a registry or an index.
Hierarchical Text-Conditional Image Generation with CLIP Latents
Open accessAbstract
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While more sophisticated methods for partially editing individual frames during training exist (e.g., text-to-image models [13], [14]), they are slow, computationally expensive and error-prone.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredIn the citing paper
“Generative machines, such as text-to-image algorithms, are contributing to the variety of cultural traits.”
From Machine culture (2023) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“To this end, we generated 512 “artistic” captions using GPT-3 [4] by prompting it with captions for existing artwork (both real and AI generated).”
MethodSelf-citationHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 2×
“[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
BackgroundMethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundMethodSelf-citationLIII. On lines and planes of closest fit to systems of points in space
Pearson, K. (1901) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440109462720 · Cited by 12,864 · cited 4×
“After applying PCA, we order the principal components by decreasing eigenvalue magnitude, quantize each of the 319 dimensions into 1,024 discrete buckets, and predict the resulting sequence using a decoder-only Transformer [53] model with a causal attention mask.”
MethodInfluentialDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Self-citationNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · cited 1×
“Concurrently, diffusion models [46, 48, 25] have emerged as a promising generative modeling framework, pushing the state-of-the-art on image and video generation tasks [11, 26, 24].”
BackgroundTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 2×
“[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
MethodDenoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 6×
“Like GLIDE and DALL-E, unCLIP is not directly trained on the MS-COCO training set, but can still generalize to the validation set zero-shot.”
BackgroundMethodResultInfluentialSelf-citationAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · cited 1×
“Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56, 63, 49, 58, 57].”
BackgroundImage Super-Resolution Via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204461 · Cited by 1,829 · cited 2×
“For the first upsampling stage, we use gaussian blur [43], and for the second, we use a more diverse BSR degradation [42,”
MethodImage Super-Resolution via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2021) · arXiv · doi:10.48550/arxiv.2104.07636 · Cited by 1,829
Improved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550
Self-citationGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 6×
“Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.”
BackgroundMethodResultInfluentialSelf-citationGenerative Visual Manipulation on the Natural Image Manifold
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_36 · Cited by 1,234
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen‐Or, D., & Lischinski, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,147 · cited 1×
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodClassifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126 · cited 4×
“We enable classifier-free guidance [24] by randomly setting the CLIP embeddings to zero (or a learned embedding) 10% of the time, and randomly dropping the text caption 50% of the time.”
BackgroundMethodInfluentialAVA: A large-scale database for aesthetic visual analysis
Murray, N., Marchesotti, L., & Perronnin, F. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247954 · Cited by 1,029 · cited 1×
“Next, we trained a CLIP linear probe to predict human aesthetic judgments using the AVA dataset [33] (Appendix A).”
MethodDesigning a Practical Degradation Model for Deep Blind Image Super-Resolution
Zhang, K., Liang, J., Gool, L. V., & Timofte, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00475 · Cited by 867 · cited 1×
“For the first upsampling stage, we use gaussian blur [43], and for the second, we use a more diverse BSR degradation [42, 59].”
MethodHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01043 · Cited by 720 · cited 1×
“Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.”
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709 · cited 1×
“[41] trains a multi-layer discrete autoencoder, allowing them to first sample coarse-grained latent codes and then use this as conditioning information when sampling higher-resolution latent codes.”
BackgroundGal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., & Cohen‐Or, D. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530164 · Cited by 682 · cited 1×
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodDM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis
Zhu, M., Pan, P., Chen, W., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00595 · Cited by 625
NVAE: A Deep Hierarchical Variational AutoencoderPreprint
Vahdat, A., & Kautz, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03898 · Cited by 564 · cited 1×
“Child, Vahdat and Kautz [5, 50] generate images using VAEs with a hierarchy of latent codes that increase progressively with resolution.”
BackgroundXia, W., Zhang, Y., Yang, Y., Xue, J., Zhou, B., & Yang, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3181070 · Cited by 558
Sharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521 · cited 2×
“Both models are trained with learning rate 3× 10−4 and SAM [15] with ρ = 0.1, where the perturbations are applied independently by the replicas, each of which uses batch size 64.”
BackgroundInfluentialCascaded Diffusion Models for High Fidelity Image Generation
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., & Salimans, T. (2021) · arXiv · doi:10.48550/arxiv.2106.15282 · Cited by 514 · cited 1×
“Concurrently, diffusion models [46, 48, 25] have emerged as a promising generative modeling framework, pushing the state-of-the-art on image and video generation tasks [11, 26, 24].”
BackgroundImproved Techniques for Training Score-Based Generative Models
Song, Y., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09011 · Cited by 482 · cited 2×
“We use diffusion models [25, 48] to produce images conditioned on CLIP image embeddings (and optionally text captions).”
BackgroundMethodContrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · cited 1×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundCogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450 · cited 1×
“Other works have adapted the VQ-VAE approach [52] to text-conditional image generation by training autoregressive transformers on sequences of text tokens followed by image tokens [40, 12, 1].”
MethodSLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 385 · cited 1×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundHow Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 382
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 376 · cited 1×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundCross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 375 · cited 1×
“Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56, 63, 49, 58, 57].”
BackgroundDiffusion Autoencoders: Toward a Meaningful and Decodable Representation
Preechakul, K., Chatthee, N., Wizadwongsa, S., & Suwajanakorn, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01036 · Cited by 338 · cited 1×
“[38] propose an autoencoder framework where diffusion models are used to render latent variables as images, and a second diffusion model is used to generate these latents (similar to our diffusion”
MethodMake-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · arXiv · doi:10.48550/arxiv.2203.13131 · Cited by 332 · cited 1×
“[17] conditions a generative image model on segmentation masks, allowing for a generative process that first samples a semantic map of an image and then conditions the generated image on this information.”
BackgroundMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 330 · cited 4×
“Figure 5: Variations of images featuring typographic attacks [20] paired with the CLIP model’s predicted probabilities across three labels.”
BackgroundMethodInfluentialTowards Language-Free Training for Text-to-Image Generation
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01738 · Cited by 292 · cited 2×
“[61] condition GAN models on randomly perturbed CLIP image embeddings, finding that these models can generalize to CLIP text embeddings to produce text-conditional images.”
BackgroundAnalytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic ModelsPreprint
Bao, F., Li, C., Zhu, J. J., & Zhang, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.06503 · Cited by 255
Score-based Generative Modeling in Latent Space
Vahdat, A., Kreis, K., & Kautz, J. (2021) · arXiv · doi:10.48550/arxiv.2106.05931 · Cited by 224 · cited 1×
“[51] use a score-based model for the latent space of a VAE, while Rombach et al.”
MethodAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial NetworksPreprint
Xu, T., Zhang, P., Huang, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10485 · Cited by 159
Generative Adversarial Networks
Krichen, M. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10306417 · Cited by 157
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Child, R. (2020) · arXiv · doi:10.48550/arxiv.2011.10650 · Cited by 140 · cited 1×
“Child, Vahdat and Kautz [5, 50] generate images using VAEs with a hierarchy of latent codes that increase progressively with resolution.”
BackgroundLearning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 128 · cited 1×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundGenerative Visual Manipulation on the Natural Image ManifoldPreprint
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03552 · Cited by 119
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPPreprint
Wang, Z., Liu, W., He, Q., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2203.00386 · Cited by 97 · cited 2×
“Another possibility is to feed the decoder the CLIP text embedding as if it were an image embedding, as previously observed [61, 54].”
MethodImproving Text-to-Image Synthesis Using Contrastive Learning
Ye, H., Yang, X., Takác, M., et al. (2021) · British Machine Vision Conference · doi:10.5244/c.35.137 · Cited by 93
High Fidelity Visualization of What Your Self-Supervised Representation Knows About
Bordes, F., Balestriero, R., & Vincent, P. (2021) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2112.09164 · Cited by 84 · cited 1×
“[3] train diffusion models conditioned on image representations from contrastive models.”
BackgroundGenerating Images from Caption and Vice Versa via CLIP-Guided Generative Latent Space Search
Galatolo, F. A., Cimino, M., & Vaglini, G. (2021) · International Conference on Image Processing and Vision Engineering (IMPROVE) · doi:10.5220/0010503701660174 · Cited by 80 · cited 1×
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodCLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D., Kopp, M., Klambauer, G., Bitto, A., et al. (2022) · neural information processing systems · doi:10.52202/068431-1487 · Cited by 67 · cited 1×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundStyleGAN-NADA: CLIP-Guided Domain Adaptation of Image GeneratorsPreprint
Gal, R., Patashnik, O., Maron, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00946 · Cited by 64
Ruxton, G. (2006) · doi:10.1641/0006-3568(2006)056[0446:tac]2.0.co;2 · Cited by 59 · cited 1×
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42
Vector Quantized Diffusion Model for Text-to-Image SynthesisPreprint
Gu, S., Chen, D., Bao, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14822 · Cited by 39
LAFITE: Towards Language-Free Training for Text-to-Image Generation
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T. (2021) · arXiv · doi:10.48550/arxiv.2111.13792 · Cited by 38
Cross-Modal Contrastive Learning for Text-to-Image GenerationPreprint
Zhang, H., Koh, J. Y., Baldridge, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04702 · Cited by 34
DF-GAN: A Simple and Effective Baseline for Text-to-Image SynthesisPreprint
Ming, T., Tang, H., Wu, F. Q., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05865 · Cited by 26
Improving Text-to-Image Synthesis Using Contrastive LearningPreprint
Ye, H., Yang, X., Takáč, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02423 · Cited by 24
SLIP: Self-supervision meets Language-Image Pre-trainingPreprint
Mu, N., Kirillov, A. M., Wagner, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.12750 · Cited by 13
Diffusion Autoencoders: Toward a Meaningful and Decodable RepresentationPreprint
Preechakul, K., Chatthee, N., Wizadwongsa, S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.15640 · Cited by 2
Research works printed but not linked: 2
- [23][23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
- [32][32] Ryan Murdock. The Big Sleep. https://twitter.com/advadnoun/status/ 1351038053033406468, 2021. 20 https://openreview.net/forum?id=qw674L9PfQE https://openreview.net/forum?id=qw674L9PfQE https://arxiv.org/abs/2203.13131 https://arxiv.org/abs/2108.00946 https://arxiv.org/abs/2102.01645 https://arxiv.org/abs/1406.2661 https://arxiv.org/abs/2111.14822 https://openreview.net/forum?id=qw8AKxfYbI https://openreview.net/forum?id=qw8AKxfYbI https://arxiv.org/abs/2006.11239 https://arxiv.org/abs/2106.15282 https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1405.0312 https://arxiv.org/abs/1711.05101 https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://arxiv.org/abs/2112.12750 https://twitter.com/advadnoun/status/1351038053033406468 https://twitter.com/advadnoun/status/1351038053033406468 · arXiv
Cited by
Sorted by citations
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
“The image diffusion process directly provides some control over color variation [53] and inpainting [66, 7].”
BackgroundMethodLarge language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
“However, active development has reduced the incidence of biased and dangerous output, with GPT-4 evaluated as 82% less likely than its predecessor, GPT-3.”
BackgroundScalable Diffusion Models with Transformers
Peebles, W., & Xie, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00387 · Cited by 2,284
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
“Next, we show comparisons of recontextualization of a subject clock, with distinctive features using our method and prompt engineering using vanilla Imagen [56] and the public API of DALL-E 2 [51].”
BackgroundMethodInfluentialDiffusion Models in Vision: A Survey
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261988 · Cited by 2,189
Metadata
- Authors
- Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M.
- ORCID
- Chen, Mark
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.06125
- OpenAlex
- W4224035735
- Cited by
- 2,511
- References
- 66
- Citations by year
- 2026: 46 · 2025: 429 · 2024: 1113 · 2023: 859 · 2022: 63
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- text-to-image generation, diffusion models, image priors, image variation, photorealism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
