Hierarchical Text-Conditional Image Generation with CLIP Latents
Open accessAbstract
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Advanced techniques that edit individual training frames, such as text-to-image models, are slow, costly, and error-prone.
“While more sophisticated methods for partially editing individual frames during training exist (e.g., text-to-image models [13], [14]), they are slow, computationally expensive and error-prone.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [13]
The Introduction identifies unCLIP as a text-to-image system, and §3.3 (p. 7) describes language-guided manipulation of existing images. §7 (p. 17) reports difficulties with rendered text and complex-scene details, and says improving the latter would require more training and inference compute. The paper does not establish that editing individual training frames with text-to-image models is generally slow, costly, and error-prone. (LLM, conf=0.84)
Experimental — accuracy not yet measuredGenerative machines, such as text-to-image algorithms, are contributing to the variety of cultural traits.
“Generative machines, such as text-to-image algorithms, are contributing to the variety of cultural traits.”
From Machine culture (2023) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Generative Adversarial Networks
Cohen, G., & Giryes, R. (2021) · International Conference on Computing Communication and Networking Technologies · doi:10.1109/icccnt56998.2023.10306417 · Cited by 28,188 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“To this end, we generated 512 “artistic” captions using GPT-3 [4] by prompting it with captions for existing artwork (both real and AI generated).”
MethodSelf-citationAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · cited 2× · see citing works
“[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
BackgroundMethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3× · see citing works
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundMethodSelf-citationLIII. On lines and planes of closest fit to systems of points in space
Karl Pearson, F. R. S. (1901) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440109462720 · Cited by 13,658 · cited 4× · see citing works
“After applying PCA, we order the principal components by decreasing eigenvalue magnitude, quantize each of the 319 dimensions into 1,024 discrete buckets, and predict the resulting sequence using a decoder-only Transformer [53] model with a causal attention mask.”
MethodInfluentialDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477 · see citing works
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362 · see citing works
Self-citationNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · see citing works
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · cited 1× · see citing works
“Concurrently, diffusion models [46, 48, 25] have emerged as a promising generative modeling framework, pushing the state-of-the-art on image and video generation tasks [11, 26, 24].”
BackgroundTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 2× · see citing works
“[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
MethodDenoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507 · see citing works
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 6× · see citing works
“Like GLIDE and DALL-E, unCLIP is not directly trained on the MS-COCO training set, but can still generalize to the validation set zero-shot.”
BackgroundMethodResultInfluentialSelf-citationAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · cited 1× · see citing works
“Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56, 63, 49, 58, 57].”
BackgroundImage Super-Resolution Via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204461 · Cited by 1,829 · cited 2× · see citing works
“For the first upsampling stage, we use gaussian blur [43], and for the second, we use a more diverse BSR degradation [42,”
MethodImage Super-Resolution via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2021) · arXiv · doi:10.48550/arxiv.2104.07636 · Cited by 1,829 · see citing works
Improved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550 · see citing works
Self-citationGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 6× · see citing works
“Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.”
BackgroundMethodResultInfluentialSelf-citationGenerative Visual Manipulation on the Natural Image Manifold
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_36 · Cited by 1,215 · see citing works
Classifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126 · cited 4× · see citing works
“We enable classifier-free guidance [24] by randomly setting the CLIP embeddings to zero (or a learned embedding) 10% of the time, and randomly dropping the text caption 50% of the time.”
BackgroundMethodInfluentialStyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen‐Or, D., & Lischinski, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,079 · cited 1× · see citing works
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodAVA: A large-scale database for aesthetic visual analysis
Murray, N., Marchesotti, L., & Perronnin, F. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247954 · Cited by 973 · cited 1× · see citing works
“Next, we trained a CLIP linear probe to predict human aesthetic judgments using the AVA dataset [33] (Appendix A).”
MethodDesigning a Practical Degradation Model for Deep Blind Image Super-Resolution
Zhang, K., Liang, J., Gool, L. V., & Timofte, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00475 · Cited by 814 · cited 1× · see citing works
“For the first upsampling stage, we use gaussian blur [43], and for the second, we use a more diverse BSR degradation [42, 59].”
MethodHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · see citing works
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01043 · Cited by 720 · cited 1× · see citing works
“Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.”
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709 · cited 1× · see citing works
“[41] trains a multi-layer discrete autoencoder, allowing them to first sample coarse-grained latent codes and then use this as conditioning information when sampling higher-resolution latent codes.”
BackgroundDM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis
Zhu, M., Pan, P., Chen, W., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00595 · Cited by 605 · see citing works
Sharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521 · cited 2× · see citing works
“Both models are trained with learning rate 3× 10−4 and SAM [15] with ρ = 0.1, where the perturbations are applied independently by the replicas, each of which uses batch size 64.”
BackgroundInfluentialCascaded Diffusion Models for High Fidelity Image Generation
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., & Salimans, T. (2021) · arXiv · doi:10.48550/arxiv.2106.15282 · Cited by 514 · cited 1× · see citing works
“Concurrently, diffusion models [46, 48, 25] have emerged as a promising generative modeling framework, pushing the state-of-the-art on image and video generation tasks [11, 26, 24].”
BackgroundNVAE: A Deep Hierarchical Variational AutoencoderPreprint
Vahdat, A., & Kautz, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03898 · Cited by 514 · cited 1× · see citing works
“Child, Vahdat and Kautz [5, 50] generate images using VAEs with a hierarchy of latent codes that increase progressively with resolution.”
BackgroundGal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., & Cohen‐Or, D. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530164 · Cited by 503 · cited 1× · see citing works
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodXia, W., Zhang, Y., Yang, Y., Xue, J., Zhou, B., & Yang, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3181070 · Cited by 496 · see citing works
Improved Techniques for Training Score-Based Generative Models
Song, Y., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09011 · Cited by 482 · cited 2× · see citing works
“We use diffusion models [25, 48] to produce images conditioned on CLIP image embeddings (and optionally text captions).”
BackgroundMethodContrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · cited 1× · see citing works
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundCogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450 · cited 1× · see citing works
“Other works have adapted the VQ-VAE approach [52] to text-conditional image generation by training autoregressive transformers on sequences of text tokens followed by image tokens [40, 12, 1].”
MethodCross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 359 · cited 1× · see citing works
“Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56, 63, 49, 58, 57].”
BackgroundVirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · cited 1× · see citing works
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundDiffusion Autoencoders: Toward a Meaningful and Decodable Representation
Preechakul, K., Chatthee, N., Wizadwongsa, S., & Suwajanakorn, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01036 · Cited by 338 · cited 1× · see citing works
“[38] propose an autoencoder framework where diffusion models are used to render latent variables as images, and a second diffusion model is used to generate these latents (similar to our diffusion”
MethodMake-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · arXiv · doi:10.48550/arxiv.2203.13131 · Cited by 332 · cited 1× · see citing works
“[17] conditions a generative image model on segmentation masks, allowing for a generative process that first samples a semantic map of an image and then conditions the generated image on this information.”
BackgroundSLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 285 · cited 1× · see citing works
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundHow Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 272 · see citing works
Towards Language-Free Training for Text-to-Image Generation
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01738 · Cited by 250 · cited 2× · see citing works
“[61] condition GAN models on randomly perturbed CLIP image embeddings, finding that these models can generalize to CLIP text embeddings to produce text-conditional images.”
BackgroundMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 240 · cited 4× · see citing works
“Figure 5: Variations of images featuring typographic attacks [20] paired with the CLIP model’s predicted probabilities across three labels.”
BackgroundMethodInfluentialScore-based Generative Modeling in Latent Space
Vahdat, A., Kreis, K., & Kautz, J. (2021) · arXiv · doi:10.48550/arxiv.2106.05931 · Cited by 224 · cited 1× · see citing works
“[51] use a score-based model for the latent space of a VAE, while Rombach et al.”
MethodAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial NetworksPreprint
Xu, T., Zhang, P., Huang, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10485 · Cited by 159 · see citing works
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Child, R. (2020) · arXiv · doi:10.48550/arxiv.2011.10650 · Cited by 140 · cited 1× · see citing works
“Child, Vahdat and Kautz [5, 50] generate images using VAEs with a hierarchy of latent codes that increase progressively with resolution.”
BackgroundLearning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 120 · cited 1× · see citing works
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundGenerative Visual Manipulation on the Natural Image ManifoldPreprint
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03552 · Cited by 119 · see citing works
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic ModelsPreprint
Bao, F., Li, C., Zhu, J. J., & Zhang, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.06503 · Cited by 109 · see citing works
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPPreprint
Wang, Z., Liu, W., He, Q., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2203.00386 · Cited by 97 · cited 2× · see citing works
“Another possibility is to feed the decoder the CLIP text embedding as if it were an image embedding, as previously observed [61, 54].”
MethodImproving Text-to-Image Synthesis Using Contrastive Learning
Ye, H., Yang, X., Takác, M., et al. (2021) · British Machine Vision Conference · doi:10.5244/c.35.137 · Cited by 93
Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search
Galatolo, F., Cimino, M., & Vaglini, G. (2021) · International Conference on Image Processing and Vision Engineering · doi:10.5220/0010503701660174 · Cited by 93 · cited 1× · see citing works
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodHigh Fidelity Visualization of What Your Self-Supervised Representation Knows About
Bordes, F., Balestriero, R., & Vincent, P. (2021) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2112.09164 · Cited by 84 · cited 1×
“[3] train diffusion models conditioned on image representations from contrastive models.”
BackgroundStyleGAN-NADA: CLIP-Guided Domain Adaptation of Image GeneratorsPreprint
Gal, R., Patashnik, O., Maron, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00946 · Cited by 64 · see citing works
Ruxton, G. (2006) · doi:10.1641/0006-3568(2006)056[0446:tac]2.0.co;2 · Cited by 59 · cited 1×
“[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
MethodCLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D., Kopp, M., Klambauer, G., Bitto, A., et al. (2022) · neural information processing systems · doi:10.52202/068431-1487 · Cited by 54 · cited 1× · see citing works
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · see citing works
Vector Quantized Diffusion Model for Text-to-Image SynthesisPreprint
Gu, S., Chen, D., Bao, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14822 · Cited by 39 · see citing works
LAFITE: Towards Language-Free Training for Text-to-Image Generation
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T. (2021) · arXiv · doi:10.48550/arxiv.2111.13792 · Cited by 38 · see citing works
Cross-Modal Contrastive Learning for Text-to-Image GenerationPreprint
Zhang, H., Koh, J. Y., Baldridge, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04702 · Cited by 34 · see citing works
DF-GAN: A Simple and Effective Baseline for Text-to-Image SynthesisPreprint
Ming, T., Tang, H., Wu, F. Q., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05865 · Cited by 26 · see citing works
Improving Text-to-Image Synthesis Using Contrastive LearningPreprint
Ye, H., Yang, X., Takáč, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02423 · Cited by 24 · see citing works
SLIP: Self-supervision meets Language-Image Pre-trainingPreprint
Mu, N., Kirillov, A. M., Wagner, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.12750 · Cited by 13 · see citing works
Diffusion Autoencoders: Toward a Meaningful and Decodable RepresentationPreprint
Preechakul, K., Chatthee, N., Wizadwongsa, S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.15640 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [23][23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
- [32][32] Ryan Murdock. The Big Sleep. https://twitter.com/advadnoun/status/ 1351038053033406468, 2021. 20 https://openreview.net/forum?id=qw674L9PfQE https://openreview.net/forum?id=qw674L9PfQE https://arxiv.org/abs/2203.13131 https://arxiv.org/abs/2108.00946 https://arxiv.org/abs/2102.01645 https://arxiv.org/abs/1406.2661 https://arxiv.org/abs/2111.14822 https://openreview.net/forum?id=qw8AKxfYbI https://openreview.net/forum?id=qw8AKxfYbI https://arxiv.org/abs/2006.11239 https://arxiv.org/abs/2106.15282 https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1405.0312 https://arxiv.org/abs/1711.05101 https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://arxiv.org/abs/2112.12750 https://twitter.com/advadnoun/status/1351038053033406468 https://twitter.com/advadnoun/status/1351038053033406468 · arXiv
Cited by
Sorted by citations
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547 · see citing works
“The image diffusion process directly provides some control over color variation [53] and inpainting [66, 7].”
BackgroundMethodLarge language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
“However, active development has reduced the incidence of biased and dangerous output, with GPT-4 evaluated as 82% less likely than its predecessor, GPT-3.”
BackgroundScalable Diffusion Models with Transformers
Peebles, W., & Xie, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00387 · Cited by 2,284 · see citing works
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · see citing works
“Next, we show comparisons of recontextualization of a subject clock, with distinctive features using our method and prompt engineering using vanilla Imagen [56] and the public API of DALL-E 2 [51].”
BackgroundMethodInfluentialDiffusion Models in Vision: A Survey
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261988 · Cited by 1,965 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704 · see citing works
“DALLE-2 [216] employs a similar strategy by training a diffusion model on the CLIP image embedding space, followed by training a separate decoder to create images based on the CLIP image embeddings.”
BackgroundMethodInfluentialA Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development
Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123618 · Cited by 1,452 · see citing works
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466 · see citing works
“Recent text-to-image diffusion models [41,48,51,54] have shown to generate realistic images from arbitrary text captions.”
BackgroundInfluentialscGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · see citing works
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., & Shan, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28226 · Cited by 1,026 · see citing works
Magic3D: High-Resolution Text-to-3D Content Creation
Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., & Lin, T.-Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00037 · Cited by 909 · see citing works
“) to generate high-quality images of objects and scenes [2, 33, 34, 36].”
BackgroundImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · see citing works
“In DALLE-2, to produce images from text prompts, the image generation model relies on text embeddings produced by the pre-trained CLIP-L/14 text encoder.”
MethodInfluentialFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
“– Dialogue 1: Available from DALL·E 2 [89].”
BackgroundZero-1-to-3: Zero-shot One Image to 3D Object
Liu, R., Wu, R., Hoorick, B. V., Tokmakov, P., Zakharov, S., & Vondrick, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00853 · Cited by 888 · see citing works
Diffusion models in medical imaging: A comprehensive survey
Kazerouni, A., Aghdam, E. K., Heidari, M., Azad, R., Fayyaz, M., Hacihaliloglu, I., & Merhof, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102846 · Cited by 850 · see citing works
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00582 · Cited by 813 · see citing works
“To achieve this feat, we take advantage of the recent success of text-to-image diffusion models [47, 50, 53].”
BackgroundMethodObjaverse: A Universe of Annotated 3D Objects
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsanit, K., Kembhavi, A., & Farhadi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01263 · Cited by 725 · see citing works
“…pairs such as Conceptual Captions [63], WIT [65], and LAION [61,62] have seen increased popularity recently as they have been used to train impressive models for vision-language representation learning [27,30,55], text-to-image generation [30,56,57,59], and vision-language multitasking [8,9,68,74].”
BackgroundNull-text Inversion for Editing Real Images using Guided Diffusion Models
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00585 · Cited by 747 · see citing works
“DALL-E 2 [29] demonstrates editing capabilities by interpolating between the CLIP image embeddings and a normalised difference of the CLIP text embeddings produced from two text descriptions.”
InfluentialAlign Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02161 · Cited by 724 · see citing works
“The most powerful models today are built on generative adversarial networks [21, 38–40, 75], autoregressive transformers [15, 63, 105], and most recently diffusion models [10, 28, 29, 57, 58, 62, 65, 68, 79, 82].”
BackgroundReproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · see citing works
“In particular, large models have led to substantial advances in speech [56], language [8, 17, 28, 57], vision [38, 84], and multi-modal language-vision settings [33,54,55,59,62].”
BackgroundMulti-Concept Customization of Text-to-Image Diffusion
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00192 · Cited by 674 · see citing works
“While earlier works [26,56,67, 78, 80, 87] were limited to a few classes [38, 71], recent text-to-image models [11,45,53,54,57,60,79], trained on extremely large-scale data, have demonstrated remarkable generalization ability.”
BackgroundEfficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
Menghani, G. (2023) · ACM Computing Surveys · doi:10.1145/3578938 · Cited by 660 · see citing works
“Generative models like DALL-E [109], Imagen [116], and Parti [144] require hundreds of GPUs/TPUs over multiple weeks, which implies more than $100K in training costs.”
MethodPlug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
Tumanyan, N., Geyer, M., Bagon, S., & Dekel, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00191 · Cited by 644 · see citing works
“Recently, text-to-image generative models [13,29,35,37, 41] have demonstrated unprecedented capabilities in generating high-quality and diverse images from text, capturing complex visual concepts (e.”
BackgroundInfluentialDiffusionDet: Diffusion Model for Object Detection
Chen, S., Sun, P., Song, Y., & Luo, P. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01816 · Cited by 657 · see citing works
“Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”
BackgroundGLIGEN: Open-Set Grounded Text-to-Image Generation
Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., & Lee, Y. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02156 · Cited by 610 · see citing works
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., & Shou, M. Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00701 · Cited by 622 · see citing works
“GLIDE [31] proposes classifier-free guidance [18] in the diffusion model to improve image quality, while DALLE-2 [36] improves text-image alignments using CLIP [35] feature space.”
A Survey on Generative Diffusion Models
Cao, H., Tan, C., Gao, Z., Xu, Y., Chen, G., Heng, P., & Li, S. Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3361474 · Cited by 679 · see citing works
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01855 · Cited by 540 · see citing works
“Meanwhile, CLIP features also play a central role in representation leaning [66, 98], AI generated content [74, 79, 80] and large dataset filtering [10, 81, 82], etc.”
BackgroundMethodZhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480 · see citing works
“Examples include the Diffusion-based text-to-speech model by [73] and various Diffusion-based image generation models such as DALL-E2 [144], Imagen [152], and Stable diffusion [151].”
MethodMotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437 · see citing works
Transfer learning in environmental remote sensing
Ma, Y., Chen, S., Ermon, S., & Lobell, D. B. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113924 · Cited by 448 · see citing works
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services
Xu, M., Du, H., Niyato, D. T., Kang, J., Xiong, Z., Mao, S., Han, Z., Jamalipour, A., Kim, D. I., Shen, X., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3353265 · Cited by 475 · see citing works
“Compared to traditional AI, AIGC technology requires overall technical maturity, transparency, robustness, impartiality, and insightful-ness of the algorithm for effective application implementation.”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., & Zheng, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02062 · Cited by 443 · see citing works
“By conditioning the text prompt into the diffusion model, various text-to-image diffusion models GLIDE [19], VQ-Diffusion [8], LDM [27], DALL·E 2 [24], and Imagen [30] have been developed.”
BackgroundMethodScaling up GANs for Text-to-Image Synthesis
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00976 · Cited by 426 · see citing works
“GLIDE [46], DALL · E 2 [53], and Imagen [59] are representative diffusion models that show realistic outputs with the aid of a pretrained language encoder [50,52].”
InfluentialTowards Universal Fake Image Detectors that Generalize Across Generative Models
Ojha, U., Li, Y., & Lee, Y. J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02345 · Cited by 428 · see citing works
“…is no longer a single source of fake images that needs to be dealt with: for example, synthesized images could take the form of realistic human faces generated using generative adversarial networks [29], or they could take the form of complex scenes generated using diffusion models [41, 44].”
BackgroundInfluentialPolak, M. P., & Morgan, D. (2024) · Nature Communications · doi:10.1038/s41467-024-45914-8 · Cited by 365 · see citing works
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., & Mello, S. D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00289 · Cited by 382 · see citing works
“Not surprisingly, we find that despite both generative models being diffusion-based, our approach of using a model trained on Internet-scale data is more effective at generalizing to open-vocabulary categories.”
BackgroundMethodInfluentialGenerative Artificial Intelligence: Trends and Prospects
Jovanovic, M., & Campbell, M. (2022) · Computer · doi:10.1109/mc.2022.3192720 · Cited by 357 · see citing works
Inversion-based Style Transfer with Diffusion Models
Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00978 · Cited by 374 · see citing works
“Ramesh [40] develop a textconditional image generator based on the diffusion models and the inverted CLIP.”
BackgroundMethodBengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2023) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 286 · see citing works
“A prime example is ‘Parti’ by Google, and DALL E-2 [145] by openAI are proficient tools focused on creating videos solely from textual prompts.”
MethodPaint by Example: Exemplar-based Image Editing with Diffusion Models
Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., & Wen, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01763 · Cited by 380 · see citing works
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation
Chen, R., Chen, Y., Jiao, N., & Jia, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02033 · Cited by 392 · see citing works
Executing your Commands via Motion Diffusion in Latent Space
Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., & Yu, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01726 · Cited by 370 · see citing works
“In light of the tremendous success of the diffusion-based generative models on other domains [53, 61, 56, 22, 79, 73], the latter category methods [75, 69, 28] propose a conditional diffusion model for human motion synthesis to learn a more powerful probabilistic mapping from the textual descriptors to human motion se…”
MethodScore Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation
Wang, H., Du, X., Li, J., Yeh, R. A., & Shakhnarovich, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01214 · Cited by 355 · see citing works
Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., & Wang, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00836 · Cited by 336 · see citing works
“Both visible and invisible video watermarks will be inserted into the produced video for generated content identification similar to Dall-E [30] and Imagen [32].”
MethodLatent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Metzer, G., Richardson, E., Patashnik, O., Giryes, R., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01218 · Cited by 336 · see citing works
“Text-guided image generation has seen tremendous success in recent years, primarily due to the breathtaking development in Language-Image models [25, 28, 36] and diffusion models [14, 21, 37–40].”
MethodGenerating Human Motion from Textual Descriptions with Discrete Representations
Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., & Ying, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01415 · Cited by 328 · see citing works
DIRE for Diffusion-Generated Image Detection
Wang, Z., Bao, J., Zhou, W., Wang, W., Hu, H., Chen, H., & Li, H. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02051 · Cited by 354 · see citing works
“…models, in which four subset images (ADM [10], DDPM [17], iDDPM [32], PNDM [23]) are generated by un-conditional diffusion models and the other seven (LDM [37], SD-v1 [37], SD-v2 [37], VQ-Diffusion [14], IF [39] 1 , DALLE-2 [35], and Midjourney 2 ) are generated by text-to-image diffusion models.”
MethodInfluentialMotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 566 · see citing works
“Inspiredbytherecentprogressofthetext-conditioned image generation [15], [16], [17], [18], we propose to incorporate Denoising Diffusion Probabilistic Models (DDPM) [19] into motion generation.”
MethodOn Distillation of Guided Diffusion Models
Meng, C., Rombach, R., Gao, R., Kingma, D. P., Ermon, S., Ho, J., & Salimans, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01374 · Cited by 318 · see citing works
“, 2021), DALL·E 2 (Ramesh et al., 2022), and Imagen (Saharia et al.”
Segment Anything for Microscopy
Archit, A., Freckmann, L., Nair, S., Khalid, N., Hilt, P., Rajashekar, V., Freitag, M., Teuber, C., Spitzner, M., Contreras, C. T., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02580-4 · Cited by 354 · see citing works
“Our main contributions are: • A training procedure to fine-tune SAM, including a new decoder that provides improved instance segmentation results.”
On The Detection of Synthetic Images Generated by Diffusion Models
Corvi, R., Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., & Verdoliva, L. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095167 · Cited by 304 · see citing works
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · see citing works
Generative Diffusion Prior for Unified Image Restoration and Enhancement
Fei, B., Lyu, Z., Pan, L., Zhang, J., Yang, W., Luo, T., Zhang, B., & Dai, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00958 · Cited by 300 · see citing works
“For instance, Generative Adversarial Networks (GANs) [20] that are trained on a large dataset of clean images learn rich knowledge of the real-world scenes have succeeded in various linear inverse problems through GAN inversion [21, 51, 58].”
BackgroundInfluentialMedSegDiff-V2: Diffusion-Based Medical Image Segmentation with Transformer
Wu, J., Ji, W., Fu, H., Xu, M., Jin, Y., & Xu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28418 · Cited by 288 · see citing works
AI Art and its Impact on Artists
Jiang, H. H., Brown, L., Cheng, J., Khan, M., Gupta, A., Workman, D., Hanna, A., Flowers, J., & Gebru, T. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604681 · Cited by 265 · see citing works
“architecture [102] but with a training dataset that is opaque to the public.”
BackgroundAll are Worth Words: A ViT Backbone for Diffusion Models
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., & Zhu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02171 · Cited by 296 · see citing works
Ross, S., Martinez, F., Houde, S., Muller, M. J., & Weisz, J. D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584037 · Cited by 272 · see citing works
“Many popular text-to-image models, such as DALL-E 2 [66] and Stable Diffusion [72], operate in a one-shot fashion, in which the user specifies a prompt, clicks a button, and gets results.”
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02240 · Cited by 255 · see citing works
“, [53, 55]); the pretrained encoder can improve multimodal and even unimodal visual tasks.”
BackgroundLin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., & He, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01469 · Cited by 263 · see citing works
“, DALL-E2 [35], indicates the strong relation between texts and corresponding components in the image.”
BackgroundSelf-citationMultimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 309 · see citing works
“In 2022, DALL-E 2, a product in Ope-nAI, utilizes a diffusion model conditioned on CLIP image embeddings [23].”
MethodInfluentialLarge AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · see citing works
“Therearegenerallytwoclasses of methods for such a task: autoregressive model [65], [66], [67] and diffusion model [67], [68], [69], [70].”
MethodELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation
Wei, Y., Zhang, Y., Ji, Z., Bai, J., Lei Zhang, A., & Zuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01461 · Cited by 274 · see citing works
“Recently, large-scale diffusion models [3, 29, 32, 34] have demonstrated impressive superiority in text-to-image generation.”
BackgroundMethodInfluentialAuditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
“The same holds of other automatic content producers, such as DALL·E 2 [46].”
BackgroundScope of machine learning in materials research—A review
Mobarak, M. H., Mimona, M. A., Islam, M. A., Hossain, N., Zohura, F. T., Imtiaz, I., & Rimon, M. I. H. (2023) · Applied Surface Science Advances · doi:10.1016/j.apsadv.2023.100523 · Cited by 243 · see citing works
Denoising Diffusion Models for Plug-and-Play Image Restoration
Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R., & Gool, L. V. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00129 · Cited by 240 · see citing works
“Recently, diffusion models have demonstrated the ability to generate images with higher quality [14, 41, 43] than previous generative models such as GANs, VAEs and NFs even when starting from pure Gaussian noise.”
BackgroundVideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00698 · Cited by 257 · see citing works
PhysDiff: Physics-Guided Human Motion Diffusion Model
Yuan, Y., Song, J., Iqbal, U., Vahdat, A., & Kautz, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01467 · Cited by 246 · see citing works
“Score-based denoising diffusion models [73, 25, 75, 74] have achieved great successes in various applications such as image generation [67, 62, 66, 72, 79], text-to-speech synthesis [35], 3D shape generation [95, 45, 91], machine learning security [54], as well as human motion generation [77, 92,…”
BackgroundFoundation models in robotics: Applications, challenges, and the future
Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., et al. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241281508 · Cited by 267 · see citing works
“For example, the GPT family [81] builds on the original transformer model by using a modification introduced in [82] that removes the transformer encoder blocks entirely, retaining just the transformer decoder blocks.”
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model
Xie, S., Zhang, Z., Lin, Z., Hinz, T., & Zhang, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02148 · Cited by 231 · see citing works
ConceptFusion: Open-set multimodal 3D mapping
Jatavallabhula, K., Kuwajerwala, A., Gu, Q., Omama, M., Iyer, G., Saryazdi, S., Chen, T., Maalouf, A., Li, S., Keetha, N., et al. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.066 · Cited by 228 · see citing works
Machine Learning-Guided Protein Engineering
Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlář, J., Damborský, J., Pluskal, T., Šivic, J., & Mazurenko, S. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c02743 · Cited by 214 · see citing works
“This has led to major breakthroughs in image generation and the emergence of the highly successful models, such as DALL-E 2 38 or Stable Diffusion.”
Chung, J., Hyun, S., & Heo, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00840 · Cited by 241 · see citing works
“Following the remarkable advances achieved by pre-trained text-to-image DMs [32, 42], there have been numerous image editing works [2, 6, 19, 37] utilizing these DMs. Notably, Prompt-to-Prompt [14] proposed text-based lo-cal image editing by manipulating the cross-attention map.”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltrušaitis, T., Shen, J., Chen, D., Wen, F., Chen, Q., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00443 · Cited by 215 · see citing works
“The most remarkable success happens in text-to-image synthesis [19, 39, 47, 50, 52], which serves as a foundation model and enables various appealing applications [21, 51, 66] previously unattainable.”
BackgroundDiffsound: Discrete Diffusion Model for Text-to-Sound Generation
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., & Yu, D. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268730 · Cited by 222 · see citing works
“AR models [22], [23] have shown powerful generation capability and have been applied for image generation [7], [24], [25], [26], [27], [28], [29], speech synthesis [30] and sound generation [3], [4].”
BackgroundRethinking FID: Towards a Better Evaluation Metric for Image Generation
Jayasumana, S., Ramalingam, S., Veit, A., Gläsner, D., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00889 · Cited by 243 · see citing works
“Recent models such as [16, 19–21 , 26 ] have been incred-ibly Fr ´ echet Inception Distance (FID) [12], may disagree with the gold standard, human raters, in some important cases; and is thus ill-suited for this purpose.”
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219 · see citing works
“Inspired by existing content policies of deployed text-to-image DMs such as DALL-E, the following text prompt is used to define a general inappropriate safety concept: “hate, harassment, violence, suffering, humiliation, harm, suicide, sexual, nudity, bodily fluids, blood, obscene gestures, illegal activity, drug use,…”
BackgroundInfluentialHierarchical Fine-Grained Image Forgery Detection and Localization
Guo, X., Liu, X., Ren, Z., Grosz, S. A., Masi, I., & Liu, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00308 · Cited by 221 · see citing works
Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering
Oppenlaender, J., Linder, R., & Silvennoinen, J. M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2431761 · Cited by 308 · see citing works
“State-of-the-art systems, such as ChatGPT [37] and DALL-E 2 [46], demonstrate impressive performance in understanding textual input prompts and user intent.”
StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models
Wang, Z., Zhao, L., & Xing, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00706 · Cited by 211 · see citing works
“Fortunately, inspired by the recent vision-language model CLIP [68] that encapsulates knowledgeable semantic information of not only the photograph domain but also the artistic domain [18, 69, 49], we can use its image encoder as our projector E off the shelf.”
BackgroundMake-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
Tang, J., Wang, T., Zhang, B., Zhang, T., Yi, R., Ma, L., & Chen, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02086 · Cited by 220 · see citing works
“Unlike the scarcity of 3D models, images are much more readily available, and recent advancements in diffusion models have sparked a revolution in 2D image generation [34, 39, 38, 52, 4].”
BackgroundKar, A. K., Varsha, P. S., & Rajan, S. (2023) · Global Journal of Flexible Systems Management · doi:10.1007/s40171-023-00356-x · Cited by 209 · see citing works
“Hence, DALLE-2 is becoming extremely prevalent as a word-to-image generator (Kim, 2022) to furnish viewers, who may be unfamiliar with the variety of AI art images, with what they are looking for (Chatterjee, 2022).”
MethodInfluentialLarge-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189 · see citing works
Artificial Intelligence and Democracy: A Conceptual Framework
Jungherr, A. M. (2023) · Social Media + Society · doi:10.1177/20563051231186353 · Cited by 217 · see citing works
“Recently, transformer models, a subclass of deep learning, have proven to be highly promising (Parmar et al., 2018; Vaswani et al., 2017) in the autonomous generation of text, image, and video content (Brown et al., 2020; Ramesh et al., 2022).”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning
Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00673 · Cited by 224 · see citing works
“In particular, the field of text-guided image synthesis has seen significant growth with the introduction of diffusion models, achieving state-of-the-art results in large-scale text-to-image synthesis tasks [39, 47, 53, 51, 4].”
BackgroundA Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,099 · see citing works
“DALL-E [184], which is now Craiyon, was first built on VQ-VAE and CLIP, then diffusion was also applied to this product, becoming DALL-E-2 [183].”
Human Preference Score: Better Aligning Text-to-image Models with Human Preference
Wu, X., Sun, K., Zhu, F., Zhao, R., & Li, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00200 · Cited by 214 · see citing works
“Some recent studies [26, 30, 3] use the CLIP [29] model as a proxy for human judgment to evaluate the alignment between generated images and text prompts.”
BackgroundMethodInfluentialAudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y.-P., Wang, W., Wang, Y., & Plumbley, M. D. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399607 · Cited by 213 · see citing works
“[39], [40], image restoration [41], speech generation [42], [43], [44], and video generation [45], [46].”
BackgroundDE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models
Sha, Z., Li, Z., Yu, N., & Zhang, Y. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3616588 · Cited by 175 · see citing works
“We can clearly observe that among the top twenty topics, “skis” and “snowboard” are ranked the highest.”
InfluentialTen years of generative adversarial nets (GANs): a survey of the state-of-the-art
Chakraborty, T., S, U. R. K., Naik, S. M., Panja, M., & Manvitha, B. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad1f77 · Cited by 198 · see citing works
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00586 · Cited by 206 · see citing works
“These mega-datasets have the power to forge commercial models like DALL·E [54] and Stable Diffusion [56], but also bring with them a number of legal and ethical risks [7].”
BackgroundAGIQA-3K: An Open Database for AI-Generated Image Quality Assessment
Li, C., Zhang, Z., Wu, H., Sun, W., Min, X., Liu, X., Zhai, G., & Lin, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3319020 · Cited by 206 · see citing works
“Through horizontal comparison, we found that Midjourney has relatively good versatility in different styles, but the versatility of DALLE2 and Stable Diffusion still need to be improved.”
MethodInfluentialText-to-3D using Gaussian Splatting
Chen, Z., Wang, F., Wang, Y., & Liu, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02022 · Cited by 202 · see citing works
“One of the most successful applications of diffusion models lies in text-to-image generation, where they have shown remarkable progress in generating realistic images from text prompts (Ho & Salimans, 2022; Ramesh et al., 2022; Alex et al., 2023).”
BackgroundmPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.488 · Cited by 179 · see citing works
SparseFusion: Distilling View-Conditioned Diffusion for 3D Reconstruction
Zhou, Z., & Tulsiani, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01211 · Cited by 174 · see citing works
“Several works extend upon denoising diffusion models [13, 37] to achieve impressive applications, such as generating images from text [23, 30] and placing foreground objects in different backgrounds [29].”
BackgroundMethodDiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., & Chau, D. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.51 · Cited by 244 · see citing works
Unleashing Text-to-Image Diffusion Models for Visual Perception
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., & Lu, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00527 · Cited by 188 · see citing works
“We also believe that text-guided generative models other than diffusion models[47, 42, 8] can also fit in VPD, which we leave to future work.”
BackgroundPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2023) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 206 · see citing works
“, DALLE-2 [243], Stable Diffusion [252], and Imagen [256]) have been proposed and achieved excellent performance on text-based image generation.”
BackgroundWhat’s in a text-to-image prompt? The potential of stable diffusion in visual arts education
Dehouche, N., & Dehouche, K. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16757 · Cited by 217 · see citing works
- Authors
- Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M.
- ORCID
- Chen, Mark
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.06125
- OpenAlex
- W4224035735
- Cited by
- 2,511
- References
- 66
- Citations by year
- 2026: 46 · 2025: 429 · 2024: 1113 · 2023: 859 · 2022: 63
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- text-to-image generation, diffusion models, image priors, image variation, photorealism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
