The DOI matches its DataCite record; 69 of the 69 references it cites were found in a registry or an index.
Hierarchical Text-Conditional Image Generation with CLIP Latents
Open accessAbstract
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While more sophisticated methods for partially editing individual frames during training exist (e.g., text-to-image models [13], [14]), they are slow, computationally expensive and error-prone.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredIn the citing paper
“Generative machines, such as text-to-image algorithms, are contributing to the variety of cultural traits.”
From Machine culture (2023) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“To this end, we generated 512 “artistic” captions using GPT-3 [4] by prompting it with captions for existing artwork (both real and AI generated).”
MethodSelf-citationHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 2×
“[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
BackgroundMethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundMethodSelf-citationLIII. On lines and planes of closest fit to systems of points in space
Pearson, K. (1901) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440109462720 · Cited by 12,864 · cited 4×
“After applying PCA, we order the principal components by decreasing eigenvalue magnitude, quantize each of the 319 dimensions into 1,024 discrete buckets, and predict the resulting sequence using a decoder-only Transformer [53] model with a causal attention mask.”
MethodInfluentialDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Research works printed but not linked: 2
- [23][23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
- [32][32] Ryan Murdock. The Big Sleep. https://twitter.com/advadnoun/status/ 1351038053033406468, 2021. 20 https://openreview.net/forum?id=qw674L9PfQE https://openreview.net/forum?id=qw674L9PfQE https://arxiv.org/abs/2203.13131 https://arxiv.org/abs/2108.00946 https://arxiv.org/abs/2102.01645 https://arxiv.org/abs/1406.2661 https://arxiv.org/abs/2111.14822 https://openreview.net/forum?id=qw8AKxfYbI https://openreview.net/forum?id=qw8AKxfYbI https://arxiv.org/abs/2006.11239 https://arxiv.org/abs/2106.15282 https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1405.0312 https://arxiv.org/abs/1711.05101 https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://arxiv.org/abs/2112.12750 https://twitter.com/advadnoun/status/1351038053033406468 https://twitter.com/advadnoun/status/1351038053033406468 · arXiv
Cited by
Sorted by citations
Items 101–200 of 1,051
Unleashing Text-to-Image Diffusion Models for Visual Perception
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., & Lu, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00527 · Cited by 249
“We also believe that text-guided generative models other than diffusion models[47, 42, 8] can also fit in VPD, which we leave to future work.”
BackgroundPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2023) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 238
“, DALLE-2 [243], Stable Diffusion [252], and Imagen [256]) have been proposed and achieved excellent performance on text-based image generation.”
BackgroundWhat’s in a text-to-image prompt? The potential of stable diffusion in visual arts education
Dehouche, N., & Dehouche, K. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16757 · Cited by 240
Pix2Video: Video Editing using Image Diffusion
Ceylan, D., Huang, C.-H. P., & Mitra, N. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02121 · Cited by 234
“Several large scale text-to-image generation models [34, 39, 44], which operate on the pixel space have been presented, achieving very high quality results.”
GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis
Tao, M., Bao, B., Tang, H., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01366 · Cited by 194
“DALL-E2 [33] combines the CLIP representation and diffusion model to make a CLIP decoder.”
BackgroundMethodMotionDiffuser: Controllable Multi-Agent Motion Prediction Using Diffusion
Jiang, C. M., Cornman, A., Park, C., Sapp, B., Zhou, Y., & Anguelov, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00930 · Cited by 208
“Diffusion models have recently gained immense popularity due to their simplicity, strong capacity to represent complex, high dimensional and multimodal distributions, ability to solve inverse problems [4, 6, 24, 44], and effectiveness across multiple problem domains, including image generation [36, 37, 39], video gene…”
BackgroundMumuni, A., & Mumuni, F. (2024) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2024.01.002 · Cited by 221
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 206
“DALL-E [72] demonstrated how CLIP embeddings can help generate images with autoregressive and diffusion-based approaches.”
BackgroundMethodImagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y., Laszlo, S., Fleet, D. J., Soricut, R., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01761 · Cited by 202
“Most relevant to our work are Stable Diffusion [33] and GLIDE/DALL-E 2 [27,31], which are also diffusion models.”
InfluentialSpaText: Spatio-Textual Representation for Controllable Image Generation
Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., & Yin, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01762 · Cited by 202
“The current diffusion-based SoTA text-to-image models are DALL · E 2 [51], Imagen [59] and Stable Diffusion [56].”
BackgroundMethodInfluentialDiffuRec: A Diffusion Model for Sequential Recommendation
Li, Z., Sun, A., & Li, C. (2023) · ACM Transactions on Information Systems · doi:10.1145/3631116 · Cited by 224
Learning Video Representations from Large Language Models
Zhao, Y., Misra, I., Krähenbühl, P., & Girdhar, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00637 · Cited by 190
“Vision-language approaches [31, 49, 80] have pushed the state-of-the-art across a variety of tasks, including zero-shot classification [49], novel object detection [87], and even image generation [52].”
BackgroundWang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 199
“1) Transformer Architecture: ChatGPT is built upon the powerful foundation of the Transformer architecture, serving as the backbone for numerous state-of-the-art models [1], such as GPT-3 [25], DALL-E-2 [35], Codex [36], and Gopher [37].”
MethodText2Tex: Text-driven Texture Synthesis via Diffusion Models
Chen, D. Z., Siddiqui, Y., Lee, H.-Y., Tulyakov, S., & Nießner, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01701 · Cited by 247
“Recently, text-to-image generators have shown remarkable progress in the 2D domain leveraging diffusion model architectures, enabling high resolution 2D content generation based on textual descriptions [1, 48].”
BackgroundTraining-Free Layout Control with Cross-Attention Guidance
Chen, M., Laina, I., & Vedaldi, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00526 · Cited by 248
“We also note that, while DALLE-v2 [35] achieves the highest OA overall, it appears to struggle more with layout instructions compared to SD, as indicated by a lower VISOR cond score.”
BackgroundResultSelf-citation3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.-A., & Dou, Q. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103324 · Cited by 156
“Foundation models trained on massive data have demonstrated impressive ability on various general tasks [1, 2, 3], and are envisaged to impact downstream domains, especially where data collection and labeling are expensive.”
BackgroundMoFusion: A Framework for Denoising-Diffusion-Based Motion Synthesis
Dabral, R., Mughal, M. H., Golyanik, V., & Theobalt, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00941 · Cited by 202
“[49] also utilise text-image embeddings by CLIP and a diffusion decoder to achieve high-quality image synthesis.”
InfluentialWu, W., Zhao, Y., Shou, M. Z., Zhou, H., & Shen, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00117 · Cited by 223
“Recently, large-scale language-image generation (LLIG) models, such as DALL-E [48], and Stable Diffusion [49], have shown phenomenal generative semantic and compositional power, as shown in Fig.”
BackgroundMethodGenerative Novel View Synthesis with 3D-Aware Diffusion Models
Chan, E. R., Nagano, K., Chan, M. A., Bergman, A. W., Park, J. J., Levy, A., Aittala, M., Mello, S. D., Karras, T., & Wetzstein, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00389 · Cited by 211
“Specifically, conditional diffusion models [59, 57, 49, 55, 58] can be directly applied to the task of NVS.”
BackgroundMethodTAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
Doersch, C., Yang, Y., Vecerík, M., Gokay, D., Gupta, A., Aytar, Y., Carreira, J., & Zisserman, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00923 · Cited by 161
“Generative modeling of images and videos has experienced an explosion in popularity due to recent striking text-conditioned generation results using diffusion models [18, 23, 44, 49, 53].”
MethodIbrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.09419 · Cited by 574
One-Step Diffusion with Distribution Matching Distillation
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00632 · Cited by 395
“Diffusion models [21, 61, 63, 64, 71, 74] have revolutionized image generation, achieving unprecedented levels of realism and diversity with a stable training procedure.”
BackgroundDCFace: Synthetic Face Generation with Dual Condition Diffusion Model
Kim, M., Liu, F., Jain, A. K., & Liu, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01223 · Cited by 176
“The de-noising diffusion model (DDPM) has shown success in producing diverse samples in text-conditioned image generation [50].”
BackgroundMethodGenerative AI-Driven Semantic Communication Networks: Architecture, Technologies, and Applications
Liang, C., Du, H., Sun, Y., Niyato, D. T., Kang, J., Zhao, D., & Imran, M. A. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3435524 · Cited by 193
“In this section, we conceive important cases for GAI-driven SemCom networks, including autonomous driving, smart cities, and the Metaverse.”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1562-4 · Cited by 1,008
“…& Nichol, 2021; Ho & Salimans, 2021) is a widely-used technique to apply DPMs for conditional sampling, which is useful in text-to-image, image-to-image, and class-to-image applications (Dhariwal & Nichol, 2021; Saharia et al., 2022b; Rombach et al., 2022; Nichol et al., 2021; Ramesh et al., 2022).”
Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., & Wang, W. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3419446 · Cited by 150
“While AL research is relatively young, vision-language (VL) multimodal learning [20], the counterpart of AL multimodal learning, has been studied for decades and has contributed to the success of many vision-language applications, such as cross-modal search [21], [22], image and video captioning [23], [24], text-to-im…”
BackgroundMM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., & Guo, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00985 · Cited by 194
“For example, DALL·E 2 [33] and DiffWave [19] can create vivid art images and produce high-fidelity audio, respectively.”
BackgroundDDP: Diffusion Model for Dense Visual Prediction
Ji, Y., Chen, Z., Xie, E., Hong, L., Liu, X., Liu, Z., Lü, T., Li, Z., & Luo, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01987 · Cited by 201
“These methods now enable easy training and achieve superior results on various generative tasks [50, 58, 56, 53].”
BackgroundConcepts and applications of digital twins in healthcare and medicine
Zhang, K., Zhou, H.-Y., Baptista‐Hon, D. T., Gao, Y., Liu, X., Oermann, E. K., Xu, S. H., Jin, S., Zhang, J., Sun, Z., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.101028 · Cited by 185
“Deep generative models, including the Generative Pre-trained Transformer, 63 DALL-E (released by OpenAI), 64 as well as the booming diffusion model, 65 have been used to create dynamic metaverse environments.”
MethodCollaborative Diffusion for Multi-Modal Face Generation and Editing
Huang, Z., Chan, K. C. K., Jiang, Y., & Liu, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00589 · Cited by 162
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
Zhang, M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., Yang, L., & Liu, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00040 · Cited by 238
“DALL-E2 [19] attempts to bridge the text embedding and image embedding from the CLIP [18].”
BackgroundHagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 227
“Additionally, DALL-E is a generative model introduced by Ramesh et al. [57], [58], which produces images from textual descriptions.”
BackgroundMethodGuo, T., Guo, S., Wang, J., Tang, X., & Xu, W. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3302410 · Cited by 181
“AI is going through a paradigm shift with the rise of models (e.g., BERT, GPT-3, CLIP, DALL-E · 2 [15], [16], [17], [18]) trained on broad data using self-supervision at scale that can be adapted to a wide range of downstream tasks.”
BackgroundYour Diffusion Model is Secretly a Zero-Shot Classifier
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E. V., & Pathak, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00210 · Cited by 242
“Diffusion Models: Diffusion models [35, 69] have recently gained significant attention from the research community due to their ability to generate high-fidelity and diverse content like images [66, 54, 24], videos [68, 34, 77], 3D [58, 49], and audio [43, 51] from various input modalities like text.”
Notice of Removal: VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., & Tan, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00984 · Cited by 144
“For intuitive illustration, we use the decoder of DALL-E 2 [24] to generate images conditioned on the same latent embedding.”
BackgroundMethodInfluentialFLAME: Free-Form Language-Based Motion Synthesis & Editing
Kim, J.-H., Kim, J., & Choi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.25996 · Cited by 203
Zou, Z., Yu, Z., Guo, Y., Li, Y., Ding, L., Cao, Y., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00983 · Cited by 234
“Recently, trans-former architecture [61] has shown its capacity and scalability in the field of computer vision [4, 12, 23, 52, 55] and other areas [3, 11].”
Oniani, D., Hilsman, J., Peng, Y., Poropatich, R. K., Pamplin, J. C., Legault, G. L., & Wang, Y. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00965-x · Cited by 143
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., & Pantic, M. (2023) · arXiv · doi:10.48550/arxiv.2301.03396 · Cited by 140
Translation between Molecules and Natural Language
Edwards, C., Lai, T., Ros, K., Honke, G., Cho, K., & Ji, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.26 · Cited by 341
Post-Training Quantization on Diffusion Models
Shang, Y., Yuan, Z., Xie, B., Wu, B., & Yan, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00196 · Cited by 193
“For example, there is no access to the training data for the industry-developed text-to-image models Dall·E2 [26] and Imagen [27].”
Q-Diffusion: Quantizing Diffusion Models
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D. D., Zhang, S., & Keutzer, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01608 · Cited by 215
“Diffusion models have shown great success in generating images with both high diversity and high fidelity [43, 13, 44, 42, 6, 33, 36, 34].”
BackgroundMethodPloennigs, J., & Berger, M. (2023) · AI in Civil Engineering · doi:10.1007/s43503-023-00018-y · Cited by 153
RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
Anciukevicius, T., Xu, Z., Fisher, M., Henderson, P., Bilen, H., Mitra, N. J., & Guerrero, P. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01213 · Cited by 164
“Many exciting applications have emerged in only the last few months, including text-to-image generation [49, 55], inpainting [54], object insertion [3], and personalization [53].”
BackgroundBridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 247
“By arguing that for LLM interfaces, “intentions are actions,” we provide de-sign recommendations to support the process of envisioning with generative AI models.”
BackgroundWavelet Diffusion Models are fast and scalable Image Generators
Phung, H., Dao, Q. T., & Tran, A. N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00983 · Cited by 163
“Recent diffusion-based text-to-image generative models [1, 34, 38] allow users to generate unbelievably realistic synthesized images just by text inputs, opening a new era of AI-based digital art and promising to various other domains.”
BackgroundDataset Distillation: A Comprehensive Review
Yu, R., Liu, S., & Wang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3323376 · Cited by 200
“The groundbreaking deep models, such as AlexNet [1] in 2012, ResNet [4] in 2016, Bert [2] in 2018, alongside the more recent ViT [5], CLIP [6], and DALLE [7], all rely on large-scale datasets for training.”
MethodDiffusion models in bioinformatics and computational biology
Guo, Z., Liu, J., Wang, Y., Chen, M., Wang, D., Xu, D., & Cheng, J. (2023) · Nature Reviews Bioengineering · doi:10.1038/s44222-023-00114-9 · Cited by 193
“In addition, diffusion models can be applied for computer vision 43,51,56–71 , natural language processing 55,72–75 , temporal data modelling 76–81 , multi-modal modelling 36,37,82,83 , and in medical image reconstruction 84–93 .”
Self-citationA review on Generative Adversarial Networks for image generation
de Souza, V. L. T., Marques, B. A. D., Batagelo, H. C., & Gois, J. P. (2023) · Computers & Graphics · doi:10.1016/j.cag.2023.05.010 · Cited by 152
Ablating Concepts in Text-to-Image Diffusion Models
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., & Zhu, J.-Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02074 · Cited by 207
“Large-scale text-to-image models have demonstrated remarkable ability in synthesizing photorealistic images [51, 43, 56, 54, 76, 14].”
BackgroundFreeU: Free Lunch in Diffusion U-Net
Si, C., Huang, Z., Jiang, Y., & Liu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00453 · Cited by 202
“Diffusion probabilistic models, a cutting-edge category of generative models, have garnered significant attention, particularly for tasks related to computer vision [7, 8, 11, 18, 33, 41, 45, 46, 49].”
BackgroundSINE: SINgle Image Editing with Text-to-Image Diffusion Models
Zhang, Z., Han, L., Ghosh, A., Metaxas, D., & Ren, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00584 · Cited by 177
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
Shi, J., Xiong, W., Lin, Z., & Jung, H. J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00816 · Cited by 303
“We encode X into a global concept embedding c g via Concept Encoder E g and obtain”
MethodTEACH: Temporal Action Composition for 3D Humans
Athanasiou, N., Petrovich, M., Black, M. J., & Varol, G. (2022) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv57658.2022.00053 · Cited by 168
“Generative models are popular for synthesizing images conditioned on textual descriptions [34, 37].”
BackgroundIntriguing properties of synthetic images: from generative adversarial networks to diffusion models
Corvi, R., Cozzolino, D., Poggi, G., Nagano, K., & Verdoliva, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00104 · Cited by 142
“Especially the recent text-image generators, which rely on text descriptions or preliminary sketches as a guide, can allow image creation across countless categories [2, 9, 23, 35, 40, 44, 51].”
InfluentialMultimodal Image Synthesis and Editing: The Generative AI Era
Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., & Xing, E. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3305243 · Cited by 197
“DALL-E 2 [25] shows that diffusion models are slightly better than autoregressive models in modeling the diffusion prior.”
BackgroundMethodInfluentialPhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Li, Z., Cao, M., Wang, X., Qi, Z., Cheng, M., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00825 · Cited by 233
“Diffusion models [23, 64] have made remarkable progress in text-conditioned image and video generation [4, 32, 52, 54, 58, 69, 71], attracting widespread attention in recent years.”
Prompttts: Controllable Text-To-Speech With Text Descriptions
Guo, Z., Leng, Y., Wu, Y., Zhao, S., & Tan, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096285 · Cited by 170
“Recent research has achieved great success in text and image generation guided with a text description as prompt [1, 2, 3, 4] (e.g., GPT-3 [5] or DALLE-2 [6]).”
BackgroundTryOnDiffusion: A Tale of Two UNets
Zhu, L., Yang, D., Zhu, T., Reda, F. A., Chan, W., Saharia, C., Norouzi, M., & Kemelmacher-Shlizerman, I. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00447 · Cited by 156
“They have achieved state-of-the-art results on various image generation tasks, such as super-resolution [36], colorization [34], novel-view synthesis [42] and text-to-image generation [28,31,33,35].”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Hu, Y., Liu, B., Kasai, J., Wang, Y., Mari Ostendorf, E., Krishna, R., & Smith, N. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01866 · Cited by 267
“While we welcome artistic freedom when we commission art from artists, images produced by deep generative models [44, 46, 43, 47, 61] should conform closely to our requests.”
BackgroundDiffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., & Pantić, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00502 · Cited by 184
“As the final test, we generated a video of a talking avatar, using an image synthesized by DALL-E 2 [31] and our own recorded speech.”
BackgroundMethodDream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
Xu, J., Wang, X., Cheng, W., Cao, Y., Shan, Y., Qie, X., & Gao, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02003 · Cited by 168
DiffRF: Rendering-Guided 3D Radiance Field Diffusion
Muller, N. E., Siddiqui, Y., Porzi, L., Bulò, S. R., Kontschieder, P., & Nießner, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00421 · Cited by 157
“Interestingly, diffusion models have shown to outperform Generative Adversarial Networks (GANs) in high-resolution image generation tasks [16, 48], achieving unprecedented results in conditional image generation [47].”
BackgroundCLIP the Gap: A Single Domain Generalization Approach for Object Detection
Vidit, V., Engilberge, M., & Salzmann, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00314 · Cited by 159
PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
Feng, Y., Wang, X., Wong, K. K., Wang, S., Lu, Y., Zhu, M., Wang, B., & Chen, W. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327168 · Cited by 124
“Image conditioning methods [31, 40, 41] support text-guided inpainting to manipulate specific areas within images.”
BackgroundInfluentialTask and Motion Planning with Large Language Models for Object Rearrangement
Ding, Y. F., Zhang, X., Paxton, C., & Zhang, S. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342169 · Cited by 196
“Their approach, called DALL-E-Bot, enables a robot to generate images based on a text description using DALL-E [24], and accordingly arrange objects in a tabletop scenario.”
MethodOn the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D. R., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. (2024) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3653070 · Cited by 384
“Although street-view images are like the natural images used in common vision-language tasks, one major difference is that common vision-language tasks usually focus on factual knowledge in images (e.g., “ how many cars in this image ”) whereas urban perception tasks are usually related to…”
BackgroundSchöbel, S., Schmitt, A., Benner, D., Saqr, M., Janson, A., & Leimeister, J. M. (2024) · Information Systems Frontiers · doi:10.1007/s10796-023-10375-9 · Cited by 127
“This development is especially fueled through the development of “text-to-output” models such Open AI’s DALL · E 2 (Ramesh et al., 2022) or other latent diffusion models e.g., Rombach et al. (2021).”
MethodResDiff: Combining CNN and Diffusion Model for Image Super-resolution
Shang, S., Shan, Z., Liu, G., Wang, L., Wang, X., Zhang, Z., & Zhang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28746 · Cited by 167
Design implications of generative AI systems for visual storytelling for young learners
Han, A., & Cai, Z. (2023) · Proceedings of the 22nd Annual ACM Interaction Design and Children Conference · doi:10.1145/3585088.3593867 · Cited by 127
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132
A Review of Modern Fashion Recommender Systems
Deldjoo, Y., Nazary, F., Arnau Ramisa, A., McAuley, J., Pellegrini, G. B., Bellogín, A., & Noia, T. D. (2023) · ACM Computing Surveys · doi:10.1145/3624733 · Cited by 158
“Similarly, cross-modal text-to-image generative models, such as DALL-E [133] and Stable Difusion [139], could be incorporated into fashion AI systems to improve the accuracy of recommendations.”
MethodGenerative AI for visualization: State of the art and future directions
Ye, Y., Hao, J., Hou, Y., Wang, Z., Xiao, S., Luo, Y., & Zeng, W. (2024) · Visual Informatics · doi:10.1016/j.visinf.2024.04.003 · Cited by 167
EDICT: Exact Diffusion Inversion via Coupled Transformations
Wallace, B., Gokul, A., & Naik, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02158 · Cited by 179
From sequence to function through structure: Deep learning for protein design
(2022) · doi:10.1016/j.csbj.2022.11.014 · Cited by 124
“Diffusion [109] leveraged diffusion models [110–112] popularized by generative image approaches [113] to train a model capable of generating sequence and structure based on a set of secondary structure input constraints.”
BackgroundVideo-P2P: Video Editing with Cross-Attention Control
Liu, S., Zhang, Y., Li, W., Lin, Z., & Jia, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00821 · Cited by 257
“DALLE-2 [33] leverages CLIP [31] for better text-image alignment.”
Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine
Zhang, K., Meng, X., Yan, X., Ji, J., Liu, J., Xu, H., Zhang, H., Liu, D., Wang, J., Wang, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/59069 · Cited by 148
“The Transformer model, introduced by Google in 2016, underpins this progress, predicting words in sentences based on statistical correlations [32,33].”
NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors
Deng, C., Jiang, C. M., Qi, C. R., Yan, X., Zhou, Y., Guibas, L. J., & Anguelov, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01977 · Cited by 144
NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
Heo, G., Lee, S., Cho, J., Choi, H., Lee, S., Ham, H., Kim, G., Mahajan, D., & Park, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651380 · Cited by 162
“Large Language Models (LLMs) are being widely deployed across various sectors such as natural language understanding [9, 12, 27, 67, 80, 85, 86], content generation [13, 57, 73, 74], and decision support [51].”
BackgroundRePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120
“[19, 51, 61, 67]) enable laypersons to quickly create visual artworks”
BackgroundMethodInfluentialVideo Probabilistic Diffusion Models in Projected Latent Space
Yu, S., Sohn, K., Kim, S., & Shin, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01770 · Cited by 144
Brain-inspired learning in artificial neural networks: A review
Schmidgall, S., Ziaei, R., Achterberg, J., Kirsch, L., Hajiseyedrazi, S. P., & Eshraghian, J. K. (2024) · APL Machine Learning · doi:10.1063/5.0186054 · Cited by 169
ReCo: Region-Controlled Text-to-Image Generation
Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01369 · Cited by 142
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models
Ruiz, N., Li, Y., Jampani, V., Wei, W., Hou, T., Pritch, Y., Wadhwa, N., Rubinstein, M., & Aberman, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00624 · Cited by 155
“Text-to-Image Models Several recent models such as Im-agen [26], DALL-E2 [22], Stable Diffusion (SD) [24], Muse [7], Parti [33], etc., demonstrate excellent image generation capabilities given a text prompt.”
MethodDiffusion Model-Based Image Editing: A Survey
Huang, Y., Huang, J., Liu, Y., Yan, M., Lv, J., Liu, J., Xiong, W., Zhang, H., Cao, L., & Chen, S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3541625 · Cited by 278
“MagicBrush [275] hires crowd workers from Amazon Mechanical Turk (AMT) to manually perform continuous edits using DALL-E 2 [29].”
InfluentialConditional Image-to-Video Generation with Latent Flow Diffusion Models
Ni, H., Shi, C., Li, K., Huang, X., & Min, M. R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01769 · Cited by 148
An Edit Friendly DDPM Noise Space: Inversion and Manipulations
Huberman-Spiegelglas, I., Куликов, В. А., & Michaeli, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01185 · Cited by 203
“Diffusion models have emerged as a powerful generative framework, achieving state-of-the-art quality on image synthesis [8, 2, 21, 18, 23, 20].”
BackgroundDiffusion Models, Image Super-Resolution, and Everything: A Survey
Moser, B. B., Shanbhag, A. S., Raue, F., Frolov, S., Palacio, S., & Dengel, A. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3476671 · Cited by 117
“Applications like Dall-E and stable diffusion demonstrate that DMs have surpassed GANs in various aspects [12], [13], [14].”
Muse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157
“Compared to Imagen (Saharia et al., 2022) or Dall-E2 (Ramesh et al., 2022) which are built on cascaded pixel-space diffusion models, Muse is significantly more efficient due to the use of a discrete latent space.Compared to Parti (Yu et al., 2022b), a state-of-the-art autoregressive model, Muse is more efficient due to t…”
BackgroundMethodInfluentialWhat the DAAM: Interpreting Stable Diffusion Using Cross Attention
Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang, G., Kumar, K., Stenetorp, P., Lin, J., & Türe, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.310 · Cited by 226
Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
Zhu, L., Liu, X., Liu, X., Qian, R., Liu, Z., & Yu, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01016 · Cited by 152
“Besides, they mainly utilize explicit guidance like pretrained classifiers [13] and CLIP similarity [28, 33, 35, 36] to guide the generation process.”
BackgroundMethodMAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
Li, T., Chang, H., Mishra, S. K., Zhang, H., Katabi, D., & Krishnan, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00213 · Cited by 156
“Another example is DALLE-2 [47], where latents conditioned on a pre-trained CLIP representation are used to create high-quality text-to-image generations.”
BackgroundFreeDoM: Training-Free Energy-Guided Conditional Diffusion Model
Yu, J., Wang, Y., Zhao, C., Ghanem, B., & Zhang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02118 · Cited by 183
“Conditional diffusion models (CDMs) with diverse conditions have been proposed, such as text [20, 1, 13, 23, 33, 37, 32, 35, 22, 29], class labels [10], degraded images [5, 6, 7, 8, 19, 24, 39, 41, 44, 34, 36, 45], segmentation maps [28, 49], landmarks [28, 49], hand-drawn sketches [28, 49], style…”
BackgroundCLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 248
“…has learned extensive knowledge from text supervision and proven useful in a wide variety of areas, including image generation (Vinker et al., 2022; Ramesh et al., 2022; Yu et al., 2022; Rombach et al., 2022), segmentation (Li et 2022a; Rao et al., 2022; Luo et al., 2023), detection (Bangalath et…”
Typology of Risks of Generative Text-to-Image Models
Bird, C., Ungless, E., & Kasirzadeh, A. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604722 · Cited by 121
“Our systematic examination yields a typology of risks associated with state-of-the-art TTI models, such as DALL-E 2 [129].”
BackgroundMotion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models
Carvalho, J., Le, A. T., Baierl, M., Koert, D., & Peters, J. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342382 · Cited by 163
“At task planning levels, [34] recently proposes DALL-E-B OT , which generates a text description from a scene image and then prompts the text-to-image generator DALL-E [28] for a “goal scene”, thus identifying the desired poses.”
BackgroundResultRawte, V., Chakraborty, S., Pathak, A., Sarkar, A., Tonmoy, S. M. T. I., Chadha, A., Sheth, A., & Das, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.155 · Cited by 119
Dibia, V. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.11 · Cited by 170
PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
Wang, Z., Huang, Y., Song, D., Ma, L., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642803 · Cited by 115
Metadata
- Authors
- Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M.
- ORCID
- Chen, Mark
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.06125
- OpenAlex
- W4224035735
- Cited by
- 2,511
- References
- 66
- Citations by year
- 2026: 46 · 2025: 429 · 2024: 1113 · 2023: 859 · 2022: 63
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- text-to-image generation, diffusion models, image priors, image variation, photorealism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
