The DOI matches its DataCite record; 69 of the 69 references it cites were found in a registry or an index.
Hierarchical Text-Conditional Image Generation with CLIP Latents
Open accessAbstract
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While more sophisticated methods for partially editing individual frames during training exist (e.g., text-to-image models [13], [14]), they are slow, computationally expensive and error-prone.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredIn the citing paper
“Generative machines, such as text-to-image algorithms, are contributing to the variety of cultural traits.”
From Machine culture (2023) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“To this end, we generated 512 “artistic” captions using GPT-3 [4] by prompting it with captions for existing artwork (both real and AI generated).”
MethodSelf-citationHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 2×
“[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
BackgroundMethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
BackgroundMethodSelf-citationLIII. On lines and planes of closest fit to systems of points in space
Pearson, K. (1901) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440109462720 · Cited by 12,864 · cited 4×
“After applying PCA, we order the principal components by decreasing eigenvalue magnitude, quantize each of the 319 dimensions into 1,024 discrete buckets, and predict the resulting sequence using a decoder-only Transformer [53] model with a causal attention mask.”
MethodInfluentialDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Research works printed but not linked: 2
- [23][23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
- [32][32] Ryan Murdock. The Big Sleep. https://twitter.com/advadnoun/status/ 1351038053033406468, 2021. 20 https://openreview.net/forum?id=qw674L9PfQE https://openreview.net/forum?id=qw674L9PfQE https://arxiv.org/abs/2203.13131 https://arxiv.org/abs/2108.00946 https://arxiv.org/abs/2102.01645 https://arxiv.org/abs/1406.2661 https://arxiv.org/abs/2111.14822 https://openreview.net/forum?id=qw8AKxfYbI https://openreview.net/forum?id=qw8AKxfYbI https://arxiv.org/abs/2006.11239 https://arxiv.org/abs/2106.15282 https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1405.0312 https://arxiv.org/abs/1711.05101 https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://github.com/openai/dalle-2-preview/blob/main/system-card.md https://arxiv.org/abs/2112.12750 https://twitter.com/advadnoun/status/1351038053033406468 https://twitter.com/advadnoun/status/1351038053033406468 · arXiv
Cited by
Sorted by citations
Items 201–300 of 1,051
Foundation models in smart agriculture: Basics, opportunities, and challenges
Li, J., Xu, M., Xiang, L., Chen, D., Zhuang, W., Yin, X., & Li, Z. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109032 · Cited by 159
Social Biases through the Text-to-Image Generation Lens
Naik, R., & Nushi, B. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604711 · Cited by 186
“Several models have been proposed, with impressive photorealism properties: DALLEv2 [22], Stable Diffusion [23], and Imagen [24].”
BackgroundIndustrial Metaverse for Smart Manufacturing: Model, Architecture, and Applications
Ren, L., Dong, J., Li, Z., Laili, Y., Wang, X., Qi, Y., Li, B. H., Wang, L., Yang, L. T., & Deen, M. J. (2024) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2024.3372591 · Cited by 114
“CLaSP, as the discriminator to judge whether generative scenes are feasible, can be utilized in a generative adversarial paradigm [61], like DALL · E [62] or 3-”
BackgroundDense Text-to-Image Generation with Attention Modulation
Kim, Y., Lee, J., Kim, J.-H., Ha, J.-W., & Zhu, J.-Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00708 · Cited by 129
“Recently developed text-to-image models can synthe-size diverse, high-quality images from short scene descriptions [24, 15, 38, 44, 46, 45, 29, 58, 7].”
BackgroundDreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2208 · Cited by 108
Chang, M., Druga, S., Fiannaca, A. J., Vergani, P., Kulkarni, C. E., Cai, C. J., & Terry, M. A. (2023) · Creativity and Cognition · doi:10.1145/3591196.3593515 · Cited by 136
“The most recent of these models include: DALL·E [48, 53] and DALL·E 2 [47, 52], Stable Diffusion [5, 54], Midjourney[3], Parti [4, 60], and Imagen [2, 55].”
MethodDiffusion-SDF: Conditional Generative Modeling of Signed Distance Functions
Chou, G., Bahat, Y., & Heide, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00215 · Cited by 136
“Recent advances [38, 10, 44, 46] show that diffusion models produce images with quality on par with state-of-the-art generative adversarial networks (GANs) [15] without the common drawbacks of mode collapse [38, 37] and unstable training [36, 2].”
BackgroundMethodInfluentialFoundation models for fast, label-free detection of glioma infiltration
Kondepudi, A., Pekmezci, M., Hou, X., Scotford, K., Jiang, C., Rao, A. T., Harake, E. S., Chowdury, A., Al‐Holou, W. N., Wang, L., et al. (2024) · Nature · doi:10.1038/s41586-024-08169-3 · Cited by 128
Machine Learning Modeling for Accelerated Battery Materials Design in the Small Data Regime
Sendek, A. D., Ransom, B., Cubuk, E. D., Pellouchoud, L. A., Nanda, J., & Reed, E. J. (2022) · Advanced Energy Materials · doi:10.1002/aenm.202200553 · Cited by 100
Shabani, M. A., Hosseini, S., & Furukawa, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00529 · Cited by 124
Token Merging for Fast Stable Diffusion
Bolya, D., & Hoffman, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00484 · Cited by 115
“With the rise of powerful diffusion [4, 21] models such as DALL-E 2 [13], Imagen [18], and Stable Diffusion [15], generating high quality images has never been easier.”
MethodFace Generation and Editing With StyleGAN: A Survey
Melnik, A., Miasayedzenkau, M., Makaravets, D., Pirshtuk, D., Akbulut, E., Holzmann, D., Renusch, T., Reichert, G., & Ritter, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3350004 · Cited by 105
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00779 · Cited by 361
“Recently, benefiting from advancements in text-to-image research[2, 19, 29, 32, 34, 36], video generation (e.g., text-to-video, video editing)[4, 10, 12, 15–17, 23, 30, 40, 48, 52] has also achieved notable progress in terms of visual quality and diversity.”
BackgroundDiffSwap: High-Fidelity and Controllable Face Swapping via 3D-Aware Masked Diffusion
Zhao, W., Rao, Y., Shi, W., Liu, Z., Zhou, J., & Lu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00828 · Cited by 126
“Very recently, diffusion-based models (DM) [22, 25, 26] have exhibited high customizability for various conditions and impressive power in generating images with high resolution and complex scenes.”
BackgroundSelf-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101
“To provide a straightforward comparison with the time delay distribution model, we reduce the dimensionality of the features by projecting them onto the top 2τ + 1 principal components, following other work in autoregressive models of features [85].”
MethodA Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101
Designing and Managing Human-AI Interactions
Abedin, B., Meske, C., Junglas, I. A., Rabhi, F., & Motahari‐Nezhad, H. R. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10313-1 · Cited by 128
“, 2020), floorplan design (Khang, 2021), job recruitment (Dastin, 2018), art (Ramesh et al., 2022), credit scoring (Wang et al.”
BackgroundAvatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model
Du, Y., Kips, R., Pumarola, A., Starke, S., Thabet, A., & Sanakoyeu, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00054 · Cited by 126
“Moreover, we follow [42] to directly predict the clean body poses x 0 instead of predicting the residual noise εθ(xt, t).”
DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models
Cao, Y., Cao, Y., Han, K., Shan, Y., & Wong, K.-Y. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00097 · Cited by 179
“With the improvements in neural networks and text-image datasets, the diffusion model has been introduced to handle more complex semantic concepts [2, 22, 34, 38].”
MethodDiffEdit: Diffusion-based semantic image editing with mask guidancePreprint
Couairon, G., Verbeek, J. J., Schwenk, H., & Cord, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.11427 · Cited by 542
“, 2022), DALL-E 2 (Ramesh et al., 2022) and Imagen (Saharia et al.”
BackgroundGenerative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
Quantum machine learning: from physics to software engineering
Melnikov, A. A., Kordzanganeh, M., Alodjants, A. P., & Lee, R. (2023) · Advances in Physics X · doi:10.1080/23746149.2023.2165452 · Cited by 125
“For smaller problems, the parameter-count could be as small as hundreds [76,77], but cutting-edge AI models such as DALL-E 2 [78], Gopher [79], and GPT-3 [80] are models that have tens or hundreds of billions of parameters and is increasing [81,82].”
MethodConditional Text Image Generation with Diffusion Models
Zhu, Y., Li, Z., Wang, T., He, M., & Yao, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01368 · Cited by 109
“UnCLIP [54] further pre-trains a CLIP model [53] to match the image and whole text, which are used as the conditions for the diffusion models in image generation.”
Opal: Multimodal Image Generation for News Illustration
Liu, V., Qiao, H., & Chilton, L. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545621 · Cited by 104
“[15, 19, 36, 37] DALL·E 2 [42] showed by conditioning on a CLIP text prior, generating CLIP image representations, and decoding those representations either autoregressively or with difusion, DALL·E 2 could create a text-to-image generator with novel functionalities like image remixing, inpainting, and interpolation.”
MethodInterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion
Xu, S., Li, Z., Wang, Y.-X., & Gui, L.-Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01371 · Cited by 126
“Following [69,70,85], we directly recover the clean signal ˜ x at each step, instead of predicting the noise ϵ [29] that was added to x 0 in Eq.”
MethodPocketFlow is a data-and-knowledge-driven structure-based molecular generative model
Jiang, Y., Zhang, G., You, J., Zhang, H., Yao, R., Xie, H., Zhang, L., Xia, Z., Dai, M., Wu, Y., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00808-8 · Cited by 97
“Deep generative models (DGMs), which have achieved great success in producing images, texts and voices 4 , offer an efficient approach to generating completely new seed compounds 5–9 .”
MixGen: A New Multi-Modal Data Augmentation
Hao, X., Zhu, Y., Appalaraju, S., Zhang, A., Zhang, W., Li, B., & Li, M. (2023) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw58289.2023.00042 · Cited by 110
“There are also work in video-text retrieval [32], text-to-image generation [39], language guided object detection [25] or segmentation [57].”
BackgroundDeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality
Handa, A., Allshire, A., Makoviychuk, V., Petrenko, A., Singh, R., Liu, J., Makoviichuk, D., Wyk, K. V., Zhurkevich, A., Sundaralingam, B., et al. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160216 · Cited by 148
“While the NLP and computer vision communities have reproduced and extended the successes of large-scale models like GPT-3 [3] and DALL-E [4, 5] respectively, similar efforts have remained elusive in robotics due to hardware and infrastructure challenges.”
BackgroundPromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
Chung, J. J. Y., & Adar, E. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606777 · Cited by 131
“Prompt mixing interpolates different prompt vectors with weights [66] from the user’s Prompt Palette (with proximity to each prompt): 𝑝 𝑚 , 𝑤 𝑝 𝑖 , and 𝑣 𝑝 𝑖 represent the interpolated vector, the weight of each prompt, and the embedded vector of each prompt, respectively.”
BackgroundMethodInfluentialDiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis
Tang, J., Nie, Y., Markhasin, L., Dai, A., Thies, J., & Nießner, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01938 · Cited by 182
“The performances degrade when we use the transformer in DALLE-2 [48].”
MethodTF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition
Lu, S., Liu, Y., & Kong, A. W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00218 · Cited by 119
“While recently large-scale text-to-image models [10, 19, 51, 56, 59, 61, 78] have achieved remarkable success in text-driven image generation, the ambiguity inherent in natural language presents challenges in conveying precise and nuanced visual details, even with highly detailed text prompts.”
BackgroundEditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protection
Zhang, X., Li, R., Yu, J., Xu, Y., Li, W., & Zhang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01137 · Cited by 133
“Moreover, they inevitably need to introduce tampered data during the training and focus solely on specific “CheapFake” tampering like slicing and copy-and-paste [12, 53, 68], or on “DeepFake” targeting human faces [3, 62], restricted in generalizability.”
BackgroundRoSteALS: Robust Steganography using Autoencoder Latent Space
Bui, T., Agarwal, S., Yu, N., & Collomosse, J. P. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00100 · Cited by 103
“Deep fakes and misinformation are major societal challenges exacerbated by recent advances in generative models [31–33].”
BackgroundDiffusion Probabilistic Model Made Slim
Yang, X., Zhou, D., Feng, J., & Wang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02160 · Cited by 118
Synthbuster: Towards Detection of Diffusion Model Generated Images
Bammey, Q. (2023) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2023.3337714 · Cited by 125
“Thanks to the advent of diffusion models [40], [48], [49], [50] and text-to-image joint embeddings, it is now possible and easy to generate images from scratch with nothing more than a text prompt describing the intended image, as seen in Fig.”
BackgroundMethodProteinSGM: Score-based generative modeling for de novo protein design
Lee, J. S., Kim, J., & Kim, P. M. (2022) · doi:10.1101/2022.07.13.499967 · Cited by 94
Vlassis, N. N., & Sun, W. (2023) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2023.116126 · Cited by 100
“Several related research works have shown that DDPMs can generate highly realistic images (Song et al., 2020; Ramesh et al., 2022; Rombach et al., 2022), audio (Chen et al.”
BackgroundMethodInfluentialDiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
Yang, S., Wu, Z., Li, M., Zhang, Z., Lei, H., Bao, W., Cheng, M., & Xiao, L. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/650 · Cited by 150
PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction
Melas-Kyriazi, L., Rupprecht, C., & Vedaldi, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01242 · Cited by 112
Feng, Z., Zhang, Z., Yu, X., Fang, Y., Li, L., Chen, X., Lu, Y., Liu, J., Yin, W., Feng, S., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00977 · Cited by 124
Alpha-CLIP: A CLIP Model Focusing on Wherever you Want
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01237 · Cited by 151
“Alpha-CLIP can be used in most image variation models that use CLIP image encoder [24, 39, 53, 55, 63].”
InfluentialDALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
Kapelyukh, I., Vosylius, V., & Johns, E. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3272516 · Cited by 114
“At the heart of our method is a web-scale image diffusion model DALL-E 2 [9], which, given a text description of the objects in a scene, can generate a goal image IG, depicting a human-like arrangement of those objects.”
BackgroundMethodInfluentialLocalizing Object-level Shape Variations with Text-to-Image Diffusion Models
Patashnik, O., Garibi, D., Azuri, I., Averbuch‐Elor, H., & Cohen‐Or, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02107 · Cited by 122
“Text-to-image diffusion models have recently shown unprecedented image quality and diversity [35, 41, 39], and have opened a new era in image synthesis.”
BackgroundA generative model of memory construction and consolidation
Spens, E., & Burgess, N. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01799-z · Cited by 122
“The increasingly sophisticated generation of images from text using modern generative models (Ramesh et al., 2022) suggests that episode construction from semantic facts is consistent with this framework.”
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., & Erhan, D. (2022) · arXiv · doi:10.48550/arxiv.2210.02399 · Cited by 92
“It is now possible to generate realistic high resolution images given a description [34, 35, 32, 38, 59], but generating high quality videos from text remains challenging.”
BackgroundTaming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow
Gou, J., Sun, S., Zhang, J., Si, J., Qian, C., & Zhang, L. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612255 · Cited by 122
Diffusion-SDF: Text-to-Shape via Voxelized Diffusion
Li, M., Duan, Y., Zhou, J., & Lu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01216 · Cited by 114
“Especially, fairly impressive results have been achieved with diffusion model-based approaches on the task of textto-image synthesis [41, 45, 47].”
BackgroundDiffCR: A Fast Conditional Diffusion Framework for Cloud Removal From Optical Satellite Images
Zou, X., Li, K., Xing, J., Zhang, Y., Wang, S., Jin, L., & Tao, P. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3365806 · Cited by 109
“It is worth noting that DALL·E 2 [47] developed by OpenAI also reports a similar finding in their work.”
BackgroundGaussianEditor: Editing 3D Gaussians Delicately with Text Instructions
Wang, J., Fang, J., Zhang, X., Xie, L., & Tian, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01975 · Cited by 151
“Some of these methodologies necessitate the manual provision of captions for both the original and edited images [12], while others mandate specific scenario-based training for optimization [39].”
MethodTransformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 170
A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond
Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., & Liu, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3277122 · Cited by 117
“Early research mainly explores their effectiveness on continuous data, such as images and audio generation [205], [206], [207], [208], [209], [210], [211], [212].”
BackgroundText-to-Audio Generation using Instruction Guided Latent Diffusion Model
Ghosal, D., Majumder, N., Mehrish, A., & Poria, S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612348 · Cited by 242
TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models
Cao, T., Kreis, K., Fidler, S., Sharp, N., & Yin, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00385 · Cited by 104
“In particular, they have demonstrated outstanding performance in image generation [14–16, 21, 56, 81], outperforming GANs, and led to breakthroughs in text-to-image synthesis [2, 54, 61, 63, 66].”
BackgroundGIT-Mol: A multi-modal large language model for molecular science with graph, image, and text
Liu, P., Ren, Y., Tao, J., & Ren, Z. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108073 · Cited by 114
“In addition, text-image generation models like DALLE-2 [34] and UniDiffuser [35] employ diffusion models [36] to generate images from text.”
MethodSynthetic data in biomedicine via generative artificial intelligence
van Breugel, B., Liu, T., Oglić, D., & van der Schaar, M. (2024) · Nature Reviews Bioengineering · doi:10.1038/s44222-024-00245-7 · Cited by 111
“Simulators are computational models that synthesize observations of a system in response to different scenarios.”
Towards the next generation of Geospatial Artificial Intelligence
Mai, G., Xie, Y., Jia, X., Lao, N., Rao, J., Zhu, Q., Liu, Z., Chiang, Y., & Jiao, J. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104368 · Cited by 145
A survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90
Illuminating protein space with a programmable generative model
Ingraham, J., Baranov, M., Costello, Z., Frappier, V., Ismail, A., Tie, S., Wang, W., Xue, V., Obermeyer, F., Beam, A., et al. (2022) · doi:10.1101/2022.12.01.518682 · Cited by 83
Generative adversarial networks and diffusion models in material discovery
Alverson, M., Baird, S. G., Murdock, R., Ho, S.-H. E., Johnson, J. A., & Sparks, T. D. (2023) · Digital Discovery · doi:10.1039/d3dd00137g · Cited by 97
“Conditional generation is the method that was responsible for all of the text-to-image break-throughs that underpin powerful tools such as DallE-2 [20] and Imagen [21].”
MethodResultLEDITS++: Limitless Image Editing Using Text-to-Image Models
Brack, M., Friedrich, F., Kornmeier, K., Tsaban, L., Schramowski, P., Kersting, K., & Passos, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00846 · Cited by 127
“Crucially, current text-to-image models [28, 32, 36] hold the potential to wield a profound influence on society.”
Background@ CREPE: Can Vision-Language Foundation Models Reason Compositionally?
Ma, Z., Hong, J., Gul, M. O., Gandhi, M., Gao, I., & Krishna, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01050 · Cited by 111
“[68] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.”
BackgroundMethodMultiscale Structure Guided Diffusion for Image Deblurring
Ren, M., Delbracio, M., Talebi, H., Gerig, G., & Milanfar, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00984 · Cited by 118
“Conditional DPMs aim to perform image synthesis with an additional input (class [14], text [66, 58], source image).”
BackgroundScore-based generative modeling for de novo protein design
Lee, J. S., Kim, J., & Kim, P. M. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00440-3 · Cited by 94
NeuralLift-360: Lifting an in-the-Wild 2D Photo to A 3D Object with 360° Views
Xu, D., Jiang, Y., Wang, P., Fan, Z., Wang, Y., & Wang, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00435 · Cited by 112
“Modern diffusion models [41,48,50] are trained on a massive dataset (e.”
BackgroundMethodInfluentialSuS-X: Training-Free Name-Only Transfer of Vision-Language Models
Udandarao, V., Gupta, A., & Albanie, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00257 · Cited by 120
“These representations can then be applied to tackle downstream tasks like semantic segmentation [67, 30], object detection [33, 23], image captioning [54, 3] and generative modelling [63, 62], In this work, we adapt such VLMs in a training-free setting to diverse downstream tasks.”
MethodHertz, A., Aberman, K., & Cohen‐Or, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00221 · Cited by 128
“Text-to-Image models [34, 30, 31], have recently raised the bar for the task of generating images conditioned on a text prompt, exploiting the powerful architecture of diffusion models [13, 36, 39, 13, 37, 31], which can be used to various image editing and guided synthesis tasks [32, 17, 45, 44].”
BackgroundVectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models
Jain, A., Xie, A., & Abbeel, P. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00190 · Cited by 121
“Recently, large captioned datasets and breakthroughs in diffusion models have led to systems capable of generating diverse images from text including DALL-E 2 [28], Imagen [33] and Latent Diffusion [31].”
BackgroundMethodYang, Z., Peng, D., Kong, Y., Zhang, Y., Yao, C., & Jin, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28482 · Cited by 109
InstanceDiffusion: Instance-Level Control for Image Generation
Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., & Misra, I. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00596 · Cited by 139
“Notably, text conditioned diffusion models now produce high quality images that contain the free form concepts specified in the text [12, 22, 44, 50, 53, 54].”
BackgroundMethodAI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
Qadri, R., Shelby, R., Bennett, C. L., & Denton, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594016 · Cited by 85
“Between the first and second focus group, the research team synthesized participant’s prompt suggestions, tested various prompts, and generated images using four state-of-the-art T2I models [88, 92, 97, 122].”
Sergeeva, O. V., Zheltukhina, M. R., Shoustikova, T., Tukhvatullina, L. R., Dobrokhotov, D. A., & Kondrashev, S. V. (2025) · Contemporary Educational Technology · doi:10.30935/cedtech/16039 · Cited by 131
DiffusionNER: Boundary Diffusion for Named Entity Recognition
Shen, Y., Song, K., Tan, X., Li, D., Lü, W., & Zhuang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.215 · Cited by 114
Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection
Flaborea, A., Collorone, L., di Melendugno, G. M. D., D’Arrigo, S., Prenkaj, B., & Galasso, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00947 · Cited by 97
“We are inspired by Denoising Diffusion Probabilistic Models (DDPMs) [40, 14], state-of-the-art, among others, in image synthesis [31, 30, 33], motion synthesis [43, 49], and 3D generation tasks [53].”
MethodInfluentialIDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models
Boutros, F., Grebe, J. H., Kuijper, A., & Damer, N. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01800 · Cited by 99
“Besides that, they can be conditioned on additional information, and demonstrated impressive results in a wide range of tasks, especially in text-to-image generation such as OpenAI’s DALL-E 2 [44], Stable Diffusion [45], and Google’s Imagen [47].”
BackgroundDoes ChatGPT have semantic understanding? A problem with the statistics-of-occurrence strategy
Titus, L. M. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101174 · Cited by 82
“For example, Marcus et al. (2022) discuss the failure of DALL-E 2 (also called unCLIP) to successfully produce one correct image out of ten tries in response to the prompt: ‘‘a red ball on top of a blue pyramid with the pyramid behind a car that is above a toaster’’ (see also Ramesh et al. (2022)).”
GNM: A General Navigation Model to Drive Any Robot
Shah, D., Sridhar, A., Bhorkar, A., Hirose, N., & Levine, S. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161227 · Cited by 85
“cessing [1], visual perception [2–4], and other domains [5, 6] by leveraging Internet-scale data.”
BackgroundData Augmentation for Object Detection via Controllable Diffusion Models
Fang, H., Han, B., Zhang, S., Zhou, S., Hu, C., & Ye, W.-M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00129 · Cited by 103
“Even more advanced data-augmentation methods are generative — they leverage the recent advances in generative models such as CLIP and stable diffusion models [15,32,36,41] to create synthetic training images.”
BackgroundHas generative artificial intelligence solved inverse materials design?
Park, H., Li, Z., & Walsh, A. (2024) · Matter · doi:10.1016/j.matt.2024.05.017 · Cited by 111
Emu Edit: Precise Image Editing via Recognition and Generation Tasks
Sheynin, S., Polyak, A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., & Taigman, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00847 · Cited by 180
“The emergence of high-performing text-to-image diffusion models [7, 16, 17, 19] facilitated the development of effective text-based image editing methods.”
BackgroundDang, H., Goller, S., Lehmann, F., & Buschek, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580969 · Cited by 83
Parallel Learning: Overview and Perspective for Computational Learning Across Syn2Real and Sim2Real
Miao, Q., Lv, Y., Huang, M., Wang, X., & Wang, F. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123375 · Cited by 83
“[18] and [19], focus on the construction of accurate 3-dimen-”
BackgroundTinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance
Wu, K., Peng, H., Zhou, Z., Xiao, B., Liu, M., Yuan, L., Xuan, H., Valenzuela, M., Chen, X. S., Wang, X., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02008 · Cited by 98
“Large-scale language-image pretraining, e.g. , CLIP [46], has recently gained significant attention due to its remarkable zero-shot transfer capability [46] and unprecedented performance in text-to-image generation [47].”
BackgroundCollaborative Diffusion: Boosting Designerly Co-Creation with Generative AI
Verheijden, M. P., & Funk, M. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585680 · Cited by 83
“With the introduction of DALL-E 2 [18, 23] and MidJourney [15], high-quality AI image generation became available to the public.”
BackgroundHow to Backdoor Diffusion Models?
Chou, S.-Y., Chen, P., & Ho, T.-Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00391 · Cited by 104
Disco: Disentangled Control for Realistic Human Dance Generation
Wang, T., Li, L., Lin, K., Zhai, Y., Lin, C.-C., Yang, Z., Zhang, H., Liu, Z., & Wang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00891 · Cited by 174
“To-wards user-specific generation, text prompts are first utilized as the condition for image generation [16, 44, 48, 69].”
BackgroundMethodLi, J., Li, B., Tu, Z., Liu, X., Guo, Q., Juefei-Xu, F., Xu, R., & Yu, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01440 · Cited by 97
“This insight inspires us to leverage VL model related techniques for enhancing low-light images [12, 25, 41].”
MethodThe Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
“This method can also transition to API or downloadable, as seen with OpenAI’s DALL·E 2 [77] was”
MethodWhat Does DALL-E 2 Know About Radiology?
Adams, L. C., Busch, F., Truhn, D., Makowski, M. R., Aerts, H. J. W. L., & Bressem, K. K. (2023) · Journal of Medical Internet Research · doi:10.2196/43110 · Cited by 82
“, “a painting of a pelvic x-ray by Claude Monet”)[Ramesh et al., 2022][Nichol et al.”
InfluentialDeep language models for interpretative and predictive materials science
Hu, Y., & Buehler, M. J. (2023) · APL Machine Learning · doi:10.1063/5.0134317 · Cited by 85
Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
Shan, S., Ding, W., Passananti, J., Wu, S. L., Zheng, H.-T., & Zhao, B. Y. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00207 · Cited by 166
“Today, latent diffusion is used in almost all state-of-the-art models [22, 23, 24, 25, 26].”
BackgroundMethodInfluentialUpscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution
Zhou, S., Yang, P., Wang, J., Luo, Y., & Loy, C. C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00245 · Cited by 108
“Diffusion models [24] have exhibited impressive proficiency in generating high-quality images [49, 50, 52, 85] and videos [6, 11, 17, 25, 87].”
BackgroundAnalyzing and Improving the Training Dynamics of Diffusion Models
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., & Laine, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02282 · Cited by 186
“(Figure 2a,b), and its variants have been widely adopted in large-scale diffusion models, including Imagen [58], Stable Diffusion [54], eDiff-I [2], DALL-E 2 [47, 53], and DALL-E 3 [4].”
Visual Prompt Tuning for Generative Transfer Learning
Sohn, K., Chang, H., Lezama, J., Polanía, L. F., Zhang, H., Hao, Y., Essa, I. A., & Jiang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01900 · Cited by 103
“NAR or diffusion models, including DALL·E 2 [52], MaskGIT [7], Latent Diffusion [57], or Imagen [58], decompose image synthesis as a series of refinement or denoising steps.”
MethodHumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation
Ju, X., Zeng, A., Zhao, C., Wang, J., Zhang, L., & Xu, Q. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01465 · Cited by 112
“Recently, diffusion models [12, 35, 40] have demonstrated unprecedented textto-image generation performance [32] and quickly become the dominant technique in this exciting field.”
BackgroundGenerative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 105
MACE: Mass Concept Erasure in Diffusion Models
Lu, S., Wang, Z., Li, L., Liu, Y., & Adams Wai Kin, K. O. N. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00615 · Cited by 130
“In large-scale text-to-image (T2I) models [9, 14, 34, 40, 49, 54, 57, 73, 74, 76, 77], the task of concept erasure aims to remove concepts that may be harmful, copyrighted, or offensive.”
BackgroundDEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations
Qi, T., Fang, S., Wu, Y., Xie, H., Liu, J., Chen, L., He, Q., & Zhang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00830 · Cited by 111
“Latest diffusion models [21, 22, 25] further achieve state-of-the-art performance on text-to-image generation, which benefits from large-scale pre-training.”
Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models
Qu, Y., Shen, X., He, X., Michael Backes, A., Zannettou, S., & Zhang, Y. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3616679 · Cited by 120
“Text-to-Image models [15, 30, 51, 55] are gaining unprecedented popularity for their out-of-the-box functionality and impressive ability to generate realistic images, such as drawings, illustrations, and photographs.”
BackgroundInfluential
Metadata
- Authors
- Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M.
- ORCID
- Chen, Mark
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.06125
- OpenAlex
- W4224035735
- Cited by
- 2,511
- References
- 66
- Citations by year
- 2026: 46 · 2025: 429 · 2024: 1113 · 2023: 859 · 2022: 63
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- text-to-image generation, diffusion models, image priors, image variation, photorealism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
