The DOI matches its DataCite record; 89 of the 89 references it cites were found in a registry or an index.
Muse: Text-To-Image Generation via Masked Generative Transformers
Open accessAbstract
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“For the base model, the decoder consists of a stack of Transformer layers which are finally mapped to patch space.”
MethodInfluentialU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 1×
“Furthermore, we believe that the use of token-based Trans-formers as a unifying architecture leads to better text-image alignment, compared to diffusion models where the image space is a convolution U-Net (Ronneberger et al., 2015).”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 4×
“Our 3B parameter model achieves a CLIP score of 0.32 and an FID score of 7.88 on the COCO (Lin et al., 2014) zero-shot validation benchmark, which compare favorably with that of other large-scale text-to-image models (see Table 2).”
MethodInfluentialSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 3×
“In Table 3, we also ablated multi-axis Trans-formers (Zhao et al., 2021) against SWIN (Liu et al., 2021), finding that the former performed better.”
MethodInfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“LLMs (including T5XXL, BERT (Devlin et al., 2018), and GPT (Brown et al., 2020; Radford et al., 2019)) have been shown to learn powerful embeddings, enabling few-shot transfer learning.”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 13×
“…Raffel et al., 2020) and vision tasks (He et al., 2022; Chang et al., 2022); new families of generative models such as diffusion (Ho et al., 2020; Rombach et al., 2022; Saharia et al., 2022) and masking-based generation (Chang et al., 2022); and finally, the availability of large scale image-text…”
BackgroundMethodInfluentialGenerative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 13×
“Our 3B parameter model achieves a CLIP score of 0.32 and an FID score of 7.88 on the COCO (Lin et al., 2014) zero-shot validation benchmark, which compare favorably with that of other large-scale text-to-image models (see Table 2).”
BackgroundMethodInfluential
Research works printed but not linked: 5
- Buolamwini, J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pp. 77-91. PMLR, 2018.
- Dulhanty, C. Issues in computer vision data collection: Bias, consent, and label taxonomy. Master's thesis, University of Waterloo, 2020.
- Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
- Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In ICLR , 2022.
Cited by
Sorted by citations
GLIGEN: Open-Set Grounded Text-to-Image Generation
Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., & Lee, Y. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02156 · Cited by 715
Self-citationScaling up GANs for Text-to-Image Synthesis
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00976 · Cited by 492
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation
Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., & Zuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01461 · Cited by 386
Adversarial Diffusion Distillation
Sauer, A., Lorenz, D., Blattmann, A., & Rombach, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73016-0_6 · Cited by 617
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Jayasumana, S., Ramalingam, S., Veit, A., Gläsner, D., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00889 · Cited by 320
“While the Muse model [3] generates multiple to-kens at a time, it still requires iterative sampling steps to generate the final image, as shown in Figure 3.”
MethodA Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182
“, 2022] • Muse reaches state-of-the-art text conditional image generation with a masked transformer approach [Chang et al., 2023].”
BackgroundSVDiff: Compact Parameter Space for Diffusion Fine-Tuning
Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00673 · Cited by 302
“Text-to-image diffusion models Diffusion models [58, 61, 19, 62, 40, 59, 18, 60, 10] have proven to be highly effective in learning data distributions and have shown impressive results in image synthesis, leading to various applications [74, 45, 9, 49, 26, 23, 34, 56, 28, 24, 29].”
BackgroundSelf-citationMoMask: Generative Masked Modeling of 3D Human Motions
Guo, C., Mu, Y., Javed, M. G., Wang, S., & Cheng, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00186 · Cited by 280
Unleashing Text-to-Image Diffusion Models for Visual Perception
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., & Lu, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00527 · Cited by 249
One-Step Diffusion with Distribution Matching Distillation
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00632 · Cited by 395
“Distribution Matching Recently, a few classes of generative models have shown success in scaling up to complex datasets by recovering samples that are corrupted by a pre-defined mechanism, such as noise injection [21, 61, 64] or token masking [5, 60, 86].”
BackgroundSelf-citationAblating Concepts in Text-to-Image Diffusion Models
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., & Zhu, J.-Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02074 · Cited by 207
“Large-scale text-to-image models have demonstrated remarkable ability in synthesizing photorealistic images [51, 43, 56, 54, 76, 14].”
BackgroundInstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
Shi, J., Xiong, W., Lin, Z., & Jung, H. J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00816 · Cited by 303
“We encode X into a global concept embedding c g via Concept Encoder E g and obtain”
MethodMultimodal Image Synthesis and Editing: The Generative AI Era
Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., & Xing, E. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3305243 · Cited by 197
“Text: CogView [69], DALL-E [70], NÜWA [44], Parti [71], Make-A-Scene [72], Muse [73]”
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models
Ruiz, N., Li, Y., Jampani, V., Wei, W., Hou, T., Pritch, Y., Wadhwa, N., Rubinstein, M., & Aberman, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00624 · Cited by 155
“Text-to-Image Models Several recent models such as Im-agen [26], DALL-E2 [22], Stable Diffusion (SD) [24], Muse [7], Parti [33], etc., demonstrate excellent image generation capabilities given a text prompt.”
MethodSelf-citationDreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2208 · Cited by 108
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 193
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition
Lu, S., Liu, Y., & Kong, A. W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00218 · Cited by 119
InstanceDiffusion: Instance-Level Control for Image Generation
Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., & Misra, I. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00596 · Cited by 139
“Image generation models [8, 9, 18, 22, 26, 27, 44, 46, 50, 53, 69] trained on web-scale data have made tremendous progress in the recent years.”
BackgroundTransformer-Based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey
Dubey, S. R., & Singh, S. K. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3404910 · Cited by 102
“Masked generative transformer (Muse) is investigated for text-to-image generation [155].”
BackgroundVisual Prompt Tuning for Generative Transfer Learning
Sohn, K., Chang, H., Lezama, J., Polanía, L. F., Zhang, H., Hao, Y., Essa, I. A., & Jiang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01900 · Cited by 103
Self-citationMACE: Mass Concept Erasure in Diffusion Models
Lu, S., Wang, Z., Li, L., Liu, Y., & Adams Wai Kin, K. O. N. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00615 · Cited by 130
“In large-scale text-to-image (T2I) models [9, 14, 34, 40, 49, 54, 57, 73, 74, 76, 77], the task of concept erasure aims to remove concepts that may be harmful, copyrighted, or offensive.”
BackgroundOnline Detection of AI-Generated Images
Epstein, D. C., Jain, I., Wang, O., & Zhang, R. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00045 · Cited by 91
“Methods such as Imagen [41], Parti [60], Muse [13], and GigaGAN [24] offer high-quality generations with alternative architectures but do not have public APIs or source code released.”
TextMesh: Generation of Realistic 3D Meshes From Text Prompts
Tsalicoglou, C., Manhardt, F., Tonioni, A., Niemeyer, M., & Tombari, F. (2024) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv62453.2024.00154 · Cited by 134
Scaling Robot Learning with Semantically Imagined Experience
Yu, T., Xiao, T. G., Tompson, J., Stone, A., Wang, S., Brohan, A., Singh, J., Tan, C., Dee, M., Peralta, J., et al. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.027 · Cited by 226
Photorealistic Video Generation with Diffusion Models
Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I. A., Jiang, L., & Lezama, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72986-7_23 · Cited by 253
Self-citationMIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis
Zhou, D., Li, Y., Ma, F., Zhang, X., & Yang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00651 · Cited by 113
“2) Inspired by the principle of divide and conquer, we introduce a novel MIGC approach that enhances pre-trained stable diffusion with improved MIG capabilities.”
MethodSelf-citationExpressive Text-to-Image Generation with Rich Text
Ge, S., Park, T., Zhu, J.-Y., & Huang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00694 · Cited by 87
“…text-image datasets [60, 8], various training and inference techniques [20, 62, 21, 22], and scalibility [51], significant progress has been made in text-to-image generation using diffusion models [4, 51, 45, 56, 17], autoregressive models [52, 76, 11, 15], GANs [59, 28], and their hybrids [54].”
MethodRich Human Feedback for Text-to-Image Generation
Liang, Y., He, J., Li, G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01835 · Cited by 122
“To ensure that the gains from our RAHF model generalize across generative model families, we mainly use Muse [6] as our target model to improve, which is based on a masked transformer architecture and thus different from the Stable Diffusion model variants in our RichHF-18K dataset.”
BackgroundMethodSelf-citationDisenDreamer: Subject-Driven Text-to-Image Generation With Sample-Aware Disentangled Tuning
Chen, H., Zhang, Y. J., Wang, X. Y., Duan, X., Zhou, Y., & Zhu, W. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3369757 · Cited by 59
“Training on large-scale datasets, the text-to-image generation models [6], [11], [22], [23], [32], [34], [35], [39], [46], [47], [48], [49], [51], [52] have achieved great success recently.”
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 71
Implicit Style-Content Separation Using B-LoRA
Frenkel, Y., Vinker, Y., Shamir, A., & Cohen‐Or, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_11 · Cited by 69
RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model
Bie, F., Yang, Y., Zhou, Z., Ghanem, A., Zhang, M., Yao, Z., Wu, X., Holmes, C., Golnari, P. A., Clifton, D. A., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3522305 · Cited by 96
“Moreover, GAN TTI models have a natural computational advantage, generating images in a singular step, whereas both diffusion and autoregressive Transformers require iterative generation process.”
BackgroundMethodInfluentialHRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models
Bakr, E. M., Sun, P., Shen, X., Khan, F. F., Li, L. E., & Elhoseiny, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01834 · Cited by 74
“Recently, there have been rapid efforts toward designing better image-generation models [20, 61, 18, 10, 57, 55, 77, 45, 59].”
BackgroundA Survey of AI Text-to-Image and AI Text-to-Video Generators
Singh, A. (2023) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc57904.2023.10303174 · Cited by 53
“These AI generators analyze the textual data using advance and complex techniques such as attention based Recurrent Neural Network [1], Generative Adversarial Network [2], and transformers [3][4] in order to generate corresponding high-quality images or videos.”
MethodITI-Gen: Inclusive Text-to-Image Generation
Zhang, C., Chen, X., Chai, S., Wu, C., Lagun, D., Beeler, T., & la Torre, F. D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00367 · Cited by 59
“Text-based image generation has been widely studied with numerous model architectures and learning paradigms [48, 62, 71, 59, 23, 79, 18, 19, 9, 69, 78, 16, 17, 38].”
BackgroundA text-guided protein design framework
Liu, S., Li, Y., Li, Z., Gitter, A., Zhu, Y., Lu, J., Xu, Z., Nie, W., Ramanathan, A., Xiao, C., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01011-z · Cited by 97
“, 2019), and text-to-image generation (Chang et al., 2023).”
Self-citationDirected Diffusion: Direct Control of Object Placement through Attention Guidance
Ma, W.-D. K., Lahiri, A., Lewis, J., Leung, T., & Kleijn, W. B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28204 · Cited by 69
ProxEdit: Improving Tuning-Free Real Image Editing with Proximal Guidance
Han, L., Song, W., Chen, Q., Zhang, Z., Song, K., Ren, M., Gao, R., Stathopoulos, A., He, X., Chen, Y., et al. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00424 · Cited by 93
A Comprehensive Survey of Image Generation Models Based on Deep Learning
Li, J., Zhang, C., Zhu, W., & Ren, Y. (2024) · Annals of Data Science · doi:10.1007/s40745-024-00544-1 · Cited by 47
“Subsequently, Google released Muse [64], which is based on the generative Transformer architecture.”
Cazenavette, G., Sud, A., Leung, T., & Usman, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01023 · Cited by 59
AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics
Jia, S., Huang, M., Zhou, Z., Ju, Y., Cai, J., & Lyu, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00096 · Cited by 50
Diffusion-Based Visual Art Creation: A Survey and New Perspectives
Wang, B., Chen, Q., & Wang, Z. (2025) · ACM Computing Surveys · doi:10.1145/3728459 · Cited by 40
Initno: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization
Guo, X., Liu, J., Cui, M., Li, J., Hongyu, Y., & Huang, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00896 · Cited by 66
“Early research mainly focused on GANs [34, 39, 42–44] and autoregressive models [5, 9, 29, 41].”
MethodSmooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Guo, J., Xu, X., Pu, Y., Ni, Z., Wang, C., Vasu, M., Song, S., Huang, G. M., & Shi, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00721 · Cited by 43
“Recently, generating images from text prompts (T2I) become an emerging field [11, 19, 28, 47, 62], among which diffusion models [16, 49, 59, 62, 64] have become quite visible to the public.”
BackgroundLee, J.-K., Jeong, H., Kim, Y., & Cha, S. H. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102686 · Cited by 37
Beyond Deepfake Images: Detecting AI-Generated Videos
Vahdati, D. S., Nguyen, T. D., Azizpour, A., & Stamm, M. C. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00443 · Cited by 37
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28
“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”
MethodDOCCI: Descriptions of Connected and Contrasting Images
Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73027-6_17 · Cited by 69
Microplastics in ecosystems: ecotoxicological threats and strategies for mitigation and governance
Das, B. K., Das, S., Kumar, V., Roy, S., Mitra, A., & Mandal, B. (2025) · Frontiers in Marine Science · doi:10.3389/fmars.2025.1672484 · Cited by 33
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
Wang, J., Liu, X., Di, Z., Liu, Y., & Wang, X. E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.160 · Cited by 50
Comparative Analysis of Generative AI Models
Rani, G., Singh, J., & Khanna, A. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10465941 · Cited by 24
“Common examples are DALL-E2 [7], Stable Fusion [8], Imagen [9], and Muse [10].”
BackgroundHigh-fidelity Person-centric Subject-to-Image Synthesis
Wang, Y., Zhang, W., Zheng, J., & Cheng, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00733 · Cited by 27
“Subject-driven text-to-image generation [3, 4, 7, 13, 21, 27] can synthesize personalized images depicting particular subjects defined by users with a few sample images.”
CapHuman: Capture Your Moments in Parallel Universes
Liang, C., Ma, F., Zhu, L., Deng, Y., & Yang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00612 · Cited by 22
“Besides, some works research on auto-regressive modeling [42] or masked generative modeling [10].”
ET-DM: Text to image via diffusion model with efficient Transformer
Li, H., Xu, F. G., & Zheng, L. L. (2023) · Displays · doi:10.1016/j.displa.2023.102568 · Cited by 18
M2T: Masking Transformers Twice for Faster Decoding
Mentzer, F., Agustson, E., & Tschannen, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00492 · Cited by 20
Identification of Systematic Errors of Image Classifiers on Rare Subgroups
Metzen, J., Hutmacher, R., Hua, N. G., Boreiko, V., & Zhang, D. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.00467 · Cited by 24
“The most similar line of work to ours is built on top of text-to-image synthesis models [42, 43, 45, 9].”
BackgroundPractical PCG Through Large Language Models
Nasir, M. U., & Togelius, J. (2023) · IEEE Conference on Games (CoG) · doi:10.1109/cog57401.2023.10333197 · Cited by 28
“It’s 2023, and the new new thing that can be applied to everything under the sun is Large Language Models (LLMs), such as image generation [8] and neural architecture search [9].”
BackgroundXu, Z., Sangineto, E., & Sebe, N. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00699 · Cited by 17
“Transformer-based text-to-image generative models [29, 8, 9, 42, 1] rely on a vector-quantized image tokenizer to produce a discrete representation of the images, e.”
MethodCustomization Assistant for Text-to-image Generation
Zhou, Y., Zhang, R., Gu, J., & Sun, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00877 · Cited by 22
“The idea of using hierarchical design is also adopted by some follow-up works [1, 25] and proven to be effective in both diffusion models and auto-regressive models.”
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
Jiang, Z., Mao, C., Pan, Y., Han, Z., & Zhang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00859 · Cited by 23
“Building upon large-scale pre-trained image diffusion models [1–3, 14, 15, 30], researchers have focused on various downstream tasks and applications, including text-to-image generation [7–9, 27, 31, 32], controllable image synthesis [18, 29, 33, 50] and image editing [4, 26, 37, 41, 48].”
Vision + Language Applications: A Survey
Zhou, Y., & Shimada, N. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00090 · Cited by 14
Chest-Diffusion: A Light-Weight Text-to-Image Model for Report-to-CXR Generation
Huang, P., Gao, X., Huang, L., Jiao, J., Li, X., Wang, Y., & Guo, Y. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635417 · Cited by 18
Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation
Shabani, M. A., Wang, Z., Liu, D., Zhao, N., Yang, J., & Furukawa, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00881 · Cited by 21
“The recent success of image generative models [6, 9, 30, 32, 41] has enabled the generation of layout design in the image space.”
BackgroundGIVT: Generative Infinite-Vocabulary Transformers
Tschannen, M., Eastwood, C., & Mentzer, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_17 · Cited by 46
Word self-update contrastive adversarial networks for text-to-image synthesis
Xiao, J., Sun, Y., & Bi, X. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.08.038 · Cited by 13
Market-GAN: Adding Control to Financial Market Data Generation with Semantic Context
Xia, H., Sun, S., Wang, X., & An, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29531 · Cited by 14
Feature-Level Cross-Attentional PPG and Motion Signal Fusion for Heart Rate Estimation
Kasnesis, P., Toumanidis, L., Burrello, A., et al. (2023) · IEEE Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00267 · Cited by 11
“Cross-Attention Module: Multi-Head Cross-Attentional (MHCA) module applies the attention operation over the different data modalities in attempt to fuse them effectively, such as text and images [17], images and lidar [18], and motion sensors [19].”
MethodArtificial Intelligence Applications in Green Architecture
MAKSOUD, N. A. E., & Ahmed, E. F. (2024) · Fayoum University Journal of Engineering · doi:10.21608/fuje.2024.345049 · Cited by 10
ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement
Butt, M. A., Wang, K. W., Vázquez-Corral, J., & van de Weijer, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72667-5_26 · Cited by 12
Chataut, S., Bhatta, S., Dahal, B., Ojha, G., Raut, S., Subedi, B., & Bastakoti, B. (2025) · European Journal of Applied Science Engineering and Technology · doi:10.59324/ejaset.2025.3(4).16 · Cited by 12
A Novel Scheme for Generating Context-Aware Images Using Generative Artificial Intelligence
Kim, H., Choi, J.-H., & Choi, J. (2024) · IEEE Access · doi:10.1109/access.2024.3368871 · Cited by 12
“Reference [28] introduced an algorithm that utilizes parallel decoding to create images more efficiently and precisely.”
MethodLMD: Faster Image Reconstruction with Latent Masking Diffusion
Ma, Z., Yu, Z., Li, J., & Zhou, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28209 · Cited by 15
Wang, J., Sun, Z., Tan, Z., Chen, X., Chen, W., Li, H., Zhang, C., & Song, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00807 · Cited by 20
“T2I as a rapidly evolving field, has witnessed the emergence of numerous model architectures and learning paradigms [3, 5–7, 21, 25, 31, 32, 39, 41, 43, 45, 46, 50–52].”
Exploring denoising diffusion models for compressible fluid field prediction
Abaidi, R., & Adams, N. A. (2025) · Computers & Fluids · doi:10.1016/j.compfluid.2025.106665 · Cited by 9
DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation
Lin, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00820 · Cited by 8
“Recent works in personalization have shown significant promise in the realm of generating customized concepts [12, 23, 36, 37], but their extensive optimization requirements limit broader applications.”
BackgroundIterInv: Iterative Inversion for Pixel-Level T2I Models
Tang, C., Wang, K., & van de Weijer, J. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687547 · Cited by 10
“This prior gap is mitigated by recent progress of T2I models [4, 7, 14, 26, 27, 29].”
MethodPix2Gif: Motion-Guided Diffusion for GIF Generation
Kandala, H., Gao, J., & Yang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_3 · Cited by 6
Eta Inversion: Designing an Optimal Eta Function for Diffusion-Based Real Image Editing
Kang, W., Galim, K., & Koo, H. I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72630-9_6 · Cited by 7
MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask
Zhou, Y., Zhou, D., Wang, Y., Feng, J., & Hou, Q. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02294-2 · Cited by 17
“Input: the prompt P ∈ R L × D , the picked tokens set S , a zeroed mask M ∈ R N × L , w 0 = 5 1: for token i in set S do 2: Using the region selection algorithm to find pixels related to the token and putting them into region R i”
Kasnesis, P., Toumanidis, L., Pagliari, D. J., & Burrello, A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3580474 · Cited by 6
“1) Cross-Attention Module: In contrast to MHSA, the Multi-Head Cross-Attentional (MHCA) facilitates the fusion of multiple modalities by leveraging the attention mechanism across them; it has been applied in scenarios involving text with images [37], motion sensors [2] and images with LiDAR [38].”
Large Language Models for Video Surveillance Applications
Silva, U. D., Fernando, L., Lau, B. P. L., Koh, Z., Joyce, S. C., Yuen, B., & Yuen, C. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902892 · Cited by 7
“GenAI [8] [9] technologies, underpinned by advanced deep learning models, have exhibited extraordinary proficiency in diverse applications, such as natural language processing [10] and image generation [11] [12].”
Regeneration Learning: A Learning Paradigm for Data Generation
Tan, X., Qin, T., Bian, J., Liu, T., & Bengio, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30271 · Cited by 10
Improved Visual Story Generation with Adaptive Context Modeling
Feng, Z., Ren, Y., Yu, X., Feng, X., Tang, D., Shi, S., & Qin, B. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.305 · Cited by 20
Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance
Chan, K. C. K., Zhao, Y., Jia, X., Yang, M., & Wang, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00643 · Cited by 7
“Subject-driven text-to-image synthesis [1, 8, 9, 18, 21, 27–29, 31, 40, 43, 47] is a sub-branch of text-to-image synthesis [3, 6, 14, 25, 36, 38, 50] with an additional requirement that the primary attributes in the output aligns with the subjects provided by the user.”
BackgroundSelf-citationAdaNAT: Exploring Adaptive Policy for Token-Based Image Generation
Ni, Z., Wang, Y., Zhou, R., Lu, R., Guo, J., Hu, J., Liu, Z., Yao, Y., & Huang, G. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_17 · Cited by 4
FCAT-Diff: Flexible and Consistent Appearance Transfer Based on Training-free Diffusion Model
Gong, Z., Shao, M., Liu, C., Lv, X., & Liu, H. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104247 · Cited by 4
Augustin, M., Neuhaus, Y., & Hein, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01055 · Cited by 12
Enhancing accuracy, diversity, and random input compatibility in face attribute manipulation
Guo, Q., & Gu, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108683 · Cited by 3
SDFD: Building a Versatile Synthetic Face Image Dataset with Diverse Attributes
Baltsou, G., Sarridis, I., Koutlis, C., & Papadopoulos, S. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10581864 · Cited by 6
“…categories [67]: a) Generative Adversarial Networks (GAN) (e.g., StyleCLIP [43], GigaGAN [19]), b) autoregressive methods (e.g., Parti [65], Muse [10]), c) diffusion models (e.g., Imagen [50], DALL-E2 [47], Stable Diffusion [48], InstructPix2Pix [7]), and d) Neural Radiance Fields (NeRFs) (e.g.,…”
MethodConfidence-Based Iterative Generation for Real-World Image Super-Resolution
Peng, J., Luo, X., Fu, J., & Liu, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73650-6_19 · Cited by 4
Liu, C., Shao, M., Gong, Z., Lv, X., & Meng, L. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110741 · Cited by 3
Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models
Lee, J., Jang, S., Jo, J., Yoon, J., Kim, Y., Kim, J.-H., Ha, J.-W., & Hwang, S. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02125 · Cited by 4
BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion
Qiao, Y., Wang, F., Su, J., Zhang, Y., Yu, Y., Wu, S., & Qi, G.-J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28255 · Cited by 3
Sarid: Arabic Storyteller Using a Fine-Tuned LLM and Text-to-Image Generation
Alabdulrahman, M., Khayyat, R., Almowallad, K., & Alharz, Z. (2024) · International Conference on Computer and Automation Engineering · doi:10.1109/iccae59995.2024.10569190 · Cited by 4
“Generative models have also been used in art generation, such as poetry and fiction stories [2].”
BackgroundMethod$$\sigma $$-GPTs: A New Approach to Autoregressive Models
Pannatier, A., Courdier, E., & Fleuret, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70368-3_9 · Cited by 3
Generating Illustrated Instructions
Menon, S., Misra, I., & Girdhar, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00600 · Cited by 3
“Recent work has examined text-conditioned visual generation through autogressive [8, 69] or diffusion models [4, 18, 21, 43, 48, 49, 51].”
BackgroundMaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
Ma, H., Mahdizadehaghdam, S., Wu, B., Fan, Z., Gu, Y., Zhao, W., Shapira, L., & Xie, X. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00707 · Cited by 6
“Muse [11] achieves state-of-the-art performance in text-to-image generation by training on large-scale text-image datasets and brings significantly efficiency improvement.”
Influential
Metadata
- Authors
- Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., Li, Y., & Krishnan, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.00704
- OpenAlex
- W4313484371
- Cited by
- 157
- References
- 74
- Citations by year
- 2025: 25 · 2024: 92 · 2023: 40
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- text-to-image generation, image inpainting, parallel decoding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
