Muse: Text-To-Image Generation via Masked Generative Transformers
Open accessAbstract
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1, Muse, and PixArt-α, presents a viable solution.
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16× · see citing works
“For the base model, the decoder consists of a stack of Transformer layers which are finally mapped to patch space.”
MethodInfluentialU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 1× · see citing works
“Furthermore, we believe that the use of token-based Trans-formers as a unifying architecture leads to better text-image alignment, compared to diffusion models where the image space is a convolution U-Net (Ronneberger et al., 2015).”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 4× · see citing works
“Our 3B parameter model achieves a CLIP score of 0.32 and an FID score of 7.88 on the COCO (Lin et al., 2014) zero-shot validation benchmark, which compare favorably with that of other large-scale text-to-image models (see Table 2).”
MethodInfluentialSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 3× · see citing works
“In Table 3, we also ablated multi-axis Trans-formers (Zhao et al., 2021) against SWIN (Liu et al., 2021), finding that the former performed better.”
MethodInfluentialGenerative Adversarial Networks
Cohen, G., & Giryes, R. (2021) · International Conference on Computing Communication and Networking Technologies · doi:10.1109/icccnt56998.2023.10306417 · Cited by 28,188 · cited 1× · see citing works
“…VAEs (Van Den Oord et al., 2017) and GANs have shown excellent image generation performance with many variants proposed for both convolutional and Transformer architectures e.g. (Goodfellow et al., 2020; Esser et al., 2021b; Karras et al., 2019; Brock et al., 2018; Donahue & Simonyan, 2019).”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“LLMs (including T5XXL, BERT (Devlin et al., 2018), and GPT (Brown et al., 2020; Radford et al., 2019)) have been shown to learn powerful embeddings, enabling few-shot transfer learning.”
BackgroundGenerative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · cited 13× · see citing works
“…Raffel et al., 2020) and vision tasks (He et al., 2022; Chang et al., 2022); new families of generative models such as diffusion (Ho et al., 2020; Rombach et al., 2022; Saharia et al., 2022) and masking-based generation (Chang et al., 2022); and finally, the availability of large scale image-text…”
BackgroundMethodInfluentialLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 13× · see citing works
“Our 3B parameter model achieves a CLIP score of 0.32 and an FID score of 7.88 on the COCO (Lin et al., 2014) zero-shot validation benchmark, which compare favorably with that of other large-scale text-to-image models (see Table 2).”
BackgroundMethodInfluentialA Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714 · cited 1× · see citing works
“…VAEs (Van Den Oord et al., 2017) and GANs have shown excellent image generation performance with many variants proposed for both convolutional and Transformer architectures e.g. (Goodfellow et al., 2020; Esser et al., 2021b; Karras et al., 2019; Brock et al., 2018; Donahue & Simonyan, 2019).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 11× · see citing works
“Third, we have a “superres” transformer model which translates (un-masked) low-res tokens into high-res tokens, again conditioned on T5-XXL text embeddings, a novel mechanism for super-resolution.”
BackgroundMethodInfluentialDenoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · cited 3× · see citing works
“Diffusion models, based on progressive denoising principles, are now able to synthesize images and video at higher fidelity (Ho et al., 2020; Kingma et al., 2021; Ho et al., 2022).”
BackgroundMethodMasked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 3× · see citing works
“Firstly, the masking approach we use is fundamentally different from the diffusion denoising strategy (Ho et al., 2020; Song et al., 2020), and recent work has shown the benefit of similar masking strategies for image generation compared to diffusion (e.g. see (Li et al., 2022), Table 7).”
BackgroundMethodSelf-citationSGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · cited 1× · see citing works
“…VAEs (Van Den Oord et al., 2017) and GANs have shown excellent image generation performance with many variants proposed for both convolutional and Transformer architectures e.g. (Goodfellow et al., 2020; Esser et al., 2021b; Karras et al., 2019; Brock et al., 2018; Donahue & Simonyan, 2019).”
BackgroundDiffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362 · cited 2× · see citing works
“Generative image models conditioned on text prompts have taken an enormous leap in quality and flexibility in the last few years (Ramesh et al., 2022; Nichol et al., 2021; Saharia et al., 2022; Yu et al., 2022b; Rombach et al., 2022; Midjourney, 2022).”
BackgroundNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 4× · see citing works
“Varied tok-enization approaches such as Discrete VAE’s (Rolfe, 2016), VQVAE (Van Den Oord et al., 2017) and VQGAN (Esser et al., 2021b) have been developed, with the latter being the highest-performing as it combines perceptual and adversarial losses.”
MethodInfluentialDPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 StepsPreprint
Lu, C., Zhou, Y., Bao, F., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.00927 · Cited by 2,705 · cited 1× · see citing works
“…Generated Images HighRes Generated Images Step = 4 Step = 2 Step = 8 Step = 4 Step = 2 Step = 6 Step = 12 Step = 14 Step = 16 Step = 8 Step = 18 (Lu et al., 2022) have greatly reduced the sampling steps of diffusion models, but they have not been widely validated in large scale text-to-image…”
BackgroundPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · cited 10× · see citing works
“The COCO results are zero-shot, using a model trained on the same dataset as Imagen (Saharia et al., 2022).”
BackgroundMethodInfluentialTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 5× · see citing works
“…VAEs (Van Den Oord et al., 2017) and GANs have shown excellent image generation performance with many variants proposed for both convolutional and Transformer architectures e.g. (Goodfellow et al., 2020; Esser et al., 2021b; Karras et al., 2019; Brock et al., 2018; Donahue & Simonyan, 2019).”
BackgroundMethodInfluentialHierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 4× · see citing works
“Compared to Imagen (Saharia et al., 2022) or Dall-E2 (Ramesh et al., 2022) which are built on cascaded pixel-space diffusion models, Muse is significantly more efficient due to the use of a discrete latent space.Compared to Parti (Yu et al., 2022b), a state-of-the-art autoregressive model, Muse is more efficient due to t…”
BackgroundMethodInfluentialDenoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507 · cited 1× · see citing works
“Firstly, the masking approach we use is fundamentally different from the diffusion denoising strategy (Ho et al., 2020; Song et al., 2020), and recent work has shown the benefit of similar masking strategies for image generation compared to diffusion (e.g. see (Li et al., 2022), Table 7).”
MethodDreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442 · see citing works
Self-citationZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 2× · see citing works
“Dreambooth (Ruiz et al., 2022) and Imagic (Kawar et al., 2022) involve fine-tuning of the generative models.”
BackgroundSelf-citationAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 1× · see citing works
“CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) train models to align pairs of text and image embeddings, showing excellent transfer and few-shot capabilities.”
BackgroundAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · cited 1× · see citing works
“Approach Model Type Params FID-30K Zero-shot FID-30K AttnGAN (Xu et al., 2017) GAN 35.”
MethodSelf-citationConceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 5× · see citing works
“For the CC3M results, both Muse models were trained on CC3M.”
MethodResultInfluentialInstructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466 · cited 1× · see citing works
“A number of techniques have been developed on diffusion models to enable editing, personalization and inversion to token space (Gal et al., 2022a; Meng et al., 2021; Ruiz et al., 2022; Kawar et al., 2022; Brooks et al., 2022; Hertz et al., 2022; Mokady et al., 2022).”
BackgroundAdafactor: Adaptive Learning Rates with Sublinear Memory CostPreprint
Shazeer, N., & Stern, M. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1804.04235 · Cited by 1,410 · cited 1× · see citing works
“We use the Adafactor optimizer (Shazeer & Stern, 2018) to save on memory consumption which allowed us to fit a 3B parameter model without model parallelization.”
MethodGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 2× · see citing works
“Generative image models conditioned on text prompts have taken an enormous leap in quality and flexibility in the last few years (Ramesh et al., 2022; Nichol et al., 2021; Saharia et al., 2022; Yu et al., 2022b; Rombach et al., 2022; Midjourney, 2022).”
BackgroundClassifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126 · cited 1× · see citing works
“Diffusion models, based on progressive denoising principles, are now able to synthesize images and video at higher fidelity (Ho et al., 2020; Kingma et al., 2021; Ho et al., 2022).”
BackgroundStyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen‐Or, D., & Lischinski, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,079 · cited 1× · see citing works
“This method works directly on the (tokenized) image and does not require “inverting” the generative process, in contrast with recent zero-shot image editing techniques leveraging generative models (Gal et al., 2022b; Patashnik et al., 2021; Kim et al., 2022; Mokady et al., 2022).”
MethodLAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · see citing works
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00582 · Cited by 813 · cited 2× · see citing works
“Dreambooth (Ruiz et al., 2022) and Imagic (Kawar et al., 2022) involve fine-tuning of the generative models.”
BackgroundSelf-citationNull-text Inversion for Editing Real Images using Guided Diffusion Models
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00585 · Cited by 747 · see citing works
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01043 · Cited by 720 · cited 1× · see citing works
“Most relevant to our method is VQ-diffusion (Gu et al., 2022; Tang et al., 2022), which generalizes our noise process (viewing masking as a discrete diffusion process (Austin et al., 2021)) to one where tokens can be either masked or randomly corrupted.”
MethodAn Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022) · arXiv · doi:10.48550/arxiv.2208.01618 · Cited by 611 · cited 1× · see citing works
“A number of techniques have been developed on diffusion models to enable editing, personalization and inversion to token space (Gal et al., 2022a; Meng et al., 2021; Ruiz et al., 2022; Kawar et al., 2022; Brooks et al., 2022; Hertz et al., 2022; Mokady et al., 2022).”
BackgroundDM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis
Zhu, M., Pan, P., Chen, W., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00595 · Cited by 605 · see citing works
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation
Kim, G., Kwon, T., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00246 · Cited by 570 · cited 1× · see citing works
“This method works directly on the (tokenized) image and does not require “inverting” the generative process, in contrast with recent zero-shot image editing techniques leveraging generative models (Gal et al., 2022b; Patashnik et al., 2021; Kim et al., 2022; Mokady et al., 2022).”
MethodData and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · cited 1× · see citing works
“Such datasets have various potentially problematic issues such as consent and subject awareness (Paullada et al., 2021; Dulhanty, 2020; Scheuerman et al., 2021).”
BackgroundSDEdit: Guided Image Synthesis and Editing with Stochastic Differential EquationsPreprint
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., & Ermon, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01073 · Cited by 528 · see citing works
Gal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., & Cohen‐Or, D. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530164 · Cited by 503 · see citing works
Prompt-to-Prompt Image Editing with Cross Attention ControlPreprint
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., & Cohen‐Or, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.01626 · Cited by 496 · cited 2× · see citing works
“A number of techniques have been developed on diffusion models to enable editing, personalization and inversion to token space (Gal et al., 2022a; Meng et al., 2021; Ruiz et al., 2022; Kawar et al., 2022; Brooks et al., 2022; Hertz et al., 2022; Mokady et al., 2022).”
BackgroundMethodXia, W., Zhang, Y., Yang, Y., Xue, J., Zhou, B., & Yang, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3181070 · Cited by 496 · cited 1× · see citing works
“Image Editing with Generative Models: GANs have been extensively studied for image editing and manipulation capabilities (see (Xia et al., 2022) for a survey).”
BackgroundSelf-citationCogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450 · see citing works
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446 · cited 8× · see citing works
“…et al., 2022; Chang et al., 2022); new families of generative models such as diffusion (Ho et al., 2020; Rombach et al., 2022; Saharia et al., 2022) and masking-based generation (Chang et al., 2022); and finally, the availability of large scale image-text paired datasets (Schuhmann et al., 2021).”
BackgroundMethodInfluentialSelf-citationWomen Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · cited 1× · see citing works
“…on such datasets simply amplifies these biases and significant additional research is required on how to mitigate such biases, and generate datasets that are free of them: this is a very important topic (Buolamwini & Gebru, 2018; Hendricks et al., 2018) that is out of the scope of this paper.”
BackgroundDF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis
Tao, M., Tang, H., Wu, F., Jing, X., Bao, B., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01602 · Cited by 399 · see citing works
Large Scale Adversarial Representation LearningPreprint
Donahue, J., & Simonyan, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02544 · Cited by 398 · cited 1× · see citing works
“…VAEs (Van Den Oord et al., 2017) and GANs have shown excellent image generation performance with many variants proposed for both convolutional and Transformer architectures e.g. (Goodfellow et al., 2020; Esser et al., 2021b; Karras et al., 2019; Brock et al., 2018; Donahue & Simonyan, 2019).”
BackgroundKingma, D. P., Salimans, T., Poole, B., & Ho, J. (2021) · arXiv · doi:10.48550/arxiv.2107.00630 · Cited by 374 · cited 1× · see citing works
“Diffusion models, based on progressive denoising principles, are now able to synthesize images and video at higher fidelity (Ho et al., 2020; Kingma et al., 2021; Ho et al., 2022).”
BackgroundScaling Autoregressive Models for Content-Rich Text-to-Image Generation
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.10789 · Cited by 371 · cited 11× · see citing works
“…steps in our base model and 4096 tokens using 8 decoding steps in our super-resolution model, as compared to the 256 or 4096 decoding steps required for autoregressive models (e.g. (Yu et al., 2022b)) and hundreds of steps for diffusion models (e.g., (Rombach et al., 2022; Saharia et al., 2022)).”
BackgroundMethodInfluentialSelf-citationCross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 359 · see citing works
Self-citationLAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 2× · see citing works
“Imagen (Saharia et al., 2022) and Parti (Yu et al., 2022b) use similar large scale text-image datasets (Schuhmann et al., 2021; 2022) to learn how to predict images from text inputs, achieving excellent results on FID and human evaluations.”
BackgroundStructured Denoising Diffusion Models in Discrete State-SpacesPreprint
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03006 · Cited by 340 · cited 1× · see citing works
“Most relevant to our method is VQ-diffusion (Gu et al., 2022; Tang et al., 2022), which generalizes our noise process (viewing masking as a discrete diffusion process (Austin et al., 2021)) to one where tokens can be either masked or randomly corrupted.”
MethodMake-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · arXiv · doi:10.48550/arxiv.2203.13131 · Cited by 332 · see citing works
A domain-specific supercomputer for training deep neural networks
Jouppi, N., Yoon, D., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., & Patterson, D. A. (2020) · Communications of the ACM · doi:10.1145/3360307 · Cited by 312 · cited 1× · see citing works
“Training is performed for 1M steps, with a batch size of 512 on 512-core TPU-v4 chips (Jouppi et al., 2020).”
MethodAutoregressive Image Generation using Residual Quantization
Lee, D., Kim, C., Kim, S., Cho, M., & Han, W.-S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01123 · Cited by 305 · see citing works
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022) · arXiv · doi:10.48550/arxiv.2204.03458 · Cited by 301 · cited 4× · see citing works
“The final logits (cid:96) g are formed by moving away from the (cid:96) u by an amount t , the guidance scale : Intuitively, CFG trades off diversity for fidelity.”
BackgroundMethodInfluentialDiscrete Variational AutoencodersPreprint
Rolfe, J. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.02200 · Cited by 291 · cited 1× · see citing works
“Varied tok-enization approaches such as Discrete VAE’s (Rolfe, 2016), VQVAE (Van Den Oord et al., 2017) and VQGAN (Esser et al., 2021b) have been developed, with the latter being the highest-performing as it combines perceptual and adversarial losses.”
MethodLarge image datasets: A pyrrhic win for computer vision?
Birhane, A., & Prabhu, V. U. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00158 · Cited by 257 · cited 1× · see citing works
“Many of the commonly used datasets tend to reflect negative social stereotypes and viewpoints (Prabhu & Birhane, 2020).”
BackgroundVector-quantized Image Modeling with Improved VQGANPreprint
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., & Wu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04627 · Cited by 227 · cited 1× · see citing works
“ViT-VQGAN (Yu et al., 2021) extends VQGAN to the Transformer architecture and introduces techniques like factorized codes and l 2 normalized codes to improve codebook usage.”
MethodSelf-citationDo Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
Scheuerman, M. K., Hanna, A., & Denton, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476058 · Cited by 216 · cited 1× · see citing works
“Such datasets have various potentially problematic issues such as consent and subject awareness (Paullada et al., 2021; Dulhanty, 2020; Scheuerman et al., 2021).”
BackgroundHughes, R. T., Zhu, L., & Bednarz, T. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.604234 · Cited by 178 · cited 1× · see citing works
“Generative models (Saharia et al., 2022; Yu et al., 2022b; Rombach et al., 2022; Midjourney, 2022) hold significant potential to augment human creativity (Hughes et al., 2021).”
BackgroundImage Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
Steed, R., & Caliskan, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445932 · Cited by 167 · cited 1× · see citing works
“However, it is well known that they can also be leveraged for misinformation, harassment and various types of social and cultural biases (Franks & Waldman, 2018; Whittaker et al., 2020; Srinivasan & Uchino, 2021; Steed & Caliskan, 2021).”
BackgroundAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial NetworksPreprint
Xu, T., Zhang, P., Huang, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10485 · Cited by 159 · see citing works
MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
Li, T., Chang, H., Mishra, S. K., Zhang, H., Katabi, D., & Krishnan, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00213 · Cited by 122 · cited 1× · see citing works
“, 2022), or to enable more effective loss functions such as classification instead of regression (Chang et al., 2022; Lezama et al., 2022; Li et al., 2022).”
BackgroundSelf-citationImproving Text-to-Image Synthesis Using Contrastive Learning
Ye, H., Yang, X., Takác, M., et al. (2021) · British Machine Vision Conference · doi:10.5244/c.35.137 · Cited by 93
Large image datasets: A pyrrhic win for computer vision?
Prabhu, V. U., & Birhane, A. (2020) · arXiv · doi:10.48550/arxiv.2006.16923 · Cited by 91 · see citing works
“All around me are synthetic faces”: The mad world of AI-generated media
Whittaker, L., Kietzmann, T. C., Kietzmann, J., & Dabirian, A. (2020) · IT Professional · doi:10.1109/mitp.2020.2985492 · Cited by 90 · cited 1× · see citing works
“However, it is well known that they can also be leveraged for misinformation, harassment and various types of social and cultural biases (Franks & Waldman, 2018; Whittaker et al., 2020; Srinivasan & Uchino, 2021; Steed & Caliskan, 2021).”
BackgroundImproved Vector Quantized Diffusion Models
Tang, Z.-C., Gu, S., Bao, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2205.16007 · Cited by 78 · cited 1×
“Most relevant to our method is VQ-diffusion (Gu et al., 2022; Tang et al., 2022), which generalizes our noise process (viewing masking as a discrete diffusion process (Austin et al., 2021)) to one where tokens can be either masked or randomly corrupted.”
MethodImproved Masked Image Generation with Token-Critic
Lezama, J., Chang, H., Jiang, L., et al. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2209.04439 · Cited by 76 · cited 2×
“…for multiple generative models due to the ability to move the bulk of the computation from input (pixel) space to latents (Rom-bach et al., 2022), or to enable more effective loss functions such as classification instead of regression (Chang et al., 2022; Lezama et al., 2022; Li et al., 2022).”
BackgroundSelf-citationImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image SynthesisPreprint
Esser, P., Rombach, R., Blattmann, A., & Ommer, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08827 · Cited by 72 · see citing works
Biases in Generative Art: A Causal Look from the Lens of Art History
Srinivasan, R., & Uchino, K. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445869 · Cited by 67 · cited 1× · see citing works
“However, it is well known that they can also be leveraged for misinformation, harassment and various types of social and cultural biases (Franks & Waldman, 2018; Whittaker et al., 2020; Srinivasan & Uchino, 2021; Steed & Caliskan, 2021).”
BackgroundImproved Transformer for High-Resolution GANsPreprint
Zhao, L., Zhang, Z., Chen, T., Metaxas, D., & Zhang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07631 · Cited by 64 · cited 5× · see citing works
“To address this issue, we experimented with two architectures, the Swin transformer (Liu et al., 2021) and Multi-axis transformer (Zhao et al., 2021), to incorporate local-global attention.”
MethodInfluentialSelf-citationInstructPix2Pix: Learning to Follow Image Editing InstructionsPreprint
Brooks, T., Holynski, A., & Efros, A. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09800 · Cited by 44 · see citing works
LAFITE: Towards Language-Free Training for Text-to-Image Generation
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T. (2021) · arXiv · doi:10.48550/arxiv.2111.13792 · Cited by 38 · see citing works
Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer
Lee, D., Kim, C., Kim, S., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.04452 · Cited by 36
Linearly Mapping from Image to Text Space
Merullo, J., Castricato, L., Eickhoff, C., & Pavlick, E. (2022) · arXiv · doi:10.48550/arxiv.2209.15162 · Cited by 34 · cited 1× · see citing works
“…Muse model learns to map these rich visual and semantic concepts in the LLM embeddings to the generated images; it has been shown in recent work (Merullo et al., 2022) that the conceptual representations learned by LLM’s are roughly linearly mappable to those learned by models trained on vision…”
MethodDF-GAN: A Simple and Effective Baseline for Text-to-Image SynthesisPreprint
Ming, T., Tang, H., Wu, F. Q., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05865 · Cited by 26 · see citing works
Improving Text-to-Image Synthesis Using Contrastive LearningPreprint
Ye, H., Yang, X., Takáč, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02423 · Cited by 24 · see citing works
Dawkins, M. (2007) · doi:10.1093/acprof:oso/9780198569350.003.0003 · Cited by 1
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2022) · arXiv · doi:10.48550/arxiv.2210.09276
Self-citationMAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
Li, T., Chang, H., Mishra, S. K., Zhang, H., Katabi, D., & Krishnan, D. (2022) · arXiv · doi:10.48550/arxiv.2211.09117
Self-citationNull-text Inversion for Editing Real Images using Guided Diffusion Models
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2022) · arXiv · doi:10.48550/arxiv.2211.09794
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y., Laszlo, S., Fleet, D. J., Soricut, R., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.06909
Flagged first, then by citations
Research works printed but not linked: 5
- Buolamwini, J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pp. 77-91. PMLR, 2018.
- Dulhanty, C. Issues in computer vision data collection: Bias, consent, and label taxonomy. Master's thesis, University of Waterloo, 2020.
- Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
- Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In ICLR , 2022.
Cited by
Sorted by citations
GLIGEN: Open-Set Grounded Text-to-Image Generation
Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., & Lee, Y. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02156 · Cited by 610 · see citing works
Self-citationScaling up GANs for Text-to-Image Synthesis
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00976 · Cited by 426 · see citing works
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation
Wei, Y., Zhang, Y., Ji, Z., Bai, J., Lei Zhang, A., & Zuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01461 · Cited by 274 · see citing works
Adversarial Diffusion Distillation
Sauer, A., Lorenz, D., Blattmann, A., & Rombach, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73016-0_6 · Cited by 306 · see citing works
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Jayasumana, S., Ramalingam, S., Veit, A., Gläsner, D., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00889 · Cited by 243 · see citing works
“While the Muse model [3] generates multiple to-kens at a time, it still requires iterative sampling steps to generate the final image, as shown in Figure 3.”
MethodA Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182 · see citing works
“, 2022] • Muse reaches state-of-the-art text conditional image generation with a masked transformer approach [Chang et al., 2023].”
BackgroundSVDiff: Compact Parameter Space for Diffusion Fine-Tuning
Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00673 · Cited by 224 · see citing works
“Text-to-image diffusion models Diffusion models [58, 61, 19, 62, 40, 59, 18, 60, 10] have proven to be highly effective in learning data distributions and have shown impressive results in image synthesis, leading to various applications [74, 45, 9, 49, 26, 23, 34, 56, 28, 24, 29].”
BackgroundSelf-citationMoMask: Generative Masked Modeling of 3D Human Motions
Guo, C., Mu, Y., Javed, M. G., Wang, S., & Cheng, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00186 · Cited by 201 · see citing works
Unleashing Text-to-Image Diffusion Models for Visual Perception
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., & Lu, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00527 · Cited by 188 · see citing works
One-Step Diffusion with Distribution Matching Distillation
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00632 · Cited by 179 · see citing works
“Distribution Matching Recently, a few classes of generative models have shown success in scaling up to complex datasets by recovering samples that are corrupted by a pre-defined mechanism, such as noise injection [21, 61, 64] or token masking [5, 60, 86].”
BackgroundSelf-citationAblating Concepts in Text-to-Image Diffusion Models
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., & Zhu, J.-Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02074 · Cited by 150 · see citing works
“Large-scale text-to-image models have demonstrated remarkable ability in synthesizing photorealistic images [51, 43, 56, 54, 76, 14].”
BackgroundInstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
Shi, J., Xiong, W., Lin, Z., & Jung, H. J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00816 · Cited by 192 · see citing works
“We encode X into a global concept embedding c g via Concept Encoder E g and obtain”
MethodMultimodal Image Synthesis and Editing: The Generative AI Era
Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., & Xing, E. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3305243 · Cited by 162 · see citing works
“Text: CogView [69], DALL-E [70], NÜWA [44], Parti [71], Make-A-Scene [72], Muse [73]”
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models
Ruiz, N., Li, Y., Jampani, V., Wei, W., Hou, T., Pritch, Y., Wadhwa, N., Rubinstein, M., & Aberman, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00624 · Cited by 155 · see citing works
“Text-to-Image Models Several recent models such as Im-agen [26], DALL-E2 [22], Stable Diffusion (SD) [24], Muse [7], Parti [33], etc., demonstrate excellent image generation capabilities given a text prompt.”
MethodSelf-citationHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101 · see citing works
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147 · see citing works
TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition
Lu, S., Liu, Y., & Kong, A. W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00218 · Cited by 105 · see citing works
InstanceDiffusion: Instance-Level Control for Image Generation
Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., & Misra, I. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00596 · Cited by 100 · see citing works
“Image generation models [8, 9, 18, 22, 26, 27, 44, 46, 50, 53, 69] trained on web-scale data have made tremendous progress in the recent years.”
BackgroundTransformer-Based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey
Dubey, S. R., & Singh, S. K. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3404910 · Cited by 96 · see citing works
“Masked generative transformer (Muse) is investigated for text-to-image generation [155].”
BackgroundVisual Prompt Tuning for Generative Transfer Learning
Sohn, K., Chang, H., Lezama, J., Polanía, L. F., Zhang, H., Hao, Y., Essa, I. A., & Jiang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01900 · Cited by 94 · see citing works
Self-citationMACE: Mass Concept Erasure in Diffusion Models
Lu, S., Wang, Z., Li, L., Liu, Y., & Adams Wai Kin, K. O. N. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00615 · Cited by 90 · see citing works
“In large-scale text-to-image (T2I) models [9, 14, 34, 40, 49, 54, 57, 73, 74, 76, 77], the task of concept erasure aims to remove concepts that may be harmful, copyrighted, or offensive.”
BackgroundOnline Detection of AI-Generated Images
Epstein, D. C., Jain, I., Wang, O., & Zhang, R. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00045 · Cited by 83 · see citing works
“Methods such as Imagen [41], Parti [60], Muse [13], and GigaGAN [24] offer high-quality generations with alternative architectures but do not have public APIs or source code released.”
TextMesh: Generation of Realistic 3D Meshes From Text Prompts
Tsalicoglou, C., Manhardt, F., Tonioni, A., Niemeyer, M., & Tombari, F. (2024) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv62453.2024.00154 · Cited by 83 · see citing works
Scaling Robot Learning with Semantically Imagined Experience
Yu, T., Xiao, T. G., Tompson, J., Stone, A., Wang, S., Brohan, A., Singh, J., Tan, C., Dee, M., Peralta, J., et al. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.027 · Cited by 83 · see citing works
Photorealistic Video Generation with Diffusion Models
Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I. A., Jiang, L., & Lezama, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72986-7_23 · Cited by 112 · see citing works
Self-citationMIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis
Zhou, D., Li, Y., Ma, F., Zhang, X.-T., & Yang, Y. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.00651 · Cited by 175 · see citing works
“2) Inspired by the principle of divide and conquer, we introduce a novel MIGC approach that enhances pre-trained stable diffusion with improved MIG capabilities.”
MethodSelf-citationExpressive Text-to-Image Generation with Rich Text
Ge, S., Park, T., Zhu, J.-Y., & Huang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00694 · Cited by 69 · see citing works
“…text-image datasets [60, 8], various training and inference techniques [20, 62, 21, 22], and scalibility [51], significant progress has been made in text-to-image generation using diffusion models [4, 51, 45, 56, 17], autoregressive models [52, 76, 11, 15], GANs [59, 28], and their hybrids [54].”
MethodRich Human Feedback for Text-to-Image Generation
Liang, Y., He, J., Li, G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01835 · Cited by 78 · see citing works
“To ensure that the gains from our RAHF model generalize across generative model families, we mainly use Muse [6] as our target model to improve, which is based on a masked transformer architecture and thus different from the Stable Diffusion model variants in our RichHF-18K dataset.”
BackgroundMethodSelf-citationDisenDreamer: Subject-Driven Text-to-Image Generation With Sample-Aware Disentangled Tuning
Chen, H., Zhang, Y. J., Wang, X. Y., Duan, X., Zhou, Y., & Zhu, W. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3369757 · Cited by 56 · see citing works
“Training on large-scale datasets, the text-to-image generation models [6], [11], [22], [23], [32], [34], [35], [39], [46], [47], [48], [49], [51], [52] have achieved great success recently.”
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
Implicit Style-Content Separation Using B-LoRA
Frenkel, Y., Vinker, Y., Shamir, A., & Cohen‐Or, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_11 · Cited by 69 · see citing works
RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model
Bie, F., Yang, Y., Zhou, Z., Ghanem, A., Zhang, M., Yao, Z., Wu, X., Holmes, C., Golnari, P. A., Clifton, D. A., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3522305 · Cited by 76 · see citing works
“Moreover, GAN TTI models have a natural computational advantage, generating images in a singular step, whereas both diffusion and autoregressive Transformers require iterative generation process.”
BackgroundMethodInfluentialHRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models
Bakr, E. M., Sun, P., Shen, X., Khan, F. F., Li, L. E., & Elhoseiny, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01834 · Cited by 55 · see citing works
“Recently, there have been rapid efforts toward designing better image-generation models [20, 61, 18, 10, 57, 55, 77, 45, 59].”
BackgroundA Survey of AI Text-to-Image and AI Text-to-Video Generators
Singh, A. (2023) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc57904.2023.10303174 · Cited by 53 · see citing works
“These AI generators analyze the textual data using advance and complex techniques such as attention based Recurrent Neural Network [1], Generative Adversarial Network [2], and transformers [3][4] in order to generate corresponding high-quality images or videos.”
MethodITI-Gen: Inclusive Text-to-Image Generation
Zhang, C., Chen, X., Chai, S., Wu, C., Lagun, D., Beeler, T., & la Torre, F. D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00367 · Cited by 51 · see citing works
“Text-based image generation has been widely studied with numerous model architectures and learning paradigms [48, 62, 71, 59, 23, 79, 18, 19, 9, 69, 78, 16, 17, 38].”
BackgroundA text-guided protein design framework
Liu, S., Li, Y., Li, Z., Gitter, A., Zhu, Y., Lu, J., Xu, Z., Nie, W., Ramanathan, A., Xiao, C., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01011-z · Cited by 66 · see citing works
“, 2019), and text-to-image generation (Chang et al., 2023).”
Self-citationDirected Diffusion: Direct Control of Object Placement through Attention Guidance
Ma, W.-D. K., Lahiri, A., Lewis, J., Leung, T., & Kleijn, W. B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28204 · Cited by 54 · see citing works
ProxEdit: Improving Tuning-Free Real Image Editing with Proximal Guidance
Han, L., Song, W., Chen, Q., Zhang, Z., Song, K., Ren, M., Gao, R., Stathopoulos, A., He, X., Chen, Y., et al. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00424 · Cited by 55 · see citing works
A Comprehensive Survey of Image Generation Models Based on Deep Learning
Li, J., Zhang, C., Zhu, W., & Ren, Y. (2024) · Annals of Data Science · doi:10.1007/s40745-024-00544-1 · Cited by 42 · see citing works
“Subsequently, Google released Muse [64], which is based on the generative Transformer architecture.”
Cazenavette, G., Sud, A., Leung, T., & Usman, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01023 · Cited by 47 · see citing works
AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics
Jia, S., Huang, M., Zhou, Z., Ju, Y., Cai, J., & Lyu, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00096 · Cited by 44 · see citing works
Diffusion-Based Visual Art Creation: A Survey and New Perspectives
Wang, B., Chen, Q., & Wang, Z. (2025) · ACM Computing Surveys · doi:10.1145/3728459 · Cited by 40 · see citing works
Initno: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization
Guo, X., Liu, J., Cui, M., Li, J., Hongyu, Y., & Huang, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00896 · Cited by 36 · see citing works
“Early research mainly focused on GANs [34, 39, 42–44] and autoregressive models [5, 9, 29, 41].”
MethodSmooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Guo, J., Xu, X., Pu, Y., Ni, Z., Wang, C., Vasu, M., Song, S., Huang, G. M., & Shi, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00721 · Cited by 32 · see citing works
“Recently, generating images from text prompts (T2I) become an emerging field [11, 19, 28, 47, 62], among which diffusion models [16, 49, 59, 62, 64] have become quite visible to the public.”
BackgroundLee, J.-K., Jeong, H., Kim, Y., & Cha, S. H. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102686 · Cited by 36 · see citing works
Beyond Deepfake Images: Detecting AI-Generated Videos
Vahdati, D. S., Nguyen, T. D., Azizpour, A., & Stamm, M. C. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00443 · Cited by 37 · see citing works
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28 · see citing works
“Because these text sources are imperfect, to improve the alignment between generated futures and the text prompt, we employ classifier-free guidance [45, 46] at inference time.”
MethodDOCCI: Descriptions of Connected and Contrasting Images
Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73027-6_17 · Cited by 26 · see citing works
Microplastics in ecosystems: ecotoxicological threats and strategies for mitigation and governance
Das, B. K., Das, S., Kumar, V., Roy, S., Mitra, A., & Mandal, B. (2025) · Frontiers in Marine Science · doi:10.3389/fmars.2025.1672484 · Cited by 32 · see citing works
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
Wang, J., Liu, X., Di, Z., Liu, Y., & Wang, X. E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.160 · Cited by 26 · see citing works
Comparative Analysis of Generative AI Models
Rani, G., Singh, J., & Khanna, A. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10465941 · Cited by 24 · see citing works
“Common examples are DALL-E2 [7], Stable Fusion [8], Imagen [9], and Muse [10].”
BackgroundHigh-fidelity Person-centric Subject-to-Image Synthesis
Wang, Y., Zhang, W., Zheng, J., & Cheng, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00733 · Cited by 23 · see citing works
“Subject-driven text-to-image generation [3, 4, 7, 13, 21, 27] can synthesize personalized images depicting particular subjects defined by users with a few sample images.”
CapHuman: Capture Your Moments in Parallel Universes
Liang, C., Ma, F., Zhu, L., Deng, Y., & Yang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00612 · Cited by 20 · see citing works
“Besides, some works research on auto-regressive modeling [42] or masked generative modeling [10].”
ET-DM: Text to image via diffusion model with efficient Transformer
Li, H., Xu, F. G., & Zheng, L. L. (2023) · Displays · doi:10.1016/j.displa.2023.102568 · Cited by 18 · see citing works
M2T: Masking Transformers Twice for Faster Decoding
Mentzer, F., Agustson, E., & Tschannen, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00492 · Cited by 17 · see citing works
Identification of Systematic Errors of Image Classifiers on Rare Subgroups
Metzen, J. H., Hutmacher, R., Hua, N. G., Boreiko, V., & Zhang, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00467 · Cited by 15 · see citing works
“The most similar line of work to ours is built on top of text-to-image synthesis models [42, 43, 45, 9].”
BackgroundPractical PCG Through Large Language Models
Nasir, M. U., & Togelius, J. (2023) · IEEE Conference on Games (CoG) · doi:10.1109/cog57401.2023.10333197 · Cited by 14 · see citing works
“It’s 2023, and the new new thing that can be applied to everything under the sun is Large Language Models (LLMs), such as image generation [8] and neural architecture search [9].”
BackgroundXu, Z., Sangineto, E., & Sebe, N. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00699 · Cited by 14 · see citing works
“Transformer-based text-to-image generative models [29, 8, 9, 42, 1] rely on a vector-quantized image tokenizer to produce a discrete representation of the images, e.”
MethodCustomization Assistant for Text-to-image Generation
Zhou, Y., Zhang, R., Gu, J., & Sun, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00877 · Cited by 17 · see citing works
“The idea of using hierarchical design is also adopted by some follow-up works [1, 25] and proven to be effective in both diffusion models and auto-regressive models.”
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
Jiang, Z., Mao, C., Pan, Y., Han, Z., & Zhang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00859 · Cited by 16 · see citing works
“Building upon large-scale pre-trained image diffusion models [1–3, 14, 15, 30], researchers have focused on various downstream tasks and applications, including text-to-image generation [7–9, 27, 31, 32], controllable image synthesis [18, 29, 33, 50] and image editing [4, 26, 37, 41, 48].”
Vision + Language Applications: A Survey
Zhou, Y., & Shimada, N. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00090 · Cited by 14 · see citing works
Chest-Diffusion: A Light-Weight Text-to-Image Model for Report-to-CXR Generation
Huang, P., Gao, X., Huang, L., Jiao, J., Li, X., Wang, Y., & Guo, Y. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635417 · Cited by 15 · see citing works
Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation
Shabani, M. A., Wang, Z., Liu, D., Zhao, N., Yang, J., & Furukawa, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00881 · Cited by 15 · see citing works
“The recent success of image generative models [6, 9, 30, 32, 41] has enabled the generation of layout design in the image space.”
BackgroundGIVT: Generative Infinite-Vocabulary Transformers
Tschannen, M., Eastwood, C., & Mentzer, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_17 · Cited by 13 · see citing works
Word self-update contrastive adversarial networks for text-to-image synthesis
Xiao, J., Sun, Y., & Bi, X. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.08.038 · Cited by 12 · see citing works
Market-GAN: Adding Control to Financial Market Data Generation with Semantic Context
Xia, H., Sun, S., Wang, X., & An, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29531 · Cited by 14 · see citing works
Feature-Level Cross-Attentional PPG and Motion Signal Fusion for Heart Rate Estimation
Kasnesis, P., Toumanidis, L., Burrello, A., et al. (2023) · IEEE Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00267 · Cited by 11 · see citing works
“Cross-Attention Module: Multi-Head Cross-Attentional (MHCA) module applies the attention operation over the different data modalities in attempt to fuse them effectively, such as text and images [17], images and lidar [18], and motion sensors [19].”
MethodArtificial Intelligence Applications in Green Architecture
MAKSOUD, N. A. E., & Ahmed, E. F. (2024) · Fayoum University Journal of Engineering · doi:10.21608/fuje.2024.345049 · Cited by 10 · see citing works
ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement
Butt, M. A., Wang, K. W., Vázquez-Corral, J., & van de Weijer, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72667-5_26 · Cited by 12 · see citing works
Chataut, S., Bhatta, S., Dahal, B., Ojha, G., Raut, S., Subedi, B., & Bastakoti, B. (2025) · European Journal of Applied Science Engineering and Technology · doi:10.59324/ejaset.2025.3(4).16 · Cited by 9 · see citing works
A Novel Scheme for Generating Context-Aware Images Using Generative Artificial Intelligence
Kim, H., Choi, J.-H., & Choi, J. (2024) · IEEE Access · doi:10.1109/access.2024.3368871 · Cited by 9 · see citing works
“Reference [28] introduced an algorithm that utilizes parallel decoding to create images more efficiently and precisely.”
MethodLMD: Faster Image Reconstruction with Latent Masking Diffusion
Ma, Z., Yu, Z., Li, J., & Zhou, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28209 · Cited by 12 · see citing works
Wang, J., Sun, Z., Tan, Z., Chen, X., Chen, W., Li, H., Zhang, C., & Song, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00807 · Cited by 13 · see citing works
“T2I as a rapidly evolving field, has witnessed the emergence of numerous model architectures and learning paradigms [3, 5–7, 21, 25, 31, 32, 39, 41, 43, 45, 46, 50–52].”
Exploring denoising diffusion models for compressible fluid field prediction
Abaidi, R., & Adams, N. A. (2025) · Computers & Fluids · doi:10.1016/j.compfluid.2025.106665 · Cited by 7 · see citing works
DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation
Lin, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00820 · Cited by 7 · see citing works
“Recent works in personalization have shown significant promise in the realm of generating customized concepts [12, 23, 36, 37], but their extensive optimization requirements limit broader applications.”
BackgroundIterInv: Iterative Inversion for Pixel-Level T2I Models
Tang, C., Wang, K., & van de Weijer, J. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687547 · Cited by 7 · see citing works
“This prior gap is mitigated by recent progress of T2I models [4, 7, 14, 26, 27, 29].”
MethodPix2Gif: Motion-Guided Diffusion for GIF Generation
Kandala, H., Gao, J., & Yang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_3 · Cited by 6 · see citing works
Eta Inversion: Designing an Optimal Eta Function for Diffusion-Based Real Image Editing
Kang, W., Galim, K., & Koo, H. I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72630-9_6 · Cited by 7 · see citing works
MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask
Zhou, Y., Zhou, D., Wang, Y., Feng, J., & Hou, Q. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02294-2 · Cited by 8 · see citing works
“Input: the prompt P ∈ R L × D , the picked tokens set S , a zeroed mask M ∈ R N × L , w 0 = 5 1: for token i in set S do 2: Using the region selection algorithm to find pixels related to the token and putting them into region R i”
Kasnesis, P., Toumanidis, L., Pagliari, D. J., & Burrello, A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3580474 · Cited by 5 · see citing works
“1) Cross-Attention Module: In contrast to MHSA, the Multi-Head Cross-Attentional (MHCA) facilitates the fusion of multiple modalities by leveraging the attention mechanism across them; it has been applied in scenarios involving text with images [37], motion sensors [2] and images with LiDAR [38].”
Large Language Models for Video Surveillance Applications
Silva, U. D., Fernando, L., Lau, B. P. L., Koh, Z., Joyce, S. C., Yuen, B., & Yuen, C. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902892 · Cited by 7 · see citing works
“GenAI [8] [9] technologies, underpinned by advanced deep learning models, have exhibited extraordinary proficiency in diverse applications, such as natural language processing [10] and image generation [11] [12].”
Regeneration Learning: A Learning Paradigm for Data Generation
Tan, X., Qin, T., Bian, J., Liu, T., & Bengio, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30271 · Cited by 6 · see citing works
Improved Visual Story Generation with Adaptive Context Modeling
Feng, Z., Ren, Y., Yu, X., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.305 · Cited by 4 · see citing works
Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance
Chan, K. C. K., Zhao, Y., Jia, X., Yang, M., & Wang, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00643 · Cited by 4 · see citing works
“Subject-driven text-to-image synthesis [1, 8, 9, 18, 21, 27–29, 31, 40, 43, 47] is a sub-branch of text-to-image synthesis [3, 6, 14, 25, 36, 38, 50] with an additional requirement that the primary attributes in the output aligns with the subjects provided by the user.”
BackgroundSelf-citationAdaNAT: Exploring Adaptive Policy for Token-Based Image Generation
Ni, Z., Wang, Y., Zhou, R., Lu, R., Guo, J., Hu, J., Liu, Z., Yao, Y., & Huang, G. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_17 · Cited by 4 · see citing works
FCAT-Diff: Flexible and Consistent Appearance Transfer Based on Training-free Diffusion Model
Gong, Z., Shao, M., Liu, C., Lv, X., & Liu, H. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104247 · Cited by 4 · see citing works
Augustin, M., Neuhaus, Y., & Hein, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01055 · Cited by 4 · see citing works
Enhancing accuracy, diversity, and random input compatibility in face attribute manipulation
Guo, Q., & Gu, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108683 · Cited by 3 · see citing works
SDFD: Building a Versatile Synthetic Face Image Dataset with Diverse Attributes
Baltsou, G., Sarridis, I., Koutlis, C., & Papadopoulos, S. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10581864 · Cited by 3 · see citing works
“…categories [67]: a) Generative Adversarial Networks (GAN) (e.g., StyleCLIP [43], GigaGAN [19]), b) autoregressive methods (e.g., Parti [65], Muse [10]), c) diffusion models (e.g., Imagen [50], DALL-E2 [47], Stable Diffusion [48], InstructPix2Pix [7]), and d) Neural Radiance Fields (NeRFs) (e.g.,…”
MethodConfidence-Based Iterative Generation for Real-World Image Super-Resolution
Peng, J., Luo, X., Fu, J., & Liu, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73650-6_19 · Cited by 3 · see citing works
Liu, C., Shao, M., Gong, Z., Lv, X., & Meng, L. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110741 · Cited by 3 · see citing works
Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models
Lee, J., Jang, S., Jo, J., Yoon, J., Kim, Y., Kim, J.-H., Ha, J.-W., & Hwang, S. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02125 · Cited by 3 · see citing works
BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion
Qiao, Y., Wang, F., Su, J., Zhang, Y., Yu, Y., Wu, S., & Qi, G.-J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28255 · Cited by 3 · see citing works
Sarid: Arabic Storyteller Using a Fine-Tuned LLM and Text-to-Image Generation
Alabdulrahman, M., Khayyat, R., Almowallad, K., & Alharz, Z. (2024) · International Conference on Computer and Automation Engineering · doi:10.1109/iccae59995.2024.10569190 · Cited by 4 · see citing works
“Generative models have also been used in art generation, such as poetry and fiction stories [2].”
BackgroundMethod$$\sigma $$-GPTs: A New Approach to Autoregressive Models
Pannatier, A., Courdier, E., & Fleuret, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70368-3_9 · Cited by 3 · see citing works
Generating Illustrated Instructions
Menon, S., Misra, I., & Girdhar, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00600 · Cited by 3 · see citing works
“Recent work has examined text-conditioned visual generation through autogressive [8, 69] or diffusion models [4, 18, 21, 43, 48, 49, 51].”
BackgroundMaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
Ma, H., Mahdizadehaghdam, S., Wu, B., Fan, Z., Gu, Y., Zhao, W., Shapira, L., & Xie, X. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00707 · Cited by 3 · see citing works
“Muse [11] achieves state-of-the-art performance in text-to-image generation by training on large-scale text-image datasets and brings significantly efficiency improvement.”
InfluentialRiverine flood hazard map prediction by neural networks
Yin, Z., & León, A. S. (2024) · HydroResearch · doi:10.1016/j.hydres.2024.10.003 · Cited by 3 · see citing works
- Authors
- Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., Li, Y., & Krishnan, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.00704
- OpenAlex
- W4313484371
- Cited by
- 157
- References
- 74
- Citations by year
- 2025: 25 · 2024: 92 · 2023: 40
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- text-to-image generation, image inpainting, parallel decoding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
