PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Open accessAbstract
The most advanced text-to-image (T2I) models require significant training costs (e.g., millions of GPU hours), seriously hindering the fundamental innovation for the AIGC community while increasing CO2 emissions. This paper introduces PIXART-$α$, a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), reaching near-commercial application standards. Additionally, it supports high-resolution image synthesis up to 1024px resolution with low training cost, as shown in Figure 1 and 2. To achieve this goal, three core designs are proposed: (1) Training strategy decomposition: We devise three distinct training steps that separately optimize pixel dependency, text-image alignment, and image aesthetic quality; (2) Efficient T2I Transformer: We incorporate cross-attention modules into Diffusion Transformer (DiT) to inject text conditions and streamline the computation-intensive class-condition branch; (3) High-informative data: We emphasize the significance of concept density in text-image pairs and leverage a large Vision-Language model to auto-label dense pseudo-captions to assist text-image alignment learning. As a result, PIXART-$α$'s training speed markedly surpasses existing large-scale T2I models, e.g., PIXART-$α$ only takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU days), saving nearly \$300,000 (\$26,000 vs. \$320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training cost is merely 1%. Extensive experiments demonstrate that PIXART-$α$ excels in image quality, artistry, and semantic control. We hope PIXART-$α$ will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1, Muse, and PixArt-α, presents a viable solution.
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [164]
Timestep t is injected into the model using adaLN-single.
“Timestep t is injected into the model using adaLN-single[7].”
From Manga Generation via Layout-controllable Diffusion (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“A (Vaswani et al., 2017) have achieved great success in language models (Radford et al., 2018; 2019), and many recent works (Dosovitskiy et al., 2020a; He et al., 2022) show it is also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et…”
BackgroundU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 3× · see citing works
“…success of Transformer architecture on many computer vision tasks, Diffusion Trans-former (DiT) (Peebles & Xie, 2023) and its variant (Bao et al., 2023; Zheng et al., 2023) further replace the Convolutional-based U-Net (Ronneberger et al., 2015) backbone with Transformers for increased scalability.”
BackgroundMethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3× · see citing works
“As shown in Table 4, among the 256 × 256 phases, our model primarily focuses on the text-to-image alignment stage, with less time on fine-tuning and only 1/8 of that time spent on ImageNet pixel dependency.”
MethodInfluentialMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 2× · see citing works
“We comprehensively evaluate P IX A RT - α via three primary metrics, i.e ., Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) on MSCOCO dataset (Lin et al., 2014), compositionality on T2I-CompBench (Huang et al., 2023), and human-preference rate on user study.”
BackgroundMethodSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1× · see citing works
“…computer vision tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021;…”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · cited 2× · see citing works
“A (Vaswani et al., 2017) have achieved great success in language models (Radford et al., 2018; 2019), and many recent works (Dosovitskiy et al., 2020a; He et al., 2022) show it is also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et…”
BackgroundMethodEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 1× · see citing works
“…et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al.,…”
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · cited 9× · see citing works
“Stable diffusion) (Rombach et al., 2022) is a recent advancement in diffusion models.”
BackgroundMethodInfluentialKirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · cited 6× · see citing works
“More image-text pair samples on the SAM dataset are shown in Figure 11.”
MethodInfluentialDenoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · cited 2× · see citing works
“DDPMs (Ho et al., 2020; Sohl-Dickstein et al., 2015) have emerged as highly successful approaches for image generation, which employs an iterative denoising process to transform Gaussian noise into an image.”
BackgroundInfluentialMasked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 2× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundVariational Inference with Normalizing FlowsPreprint
Rezende, D. J., & Mohamed, S. (2015) · International Conference on Machine Learning · doi:10.48550/arxiv.1505.05770 · Cited by 5,748 · cited 2× · see citing works
“…et al., 2015) and score-based generative models (Song & Ermon, 2019; Song et al., 2021) have emerged as highly successful approaches for image generation, surpassing previous generative models such as GANs (Goodfellow et al., 2014), VAEs (Kingma & Welling, 2013), and Flow (Rezende & Mohamed, 2015).”
MethodInfluentialPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 3× · see citing works
“…tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021)…”
BackgroundMethodSelf-citationMeigs, J. (1954) · The Yale Journal of Biology and Medicine · doi:10.1515/cclm.1994.32.8.631 · Cited by 5,224
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090 · see citing works
Self-citationAdding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547 · cited 5× · see citing works
“Following the general design of ControlNet (Zhang et al., 2023), we freeze each DiT Block and create a trainable copy, augmenting with two zero linear layers before and after it.”
MethodInfluentialRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 2× · see citing works
“…2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et…”
BackgroundDiffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362 · see citing works
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 StepsPreprint
Lu, C., Zhou, Y., Bao, F., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.00927 · Cited by 2,705 · cited 3× · see citing works
“In this study, we incorporated three sampling algorithms, namely iD-DPM (Nichol & Dhariwal, 2021), DPM-Solver (Lu et al., 2022), and SA-Solver (Xue et al., 2023a).”
BackgroundMethodSelf-citationDeep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · cited 3× · see citing works
“DDPMs (Ho et al., 2020; Sohl-Dickstein et al., 2015) have emerged as highly successful approaches for image generation, which employs an iterative denoising process to transform Gaussian noise into an image.”
BackgroundMethodInfluentialPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 1× · see citing works
“Additionally, Stable diffusion and its variants have been effectively combined with various low-cost fine-tuning (Hu et al., 2021; Xie et al., 2023) and customization (Zhang et al.”
BackgroundFilm: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · cited 1× · see citing works
“We find that the linear projections in the adaptive normalization layers (Perez et al., 2018) ( adaLN ) module of the DiT account for a substantial proportion (27”
BackgroundPvt v2: Improved baselines with pyramid vision transformer
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0274-8 · Cited by 2,439 · see citing works
Self-citationTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 1× · see citing works
“…2022) show it is also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic…”
BackgroundDreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 3× · see citing works
“We extend the capabilities of P IX A RT - α by incorporating two commonly used customization methods: DreamBooth (Ruiz et al., 2022) and ControlNet (Zhang et al., 2023).”
MethodLiu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294 · cited 1× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundScalable Diffusion Models with Transformers
Peebles, W., & Xie, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00387 · Cited by 2,284 · cited 4× · see citing works
“We follow Imagen (Saharia et al., 2022) and DeepFloyd (DeepFloyd, 2023) to employ the T5 large language model ( i.e ., 4.3B Flan-T5-XXL) as the text encoder for conditional feature extraction, and use DiT-XL/2 (Peebles & Xie, 2023) as our base network architecture.”
BackgroundMethodInfluentialSegmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R. A., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,254 · cited 1× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundScore-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953 · cited 1× · see citing works
“…PROBABILISTIC MODELS Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) and score-based generative models (Song & Ermon, 2019; Song et al., 2021) have emerged as highly successful approaches for image generation, surpassing previous generative models such as GANs (Goodfellow…”
MethodMou, C., Wang, X., Xie, L.-B., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2302.08453 · Cited by 1,907 · cited 1×
“, 2023) and customization (Zhang et al., 2023; Mou et al., 2023) technologies.”
Self-citationImproved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550 · cited 1× · see citing works
“In this study, we incorporated three sampling algorithms, namely iD-DPM (Nichol & Dhariwal, 2021), DPM-Solver (Lu et al., 2022), and SA-Solver (Xue et al., 2023a).”
MethodGenerative Modeling by Estimating Gradients of the Data Distribution
Song, Y., & Ermon, S. (2019) · arXiv · doi:10.48550/arxiv.1907.05600 · Cited by 1,453 · cited 1× · see citing works
“…A.1.1 D ENOISING DIFFUSION PROBABILISTIC MODELS Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) and score-based generative models (Song & Ermon, 2019; Song et al., 2021) have emerged as highly successful approaches for image generation, surpassing previous generative models such…”
MethodInfluentialLi, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Yu, Q., & Dai, J. (2022) · arXiv · doi:10.48550/arxiv.2203.17270 · Cited by 1,419 · cited 1× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundSelf-citationHan, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394 · cited 1× · see citing works
“…also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng…”
BackgroundPick-a-Pic: An Open Dataset of User Preferences for Text-to-Image GenerationPreprint
Kirstain, Y., Polyak, A., Singer, U., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.01569 · Cited by 1,198 · cited 1× · see citing works
“Recent studies such as SDXL (Podell et al., 2023) and Pick-a-pic (Kirstain et al., 2023) have presented evidence suggesting that the COCO zero-shot FID is negatively correlated with visual aesthetics.”
BackgroundClassifier-Free Diffusion Guidance
Ho, J., & Salimans, T. (2022) · arXiv · doi:10.48550/arxiv.2207.12598 · Cited by 1,126 · cited 1× · see citing works
“In Figure 20b and 20c, we demonstrate the corresponding T2ICompBench scores across a range of classifier-free guidance (cfg) (Ho & Salimans, 2022) scales.”
MethodImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · cited 2× · see citing works
“2021), effective modality fusion (Girdhar et al., 2023), and scalability (Peebles & Xie, 2023).”
BackgroundLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 4× · see citing works
“To address them, we propose an auto-labeling pipeline utilizing the state-of-the-art vision-language model (LLaVA (Liu et al., 2023)) to generate captions on the SAM (Kirillov et al., 2023).”
MethodInfluentialHolistically-nested edge detection
Xie, S., & Tu, Z. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1004-z · Cited by 682 · cited 1× · see citing works
“We trained the ControlNet on HED (Xie & Tu, 2015) signals using a learning rate of 5e-6 for 20,000 steps.”
MethodTransTrack: Multiple Object Tracking with TransformerPreprint
Sun, P., Cao, J., Jiang, Y., Zhang, R., Xie, E., Yuan, Z., Wang, C., & Luo, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15460 · Cited by 640 · cited 1× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundSelf-citationTune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., & Shou, M. Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00701 · Cited by 622 · cited 1× · see citing works
“…et al., 2022), and Stable Diffusion (Rombach et al., 2022) has started a new era of photorealistic image synthesis, profoundly impacting numerous downstream applications, such as image editing (Kim et al., 2022), video generation (Wu et al., 2022), 3D assets creation (Poole et al., 2022), etc .”
BackgroundDiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation
Kim, G., Kwon, T., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00246 · Cited by 570 · cited 2× · see citing works
“, 2022) has started a new era of photorealistic image synthesis, profoundly impacting numerous downstream applications, such as image editing (Kim et al., 2022), video generation (Wu et al.”
BackgroundDreamFusion: Text-to-3D using 2D DiffusionPreprint
Poole, B., Jain, A., Barron, J. T., & Mildenhall, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14988 · Cited by 475 · cited 1× · see citing works
“…et al., 2022), and Stable Diffusion (Rombach et al., 2022) has started a new era of photorealistic image synthesis, profoundly impacting numerous downstream applications, such as image editing (Kim et al., 2022), video generation (Wu et al., 2022), 3D assets creation (Poole et al., 2022), etc .”
BackgroundSDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., & Rombach, R. (2023) · arXiv · doi:10.48550/arxiv.2307.01952 · Cited by 455 · cited 6× · see citing works
“We also employ multi-aspect augmentation introduced in SDXL (Podell et al., 2023) to enable arbitrary aspect image generation.”
BackgroundMethodInfluentialScaling up GANs for Text-to-Image Synthesis
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00976 · Cited by 426 · cited 1× · see citing works
“The baseline data is sourced from GigaGAN (Kang et al., 2023).”
MethodDeepViT: Towards Deeper Vision Transformer
Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., & Feng, J. (2021) · arXiv · doi:10.48550/arxiv.2103.11886 · Cited by 398 · cited 2× · see citing works
“, 2022) show it is also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al.”
BackgroundLAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 4× · see citing works
“Our investigation reveals notable shortcomings in existing text-image pair datasets, exemplified by LAION (Schuhmann et al., 2021), where textual captions often suffer from a lack of informative content ( i.e ., typically describing only a partial of objects in the images) and a severe long-tail effect ( i.e ., with a…”
BackgroundInfluentialEngel, N., Belagiannis, V., & Dietmayer, K. (2021) · IEEE Access · doi:10.1109/access.2021.3116304 · Cited by 304 · cited 1× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundAll are Worth Words: A ViT Backbone for Diffusion Models
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., & Zhu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02171 · Cited by 296 · cited 2× · see citing works
“The Diffusion Transformer (DiT) (Peebles & Xie, 2023) and its variant (Bao et al., 2023; Zheng et al., 2023) follow the step to further replace the Convolutional-based U-Net (Ronneberger et al., 2015) backbone with Transformers.”
BackgroundMethodeDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert DenoisersPreprint
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.01324 · Cited by 253 · see citing works
JourneyDB: A Benchmark for Generative Image Understanding
Pan, J., Sun, K., Ge, Y.-Y., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2307.00716 · Cited by 242 · cited 2×
“Note that HQ (High Quality) dataset here includes 4M JourneyDB (Pan et al., 2023) and 10M internal data.”
BackgroundMethodRAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion PathsPreprint
Xue, Z., Song, G.-L., Guo, Q., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.18295 · Cited by 202 · cited 8× · see citing works
“…photorealisticquality with cartoon vibes Oppenheimer sits on the beach on a chair, watching a nuclear exposition with a huge mushroom cloud, 120mm $320,000, and the recent larger model, RAPHAEL (Xue et al., 2023b), even costs 60K A100 GPU days – requiring around $3,080,000, as detailed in Table 2.”
BackgroundMethodInfluentialSelf-citationEstimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2022) · arXiv · doi:10.48550/arxiv.2211.02001 · Cited by 192 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · cited 1× · see citing works
“…works (Dosovitskiy et al., 2020a; He et al., 2022) show it is also a promising architecture on many computer vision tasks like image classification (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et…”
BackgroundPanoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Li, Z., Wang, W., Xie, E., Yu, Z., Anandkumar, A., Álvarez, J. M., Luo, P., & Lü, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00134 · Cited by 166 · cited 1× · see citing works
“…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
BackgroundSelf-citationUnderstanding The Robustness in Vision Transformers
Zhou, D., Yu, Z., Xie, E., Xiao, C., Anandkumar, A., Feng, J., & Álvarez, J. M. (2022) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2204.12451 · Cited by 122 · cited 1× · see citing works
“U-N ET The Transformer-based network’s superiority over convolutional networks has been widely established in various studies, showcasing attributes such as robustness (Zhou et al., 2022; Xie et al., 2021), effective modality fusion (Girdhar et al., 2023), and scalability (Peebles & Xie, 2023).”
BackgroundSelf-citationFeng, Z., Zhang, Z., Yu, X., Fang, Y., Li, L., Chen, X., Lu, Y., Liu, J., Yin, W., Feng, S., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00977 · Cited by 98 · cited 5× · see citing works
“Following the general design of ControlNet (Zhang et al., 2023), we freeze each DiT Block and create a trainable copy, augmenting with two zero linear layers before and after it.”
MethodInfluentialSA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models
Xue, S., Yi, M.-Y., Luo, W., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2309.05019 · Cited by 91 · cited 1×
“In this study, we incorporated three sampling algorithms, namely iD-DPM (Nichol & Dhariwal, 2021), DPM-Solver (Lu et al., 2022), and SA-Solver (Xue et al., 2023a).”
MethodSelf-citationT2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation
Huang, K., Duan, C., Sun, K., Xie, E., Li, Z., & Liu, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3531907 · Cited by 74 · cited 5× · see citing works
“Beyond the above evaluation, we also assess the alignment between the generated images and text condition using T2I-Compbench (Huang et al., 2023), a comprehensive benchmark for evaluating the compositional text-to-image generation capability.”
BackgroundMethodInfluentialSelf-citationMetaBEV: Solving Sensor Failures for 3D Detection and Map Segmentation
Ge, C., Chen, J. S., Xie, E., Wang, Z., Hong, L., Lu, H., Li, Z., & Luo, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00801 · Cited by 73 · cited 2× · see citing works
“…(Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han et al., 2021), object detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al.,…”
BackgroundSelf-citationA Study on the Evaluation of Generative Models
Betzalel, E., Penso, C., Navon, A., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2206.10935 · Cited by 66
Xie, E., Yao, L., Shi, H., Liu, Z., Zhou, D., Liu, Z., Li, J., & Li, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00390 · Cited by 62 · cited 3× · see citing works
“We adopt the positional encoding trick in DiffFit (Xie et al., 2023) since the image resolution and aspect change during different training stages.”
BackgroundMethodSelf-citationFast Training of Diffusion Models with Masked TransformersPreprint
Zheng, H., Nie, W., Vahdat, A., & Anandkumar, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.09305 · Cited by 10 · cited 4× · see citing works
“…success of Transformer architecture on many computer vision tasks, Diffusion Trans-former (DiT) (Peebles & Xie, 2023) and its variant (Bao et al., 2023; Zheng et al., 2023) further replace the Convolutional-based U-Net (Ronneberger et al., 2015) backbone with Transformers for increased scalability.”
BackgroundMethodInfluential
Flagged first, then by citations
Research works printed but not linked: 6
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv , 2020b.
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.
- Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In arXiv , 2013.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In arXiv , 2017.
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR , 2022.
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
GenTron: Diffusion Transformers for Image and Video Generation
Chen, S., Xu, M., Ren, J., Cong, Y., He, S., Xie, Y., Sinha, A., Luo, P., Xiang, T., & Pérez-Rúa, J.-M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00616 · Cited by 48 · see citing works
“We mainly adopt the recent T2I-CompBench [31] to compare GenTron model variants, following [4, 10].”
MethodSelf-citationLossy Image Compression with Foundation Diffusion Models
Relic, L., Azevedo, R. G. D. A., Groß, M. H., & Schroers, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73030-6_17 · Cited by 20 · see citing works
Deepfakes on Demand: The rise of accessible non-consensual deepfake image generators
Hawkins, W., Mittelstadt, B., & Russell, C. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732107 · Cited by 20 · see citing works
ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models
Patel, M., Gokhale, T., Baral, C. R., & Yang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29371 · Cited by 7 · see citing works
Few-shot exemplar-driven inpainting with parameter-efficient diffusion fine-tuning
Yang, S., Gu, Z., Hao, W., Wang, Y., Cai, H., & Chen, X. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2400395 · Cited by 3 · see citing works
Improving Text Generation on Images with Synthetic Captions
Koh, J., Park, S., & Song, J. (2024) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiai-aai63651.2024.00121 · Cited by 3 · see citing works
“(3) We measure the impact of using synthetic captions on generating text displayed in an image by experimenting with varying increments of data and different types of synthetic data (random and detailed).”
BackgroundRegionDiffusion: Generative data augmentation for object detection with diffusion models
Li, K. Q.-Q., Luo, S., & Tseng, K.-K. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131115 · Cited by 4 · see citing works
Video Diffusion Models: A Survey
Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., & Ritter, H. (2024) · arXiv · doi:10.48550/arxiv.2405.03150 · Cited by 4 · see citing works
Manga Generation via Layout-controllable Diffusion
Chen, S., Li, D., Bao, Z., Zhou, Y., Tan, L., Zhong, Y., & Zhao, Z. (2024) · arXiv · doi:10.48550/arxiv.2412.19303
OmniGen2: Towards Instruction-Aligned Multimodal Generation
Wu, C., Zheng, P., Yan, R., Xiao, S., Luo, X., Wang, Y., Li, W., Jiang, X., Liu, Y., Zhou, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.18871
Self-citation
- Authors
- Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., & Li, Z.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.00426
- OpenAlex
- W7051896053
- Cited by
- 30
- References
- 64
- Citations by year
- 2025: 18 · 2024: 12
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- text-to-image generation, diffusion transformer, efficient training, training cost reduction, CO2 emission reduction, cross-attention, image-text alignment, high-resolution image generation, semantic control, generative AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
