CoChat
Sign Up
PreprintCited by 151Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsAdversarial Robustness in Machine Learning

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Open access
Esser, Patrick, Kulal, Sumith, Blattmann, Andreas, Entezari, Rahim, Müller, Jonas, Saini, Harry, Levi, Yam, Lorenz, Dominik +9 more·2024·arXiv
DOI 10.48550/arxiv.2403.03206arXiv 2403.03206
Open this paper in CoChat — free accountOpen PDF

Abstract

Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.03206

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Flow matching models have demonstrated improvements in both performance and efficiency, outperforming diffusion-based models in many tasks.

    “flow matching models have demonstrated improvements in both performance and efficiency, outperforming diffusion-based models in many tasks.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Esser et al., 2024

    The Abstract reports superior performance for the paper’s rectified-flow approach over established diffusion formulations in high-resolution text-to-image synthesis. Figure 3 reports better sample efficiency with fewer sampling steps, but the paper does not establish that flow-matching models outperform diffusion models across many tasks. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • MoCha is built on the pretrained text-to-video latent diffusion model using the Rectified Flow framework.

    “MoCha builds on the pretrained text-to-video latent diffusion model using the Rectified Flow framework [6].”

    From MoCha: End-to-End Video Character Replacement without Structural Guidance (2026) · cites this paper as [6]

    The Introduction describes the paper’s rectified-flow formulation, and Section 4 describes a latent-space diffusion architecture. Both concern text-to-image synthesis. Our copy does not establish that MoCha builds on a pretrained text-to-video model or uses that model with rectified flow. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Esser et al. propose sampling the flow matching timestep from a logit-normal distribution that emphasizes the middle timesteps.

    “Esser et al. [14] instead propose sampling from a logit-normal distribution that emphasizes the middle timesteps”

    From $π_0$: A Vision-Language-Action Flow Model for General Robot Control (2024) · cites this paper as [14]

  • The logit-normal distribution places higher probability mass in the middle range of [0, 1], as shown in Esser et al. (2024).

    “We draw training timesteps from a logit-normal distribution, which, as shown in (Esser et al., 2024), places higher probability mass in the middle range of [0, 1].”

    From World Simulation with Video Foundation Models for Physical AI (2025) · cites this paper as Esser et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 38

  1. Atchison, J. and Shen, S. M. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261–272, 1980. autofaiss. autofaiss, 2023. URL https://github.c om/criteo/autofaiss.
  2. Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3), 2023.
  3. Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023a. · arXiv
  4. Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392–18402, 2023.
  5. USENIX Security Symposium (USENIX Security 23), pp. 5253–5270, 2023.
  6. Changpinyo, S., Sharma, P. K., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. 2021 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 3557–3567, 2021. URL https://api.semanticscholar.org/Corp usID:231951742.
  7. Chen, T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. In Neural Information Processing Systems, 2018. URL https: //api.semanticscholar.org/CorpusID:49 310446.
  8. Dockhorn, T., Vahdat, A., and Kreis, K. Score-based gener- ative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068, 2021. · arXiv
  9. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2020.
  10. Fischer, J. S., Gui, M., Ma, P., Stracke, N., Baumann, S. A., and Ommer, B. Boosting latent diffusion with flow match- ing. arXiv preprint arXiv:2312.07360, 2023. · arXiv
  11. Ghosh, D., Hajishirzi, H., and Schmidt, L. Geneval: An object-focused framework for evaluating text-to-image alignment. arXiv preprint arXiv:2310.11513, 2023. · arXiv
  12. Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i- compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350, 2023. · arXiv
  13. Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. ArXiv, abs/2206.00364, 2022. URL https://api.se manticscholar.org/CorpusID:249240415. · arXiv
  14. Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the train- ing dynamics of diffusion models. arXiv preprint arXiv:2312.02696, 2023. · arXiv
  15. Kingma, D. P. and Gao, R. Understanding diffusion ob- jectives as the elbo with simple data augmentation. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023.
  16. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating train- ing data makes language models better. arXiv preprint arXiv:2107.06499, 2021. · arXiv
  17. Lin, S., Liu, B., Li, J., and Yang, X. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5404–5411, 2024.
  18. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common Objects in Context, pp. 740–755. Springer In- ternational Publishing, 2014. ISBN 9783319106021. doi: https://books.google.de/books?id=Vg8OAAAAQAAJ https://books.google.de/books?id=Vg8OAAAAQAAJ http://dx.doi.org/10.18653/v1/2021.emnlp-main.595 http://dx.doi.org/10.18653/v1/2021.emnlp-main.595 http://dx.doi.org/10.18653/v1/2021.emnlp-main.595 https://api.semanticscholar.org/CorpusID:1152227 https://api.semanticscholar.org/CorpusID:1152227 https://api.semanticscholar.org/CorpusID:249240415 https://api.semanticscholar.org/CorpusID:249240415 · DOI · Paper page
  19. Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https://openreview.net /forum?id=PqvMRDCJT9t.
  20. Loshchilov, I. and Hutter, F. Fixing weight decay regular- ization in adam. ArXiv, abs/1711.05101, 2017. URL https://api.semanticscholar.org/Corp usID:3312944. · arXiv
  21. Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., and Douze, M. A self-supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14532–14542, 2022.
  22. Po, R., Yifan, W., Golyanik, V., Aberman, K., Barron, J. T., Bermano, A. H., Chan, E. R., Dekel, T., Holynski, A., Kanazawa, A., et al. State of the art on diffusion models for visual computing. arXiv preprint arXiv:2310.07204, 2023. · arXiv
  23. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Man- ning, C. D., and Finn, C. Direct Preference Optimiza- tion: Your Language Model is Secretly a Reward Model. arXiv:2305.18290, 2023. · arXiv
  24. Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–10, 2022a.
  25. Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2022c.
  26. Sauer, A., Chitta, K., Müller, J., and Geiger, A. Projected gans converge faster. Advances in Neural Information Processing Systems, 2021.
  27. Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023. · arXiv
  28. Sheynin, S., Polyak, A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., and Taigman, Y. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311.10089, 2023. · arXiv
  29. Sohl-Dickstein, J. N., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. ArXiv, abs/1503.03585, 2015. URL https://api.semanticscholar. org/CorpusID:14888175. · arXiv
  30. Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion art or digital forgery? investigat- ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048–6058, 2023a.
  31. Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Understanding and mitigating copying in diffusion models. arXiv preprint arXiv:2305.20086, 2023b. · arXiv
  32. Song, Y., Sohl-Dickstein, J. N., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011.13456, 2020. URL https://api.semant icscholar.org/CorpusID:227209335. · arXiv
  33. Villani, C. Optimal transport: Old and new. 2008. URL https://api.semanticscholar.org/Corp usID:118347220.
  34. Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion Model Alignment Using Direct Preference Optimization. arXiv:2311.12908, 2023. · arXiv
  35. Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. · arXiv
  36. Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. arXiv:2206.10789, 2022. · arXiv
  37. Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In CVPR, pp. 12104–12113, 2022.
  38. Zhang, B. and Sennrich, R. Root mean square layer normal- ization, 2019. https://api.semanticscholar.org/CorpusID:14888175 https://api.semanticscholar.org/CorpusID:14888175 https://api.semanticscholar.org/CorpusID:227209335 https://api.semanticscholar.org/CorpusID:227209335 https://api.semanticscholar.org/CorpusID:118347220 https://api.semanticscholar.org/CorpusID:118347220 https://api.semanticscholar.org/CorpusID:5560643 https://api.semanticscholar.org/CorpusID:5560643

Cited by

Sorted by citations

Related

Authors
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y., & Rombach, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.03206
arXiv
2403.03206
OpenAlex
W4392538976
Cited by
140
References
0
Citations by year
2026: 10 · 2025: 100 · 2024: 30
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
rectified flow, diffusion models, noise sampling, text-to-image generation, high-resolution image generation, transformer models, model scaling, perceptual scales

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review