CoChat
Sign Up
PreprintCited by 15Generative Adversarial Networks and Image SynthesisAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

High-Resolution Complex Scene Synthesis with Transformers

Open access
Jahn, Manuel, Rombach, Robin, Ommer, Björn·2021·arXiv
DOI 10.48550/arxiv.2105.06458arXiv 2105.06458
Open this paper in CoChat — free accountOpen PDF

Abstract

The use of coarse-grained layouts for controllable synthesis of complex scene images via deep generative models has recently gained popularity. However, results of current approaches still fall short of their promise of high-resolution synthesis. We hypothesize that this is mostly due to the highly engineered nature of these approaches which often rely on auxiliary losses and intermediate steps such as mask generators. In this note, we present an orthogonal approach to this task, where the generative model is based on pure likelihood training without additional objectives. To do so, we first optimize a powerful compression model with adversarial training which learns to reconstruct its inputs via a discrete latent bottleneck and thereby effectively strips the latent representation of high-frequency details such as texture. Subsequently, we train an autoregressive transformer model to learn the distribution of the discrete image representations conditioned on a tokenized version of the layouts. Our experiments show that the resulting system is able to synthesize high-quality images consistent with the given layouts. In particular, we improve the state-of-the-art FID score on COCO-Stuff and on Visual Genome by up to 19% and 53% and demonstrate the synthesis of images up to 512 x 512 px on COCO and Open Images.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2105.06458

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our OpenImages model surpasses the results of Jahn et al [37] by a margin of nearly 11 in terms of FID.

    “Our OpenImages model surpasses the results of Jahn et al [37] by a margin of nearly 11 in terms of FID.”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [37]

  • For layout-to-image synthesis on COCO, the exact same samples as in [37] are used to obtain better comparability.

    “To obtain better comparability, we use the exact same samples as in [37].”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [2][2] Oron Ashual and Lior Wolf. Specifying Object Attributes and Relations in Interactive Scene Generation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019 , pages 4560-4568. IEEE, 2019. 2 -
  2. [3][3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015 , 2015. 2 -
  3. [4][4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In 7th International Conference on Learning Representations, ICLR , 2019. 2 -
  4. [5][5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCOStuff: Thing and Stuff Classes in Context. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018 , pages 1209-1218. IEEE Computer Society, 2018. 1, 3, 7 -
  5. [7][7] Patrick Esser, Robin Rombach, and Bj¨ orn Ommer. Taming transformers for high-resolution image synthesis. arXiv:2012.09841 [cs] , 02 2021. 2, 3, 4, 7 - · arXiv
  6. [8][8] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.14701 [cs] , 11 2020. 2, 4, 7 - · arXiv
  7. [9][9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 , pages 66266637, 2017. 4 -
  8. [10][10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020. 4 -
  9. [11][11] Maor Ivgi, Yaniv Benny, Avichai Ben-David, Jonathan Berant, and Lior Wolf. Scene Graph to Image Generation with Contextualized Object Layout Refinement. CoRR , abs/2009.10939, 2020. 2 - · arXiv
  10. [12][12] Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Representations, ICLR 2017 . OpenReview.net, 2017. 2 -
  11. [13][13] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image Generation from Scene Graphs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018 , pages 1219-1228. IEEE Computer Society, 2018. 2, 7 -
  12. [15][15] Tero Karras, Samuli Laine, and Timo Aila. A StyleBased Generator Architecture for Generative Adversarial Networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019 , pages 4401-4410. Computer Vision Foundation / IEEE, 2019. 2 -
  13. [17][17] Yann LeCun, Yoshua Bengio, et al. Convolutional Networks for Images, Speech, and Time Series. The Handbook of Brain Theory and Neural Networks , 3361(10):1995, 1995. 2 -
  14. [19][19] Ankur P. Parikh, Oscar T¨ ackstr¨ om, Dipanjan Das, and Jakob Uszkoreit. A Decomposable Attention Model for Natural Language Inference. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016 , pages 2249-2255. The Association for Computational Linguistics, 2016. 2 -
  15. [20][20] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic Image Synthesis with Spatially-Adaptive Normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR , 2019. 2, 4 -
  16. [21][21] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic Image Synthesis with Spatially-Adaptive Normalization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019 , pages 2337-2346. Computer Vision Foundation / IEEE, 2019. 2 -
  17. [23][23] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. arXiv:2102.12092 [cs] , 02 2021. 2, 3, 7 - · arXiv
  18. [24][24] Ali Razavi, A¨ aron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQ-VAE-2. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch´ e-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 , pages 14837-14847, 2019. 2, 3 -
  19. [25][25] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. 2, 3 -
  20. [26][26] Wei Sun and Tianfu Wu. Image Synthesis From Reconfigurable Layout and Style. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019 , pages 1053010539. IEEE, 2019. 2, 4 -
  21. [27][27] Wei Sun and Tianfu Wu. Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis. arXiv:2003.11571 [cs] , 03 2020. 2, 3, 4, 11 - · arXiv
  22. [28][28] Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R. Devon Hjelm, and Shikhar Sharma. Object-Centric Image Generation from Layouts. arXiv:2003.07449 [cs, eess] , 12 2020. 2, 4, 7, 10 - · arXiv
  23. [29][29] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep Image Prior. arXiv:1711.10925 , 2017. 2 - · arXiv
  24. [31][31] A¨ aron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 , pages 6306-6315, 2017. 2, 3 -
  25. [32][32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, NeurIPS , 2017. 2 -
  26. [33][33] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. 2, 4 -
  27. [35][35] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018. 2

Cited by

Sorted by citations

Metadata

Authors
Jahn, M., Rombach, R., & Ommer, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2105.06458
arXiv
2105.06458
OpenAlex
W3162003367
Cited by
15
References
2
Citations by year
2025: 2 · 2024: 4 · 2023: 7 · 2022: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
high-resolution image generation, adversarial training, discrete latent representations, autoregressive transformers, controllable image generation, COCO-Stuff, Visual Genome

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review