CoChat
Sign Up
Cited by 35Generative Adversarial Networks and Image SynthesisHandwritten Text Recognition TechniquesGraphic Design and Typography

LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models

Open access
Zhang, Junyi, Guo, Jiaqi, Sun, Shizhao, Lou, Jian-Guang, Zhang, Dongmei·2023·2023 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv51070.2023.00664
Open this paper in CoChat — free accountOpen PDF

Abstract

Creating graphic layouts is a fundamental step in graphic designs. In this work, we present a novel generative model named LayoutDiffusion for automatic layout generation. As layout is typically represented as a sequence of discrete tokens, LayoutDiffusion models layout generation as a discrete denoising diffusion process. It learns to reverse a mild forward process, in which layouts become increasingly chaotic with the growth of forward steps and layouts in the neighboring steps do not differ too much. Designing such a mild forward process is however very challenging as layout has both categorical attributes and ordinal attributes. To tackle the challenge, we summarize three critical factors for achieving a mild forward process for the layout, i.e., legality, coordinate proximity and type disruption. Based on the factors, we propose a block-wise transition matrix coupled with a piece-wise linear noise schedule. Experiments on RICO and PubLayNet datasets show that LayoutDiffusion outperforms state-of-the-art approaches significantly. Moreover, it enables two conditional layout generation tasks in a plug-and-play manner without re-training and achieves better performance than existing methods. Project page: https://layoutdiffusion.github.io.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv51070.2023.00664

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In design, AI has been integrated into product design workflows, fostering innovation in idea generation and advancing user interface design through adaptability.

    “In design, AI has been integrated into product design workflows, fostering innovation in idea generation [22, 39] and advancing user interface design through features such as adaptability [21, 109].”

    From Understanding Screenwriters' Practices, Attitudes, and Future Expectations in Human-AI Co-Creation (2025) · cites this paper as [109]

    Section 5.1 uses a dataset of mobile-app user interface designs, and Section 4.3 shows that LayoutDiffusion can refine user-provided layouts and generate layouts for specified element types. This supports flexible AI-assisted UI layout design, but the paper does not establish that the resulting interfaces themselves are adaptable. The product-design and idea-generation clause is attached to different citations. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 13642-13652, 2021. 1, 2, 3, 5
  2. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tar- low, and Rianne van den Berg. Structured denoising dif- fusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993, 2021. 1, 2, 3, 4, 5, 6, 8, 13, 14
  3. Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022. 3 · arXiv
  4. Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A con- tinuous time framework for discrete denoising models. arXiv preprint arXiv:2205.14987, 2022. 3 · arXiv
  5. Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022. 3 · arXiv
  6. Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hib- schman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ran- jitha Kumar. Rico: A mobile app dataset for building data- driven design applications. In Proceedings of the 30th An- nual ACM Symposium on User Interface Software and Tech- nology, pages 845-854, 2017. 5, 13
  7. Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander G Schwing, and David Forsyth. Fast, diverse and accurate im- age captioning guided by part-of-speech. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10695-10704, 2019. 15
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 13 · arXiv
  9. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Aus- tralia, July 2018. Association for Computational Linguistics. 17
  10. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, 2019. 1
  11. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324, 2019. 3 · arXiv
  12. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022. 3 · arXiv
  13. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vec- tor quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 10696-10706, 2022. 3, 4, 5
  14. Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layout- transformer: Layout generation and completion with self- attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1004-1014, 2021. 1, 2, 3, 5
  15. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 1 · arXiv
  16. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 1
  17. Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 9 · arXiv
  18. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, abs/2102.05379, 2021. 3 · arXiv
  19. Zhaoyun Jiang, Jiaqi Guo, Shizhao Sun, Huayu Deng, Zhongkai Wu, Vuksan Mijovic, Zijiang James Yang, Jian- Guang Lou, and Dongmei Zhang. Layoutformer++: Con- ditional graphic layout generation via constraint serializa- tion and decoding space restriction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18403-18412, 2023. 1, 2, 5, 6, 14, 17
  20. Zhaoyun Jiang, Shizhao Sun, Jihua Zhu, Jian-Guang Lou, and Dongmei Zhang. Coarse-to-fine generative modeling for graphic layouts. Proceedings of the AAAI Conference on Ar- tificial Intelligence, 36(1):1096-1103, Jun. 2022. 1, 2, 3, 5
  21. Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Si- gal, and Greg Mori. Layoutvae: Stochastic scene layout gen- eration from a label set. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9895- 9904, 2019. 2
  22. Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Constrained graphic layout generation via latent optimization. In Proceedings of the 29th ACM International Conference on Multimedia, pages 88-96, 2021. 2, 5, 6, 14
  23. Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: Bidirectional lay- out transformer for controllable layout generation. arXiv preprint arXiv:2112.05112, 2021. 1, 2, 3, 5, 14 · arXiv
  24. Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020. 1 · arXiv
  25. Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. Neural de- sign network: Graphic layout generation with constraints. In European Conference on Computer Vision, pages 491-506. Springer, 2020. 1, 3, 5, 6
  26. Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. In International Conference on Learning Representations, 2018. 2
  27. Jianan Li, Jimei Yang, Jianming Zhang, Chang Liu, Christina Wang, and Tingfa Xu. Attribute-conditioned lay- out gan for automatic graphic design. IEEE Transactions on Visualization and Computer Graphics, 27(10):4039-4048, 2020. 3
  28. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-lm im- proves controllable text generation. arXiv preprint arXiv:2205.14217, 2022. 3, 6, 8, 13 · arXiv
  29. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6, 13 · arXiv
  30. David D Nguyen, Surya Nepal, and Salil S Kanhere. Diverse multimedia layout generation with multi choice learning. In Proceedings of the 29th ACM International Conference on Multimedia, pages 218-226, 2021. 3
  31. Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021. 9 · arXiv
  32. Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021. 2, 6
  33. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.09254, 2017. 13 · arXiv
  34. Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. Designscape: Design with interactive layout suggestions. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, pages 1221-1224, 2015. 2
  35. Peter O'Donovan, Aseem Agarwala, and Aaron Hertz- mann. Learning layouts for single-pagegraphic designs. IEEE transactions on visualization and computer graphics, 20(8):1200-1213, 2014. 2
  36. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems, 32, 2019. 14
  37. Soliha Rahman, Vinoth Pandian Sermuga Pandian, and Matthias Jarke. Ruite: Refining ui layout aesthetics using transformer encoder. In 26th International Conference on Intelligent User Interfaces-Companion, pages 81-83, 2021. 2, 3, 5, 6, 14, 17
  38. Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1- 10, 2022. 1
  39. Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022. 1 · arXiv
  40. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Confer- ence on Machine Learning, pages 2256-2265. PMLR, 2015. 1, 5, 13
  41. Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint arXiv:2205.16007, 2022. 3 · arXiv
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1, 2, 5
  43. Pan Xie, Qipeng Zhang, Zexian Li, Hao Tang, Yao Du, and Xiaohui Hu. Vector quantized diffusion model with codeunet for text-to-sign pose sequences generation. arXiv preprint arXiv:2208.09141, 2022. 3 · arXiv
  44. Kota Yamaguchi. Canvasvae: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5481-5489, 2021. 2, 3
  45. Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Goj- cic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: La- tent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978, 2022. 1 · arXiv
  46. Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG), 38(4):1-15, 2019.
  47. Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Pub- laynet: largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1015-1022. IEEE, 2019. 5, 13
  48. Min Zhou, Chenchen Xu, Ye Ma, Tiezheng Ge, Yuning Jiang, and Weiwei Xu. Composition-aware graphic layout gan for visual-textual presentation designs. arXiv preprint arXiv:2205.00303, 2022. 3 · arXiv
  49. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmark- ing platform for text generation models. In The 41st Interna- tional ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097-1100, 2018. 15

Cited by

Sorted by citations

Authors
Zhang, J., Guo, J., Sun, S., Lou, J.-G., & Zhang, D.
Affiliations
  • Shanghai Jiao Tong University (CN)
  • Microsoft Research Asia (China) (CN)
ORCID
Venue
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2023-10-01
Pages
7192-7202
DOI
10.1109/iccv51070.2023.00664
OpenAlex
W4390873073
Cited by
35
References
4
Citations by year
2026: 7 · 2025: 13 · 2024: 13 · 2023: 2
FWCI
3.74
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Graphic Design and Typography
Keywords
denoising diffusion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review