CoChat
Sign Up
Cited by 47Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsDigital Media and Visual Art

Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models

Open access
Pan, Xichen, Qin, Pengda, Li, Yuhong, Xue, Hui, Chen, Wenhu·2024·2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
DOI 10.1109/wacv57701.2024.00290
Open this paper in CoChat — free accountOpen PDF

Abstract

Conditioned diffusion models have demonstrated state-of-the-art text-to-image synthesis capacity. Recently, most works focus on synthesizing independent images; While for real-world applications, it is common and necessary to generate a series of coherent images for story-stelling. In this work, we mainly focus on story visualization and continuation tasks and propose AR-LDM, a latent diffusion model auto-regressively conditioned on history captions and generated images. Moreover, AR-LDM can generalize to new characters through adaptation. To our best knowledge, this is the first work successfully leveraging diffusion models for coherent visual story synthesizing. It also extends the text-conditioned method to multimodal conditioning. Quantitative results show that AR-LDM achieves SoTA FID scores on PororoSV, FlintstonesSV, and the adopted challenging dataset VIST containing natural images. Large-scale human evaluations show that AR-LDM has superior performance in terms of quality, relevance, and consistency. Code available at this https URL

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/wacv57701.2024.00290

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Manga109 is currently the largest open-source manga dataset.

    “Manga109 is currently the largest opensource manga dataset and it offers rich annotation resources that can be utilized”

    From Manga Generation via Layout-controllable Diffusion (2024) · cites this paper as [34]

    The Datasets section (4.1; page not provided) identifies PororoSV, FlintstonesSV, and VIST as the paper’s testbeds. Searches for Manga109 and manga annotations found no passage addressing whether Manga109 is open-source, the largest manga dataset, or richly annotated. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Manga109 is widely used in the fields of image super-resolution and image restoration.

    “This dataset is widely used in the fields of image super-resolution and image restoration.”

    From Manga Generation via Layout-controllable Diffusion (2024) · cites this paper as [34]

    The cited paper’s §4.1 (Datasets; page unavailable) identifies PororoSV, FlintstonesSV, and VIST as its testbeds, but does not address Manga109. Searches for Manga109, super-resolution, and image restoration found no passage supporting the claimed widespread use. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • The Manga109 dataset includes coordinates of panels, characters, texts, and contents of dialogues.

    “including the coordinates of the bounding boxes for each frame (referred to as panel in this paper), the names and coordinates of each character, the coordinates of each face, and the coordinates and content of each text.”

    From Manga Generation via Layout-controllable Diffusion (2024) · cites this paper as [34]

    Section 4.1 (Datasets; page not provided) describes PororoSV, FlintstonesSV, and VIST, but does not address Manga109 or its annotations. Searches for “Manga109” and for panel, character, face, and text coordinates found no passage supporting the attributed description. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • The Manga109Story dataset is constructed based on the existing Manga109 dataset.

    “We construct the Manga109Story dataset by using the existing Manga109 dataset [34]”

    From Manga Generation via Layout-controllable Diffusion (2024) · cites this paper as [34]

    Section 4.1 (page unavailable) identifies PororoSV, FlintstonesSV, and VIST as the paper’s datasets. Searches for “Manga109Story” and “Manga109” found no passage addressing Manga109 or the construction of Manga109Story, so our copy cannot verify the cited dataset identification. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Manga109 consists of 109 volumes of manga drawn by professional manga artists in Japan, totaling 21,142 pages.

    “Manga109 consists of 109 volumes of manga drawn by professional manga artists in Japan, totaling 21,142 pages.”

    From Manga Generation via Layout-controllable Diffusion (2024) · cites this paper as [34]

    Section 4.1 (Datasets; page unavailable) describes PororoSV, FlintstonesSV, and VIST, not Manga109. Searches for Manga109 and “21,142 pages” found no passage in our copy supporting the claimed number of volumes, artists, or pages. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Al- varo Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Ma- linowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learn- ing, and graph networks. arXiv preprint arXiv:1806.01261, 2018. 8 · arXiv
  2. Hong Chen, Rujun Han, Te-Lin Wu, Hideki Nakayama, and Nanyun Peng. Character-centric story visualization via visual planning and token alignment. arXiv preprint arXiv:2210.08465, 2022. 2, 5 · arXiv
  3. Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmented text-to-image gen- erator. arXiv preprint arXiv:2209.14491, 2022. 1 · arXiv
  4. Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene- based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022. 2 · arXiv
  5. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139-144, 2020.
  6. Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to composi- tions to videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 598-613, 2018. 5
  7. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems, 30, 2017. 5
  8. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 3
  9. Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 3 · arXiv
  10. Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Gir- shick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In 15th Annual Conference of the North American Chapter of the Association for Computational Lin- guistics (NAACL 2016), 2016. 2, 4, 5
  11. Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. Deepstory: Video story qa by deep embedded memory networks. arXiv preprint arXiv:1707.00836, 2017. 5 · arXiv
  12. Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In International Conference on Machine Learn- ing, pages 5583-5594. PMLR, 2021. 3
  13. Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 3 · arXiv
  14. Bowen Li. Word-level fine-grained story visualization. In European Conference on Computer Vision, pages 347-362. Springer, 2022. 2, 5
  15. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. arXiv preprint arXiv:2201.12086, 2022. 2, 4 · arXiv
  16. Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. Storygan: A sequential conditional gan for story vi- sualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6329- 6338, 2019. 1, 2, 3, 4, 5
  17. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 5
  18. Adyasha Maharana and Mohit Bansal. Integrating visuospa- tial, linguistic and commonsense structure into story visual- ization. arXiv preprint arXiv:2110.10834, 2021. 2, 4, 5 · arXiv
  19. Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Im- proving generation and evaluation of visual stories via se- mantic consistency. arXiv preprint arXiv:2105.10026, 2021. 2, 5, 6 · arXiv
  20. Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Storydall-e: Adapting pretrained text-to-image transform- ers for story continuation. arXiv preprint arXiv:2209.06192, 2022. 1, 2, 3, 5, 6 · arXiv
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021. 2, 4
  22. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Confer- ence on Machine Learning, pages 8821-8831. PMLR, 2021. 2, 3
  23. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022. 1, 4 · arXiv
  24. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. 5 · arXiv
  25. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Confer- ence on Machine Learning, pages 2256-2265. PMLR, 2015. 1, 3
  26. Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 5 · arXiv
  27. Yun-Zhu Song, Zhi Rui Tam, Hung-Jen Chen, Huiao-Han Lu, and Hong-Han Shuai. Character-preserving coherent story visualization. In European Conference on Computer Vision, pages 18-33. Springer, 2020. 5
  28. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015. 8
  29. Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 3
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 3
  31. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXiv:2208.10442, 2022. 26 · arXiv
  32. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022. 2 · arXiv
  33. Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to- image generation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 833-842, 2021. 2
  34. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907- 5915, 2017.

Cited by

Sorted by citations

Authors
Pan, X., Qin, P., Li, Y., Xue, H., & Chen, W.
Affiliations
  • New York University (US)
  • Alibaba Group (China) (CN)
  • University of Waterloo (CA)
ORCID
Venue
2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Publisher
IEEE
Published
2024-01-03
Pages
2908-2918
ISSN
2472-6737 (ISSN-L)
DOI
10.1109/wacv57701.2024.00290
OpenAlex
W4394593186
Cited by
47
References
8
Citations by year
2026: 6 · 2025: 23 · 2024: 13 · 2023: 5
FWCI
8.67
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
Keywords
latent diffusion models, autoregressive generation, story visualization, story continuation, text-to-image generation, image sequence generation, multimodal conditioning, character adaptation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review