Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
The DOI matches its DataCite record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Generating videos for visual storytelling can be a tedious and complex process that typically requires either live-action filming or graphics animation rendering. To bypass these challenges, our key idea is to utilize the abundance of existing video clips and synthesize a coherent storytelling video by customizing their appearances. We achieve this by developing a framework comprised of two functional modules: (i) Motion Structure Retrieval, which provides video candidates with desired scene or motion context described by query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates plot-aligned videos under the guidance of motion structure and text prompts. For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure. For the second module, we propose a controllable video generation model that offers flexible controls over structure and characters. The videos are synthesized by following the structural guidance and appearance instruction. To ensure visual consistency across clips, we propose an effective concept personalization approach, which allows the specification of the desired character identities through text prompts. Extensive experiments demonstrate that our approach exhibits significant advantages over various existing baselines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Animate-AStory [174] optimizes the video generator with image data, and then finetunes a LoRA [175] adapter to capture the appearance details of the given character.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [174]
In the citing paper
“Animate-A-Story [174] simplifies text-to-video generation by dividing it into plot-based video augmentation and videodiffusion generation conditioned on text and video inputs.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [174]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 1×
“Initially, they project videos into a latent space by learning a VAE or VQGAN [Esser et al. 2021], followed by training a transformer to model the latent distribution.”
MethodDreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 6×
“To evaluate the effectiveness of our personalization module, we compare our approach with the previous three baseline approaches: Dreambooth[Ruiz et al. 2023], Textual inversion [Gal et al.”
BackgroundMethodResultInfluentialMulti-Concept Customization of Text-to-Image Diffusion
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00192 · Cited by 674 · cited 3×
“To evaluate the effectiveness of our personalization module, we compare our approach with the previous three baseline approaches: Dreambooth[Ruiz et al. 2023], Textual inversion [Gal et al. 2022], and Custom Diffusion [Kumari et al. 2023].”
BackgroundMethodResultImagen Video: High Definition Video Generation with Diffusion Models
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02303 · Cited by 399
Make-A-Video: Text-to-Video Generation without Text-Video Data
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14792 · Cited by 395
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022) · arXiv · doi:10.48550/arxiv.2204.03458 · Cited by 301
MagicVideo: Efficient Video Generation With Latent Diffusion Models
Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., & Feng, J. (2022) · arXiv · doi:10.48550/arxiv.2211.11018 · Cited by 82 · cited 3×
“Inspired by LDM [Rombach et al. 2022], several works [Blattmann et al. 2023; He et al. 2022; Luo et al. 2023; Mei and Patel 2022; Yu et al. 2023b; Zhou et al. 2022] extend LDM for video generation.”
BackgroundMethod
Cited by
VBench: Comprehensive Benchmark Suite for Video Generative Models
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02060 · Cited by 407
Self-citationRetrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240
“SQL is not only a programming language but can also be considered as a structured representation, so we place text-to-SQL (a special case of code generation) in this subsection.”
Sora as a World Model? A Complete Survey on Text-to-Video Generation
Puspitasari, F. D., Zhang, C., Cho, J., Haider, A., Eman, N. U., Amin, O., Mankowski, A., Umair, M., Zheng, J., Zheng, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05131 · Cited by 10
Metadata
- Authors
- He, Y., Xia, M., Chen, H., Cun, X., Gong, Y., Xing, J., Zhang, Y., Wang, X., Weng, C., Shan, Y., & Chen, Q.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2307.06940
- References
- 8
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation. https://app.cochat.ai/papers/10.48550/arxiv.2307.06940
- The paper
- He, Y., Xia, M., Chen, H., Cun, X., Gong, Y., Xing, J., Zhang, Y., Wang, X., Weng, C., Shan, Y., & Chen, Q. (2023). Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation. arXiv. https://doi.org/10.48550/arxiv.2307.06940
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.