CoChat
Sign Up
PreprintCited by 171Generative Adversarial Networks and Image SynthesisDigital Media Forensic DetectionImage Enhancement Techniques

The DOI matches its DataCite record.

Pix2Video: Video Editing using Image Diffusion

Open access
Ceylan, Duygu, Huang, Chun-Hao Paul, Mitra, Niloy J.·2023·arXiv
DOI 10.48550/arxiv.2303.12688arXiv 2303.12688
Open this paper in CoChat — free accountOpen PDF

Abstract

Image diffusion models, trained on massive image collections, have emerged as the most versatile image generator model in terms of quality and diversity. They support inverting real images and conditional (e.g., text) generation, making them attractive for high-quality image editing applications. We investigate how to use such pre-trained image models for text-guided video editing. The critical challenge is to achieve the target edits while still preserving the content of the source video. Our method works in two simple steps: first, we use a pre-trained structure-guided (e.g., depth) image diffusion model to perform text-guided edits on an anchor frame; then, in the key step, we progressively propagate the changes to the future frames via self-attention feature injection to adapt the core denoising step of the diffusion model. We then consolidate the changes by adjusting the latent code for the frame before continuing the process. Our approach is training-free and generalizes to a wide range of edits. We demonstrate the effectiveness of the approach by extensive experimentation and compare it against four different prior and parallel efforts (on ArXiv). We demonstrate that realistic text-guided video edits are possible, without any compute-intensive preprocessing or video-specific finetuning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.12688

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pix2Video demonstrates good editability and consistency in its results despite using a comparatively lightweight architecture.

    “Despite using a comparatively lightweight architecture, the authors demonstrate good editability and consistency in their results.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Ceylan et al. (2023)

    The Results (p. 7) say Pix2Video balances respecting the edit with temporal consistency, and the Introduction (p. 2) says it performs on par with or better than baselines without compute-intensive preprocessing or video-specific finetuning. These passages support good editability and consistency with relatively low processing demands, but they do not establish that the underlying architecture is lightweight; the paper describes using a pretrained large-scale image generation model. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Several video diffusion models have implemented video editing conditioned on extracted frame features such as depth or pose estimates.

    “Several video diffusion models have also implemented video editing that is conditioned on extracted frame features such as depth (Ceylan et al. 2023; Esser et al. 2023; Xing et al. 2023a, see Sec. 10.2) or pose estimates (Ma et al. 2023; Zhao et al. 2023, see Sec. 10.3).”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Ceylan et al. 2023

  • Pix2Video eliminates the need for fine-tuning the model on each individual video.

    “it however eliminates the need for fine-tuning the model on each individual video.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Ceylan et al. (2023)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 61

  1. [2][2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022. 2, 3 - · arXiv
  2. [3][3] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2Live: Text-driven layered image and video editing. In European Conference on Computer Vision (ECCV) , pages 707-723. Springer, 2022. 2, 3, 5, 7, 12 -
  3. [4][4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR) , 2019. 2 -
  4. [5][5] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A Efros, and Tero Karras. Generating long videos of dynamic scenes. In Conference on Neural Information Processing Systems (NeurIPS) , 2022. 2 -
  5. [6][6] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022. 8 - · arXiv
  6. [7][7] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. In International Conference on Computer Vision (ICCV) , pages 1436714376, October 2021. 2 -
  7. [8][8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. arXiv preprint arXiv:2209.04747 , 2022. 1, 2 - · arXiv
  8. [9][9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Conference on Neural Information Processing Systems (NeurIPS) , volume 34, pages 8780-8794, 2021. 2 -
  9. [10][10] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 , 2023. 3, 5 - · arXiv
  10. [11][11] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scene: Scenebased text-to-image generation with human priors. arXiv , 2022. 3 -
  11. [12][12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Conference on Neural Information Processing Systems (NeurIPS) , volume 27, 2014. 2 -
  12. [13][13] Sonam Gupta, Arti Keshari, and Sukhendu Das. RV-GAN: Recurrent gan for unconditional video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 20242033, June 2022. 2 -
  13. [14][14] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Gir- shick. Mask r-cnn. In International Conference on Computer Vision (ICCV) , Oct 2017. 12 -
  14. [15][15] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 , 2022. 3, 4 - · arXiv
  15. [16][16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In International Conference on Learning Representations (ICLR) , 2023. 1, 2, 12, 13 -
  16. [17][17] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 , 2021. 7 - · arXiv
  17. [18][18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv , 2022. 3, 7 -
  18. [19][19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Conference on Neural Information Processing Systems (NeurIPS) , volume 33, pages 6840-6851, 2020. 1, 2 -
  19. [20][20] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 , 2022. 3 - · arXiv
  20. [21][21] Ondˇ rej Jamriˇ ska, ˇ S´ arka Sochorov´ a, Ondˇ rej Texler, Michal Luk´ aˇ c, Jakub Fiˇ ser, Jingwan Lu, Eli Shechtman, and Daniel S´ ykora. Stylizing video by example. ACM Transactions on Graphics , 38(4), 2019. 1, 2, 3, 5, 7, 8, 12, 13 -
  21. [22][22] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Conference on Neural Information Processing Systems (NeurIPS) , 2022. 2 -
  22. [23][23] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Computer Vision and Pattern Recognition (CVPR) , pages 4401-4410, 2019. 2 -
  23. [24][24] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG) , 40(6):1-12, 2021. 3, 5 -
  24. [25][25] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 2022. 2 - · arXiv
  25. [26][26] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In International Conference on Learning Representations (ICLR) , 2023. 5 -
  26. [27][27] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022. 5 -
  27. [28][28] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. In SIGGRAPH Asia , 2020. 3 -
  28. [29][29] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations (ICLR) , 2021. 2, 5, 7 -
  29. [30][30] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022. 1, 5, 12, 13 - · arXiv
  30. [31][31] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. arXiv , 2023. 1, 3 -
  31. [32][32] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv , 2023. 2 -
  32. [33][33] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162, pages 16784-16804, 2022. 2, 4 -
  33. [34][34] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-toimage synthesis. In Neural Information Processing Systems Datasets and Benchmarks Track , 2021. 7 -
  34. [36][36] Haonan Qiu, Yuming Jiang, Hang Zhou, Wayne Wu, and Ziwei Liu. Stylefacev: Face video generation via decomposing and recomposing pretrained stylegan3. arXiv , 2022. 2 -
  35. [37][37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , pages 8748-8763, 2021. 7 -
  36. [38][38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. 2 - · arXiv
  37. [39][39] Ren´ e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 44(3), 2022. 3, 8 -
  38. [40][40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High-resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022. 1, 2 -
  39. [41][41] Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In German Conference on Pattern Recognition , pages 26-36, 2016. 3 -
  40. [42][42] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mo- hammad Norouzi. Palette: Image-to-image diffusion models. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) , 2022. 2 -
  41. [43][43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Conference on Neural Information Processing Systems (NeurIPS) , 2022. 2 -
  42. [44][44] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In ICCV , 2017. 2 -
  43. [45][45] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Video: Text-to-video generation without text-video data. In International Conference on Learning Representations (ICLR) , 2023. 1, 3 -
  44. [46][46] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Computer Vision and Pattern Recognition (CVPR) , pages 3626-3636, 2022. 2 -
  45. [47][47] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR) , 2021. 1, 2, 5 -
  46. [48][48] Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV) , 2020. 7 -
  47. [49][49] Ondˇ rej Texler, David Futschik, Michal Kuˇ cera, Ondˇ rej Jamriˇ ska, ˇ S´ arka Sochorov´ a, Meglei Chai, Sergey Tulyakov, and Daniel S´ ykora. Interactive video stylization using fewshot patch-based training. ACM Transactions on Graphics , 39(4):73, 2020. 3 -
  48. [50][50] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In Computer Vision and Pattern Recognition (CVPR) , pages 1526-1535, 2018. 2 -
  49. [51][51] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for textdriven image-to-image translation. arXiv preprint arXiv:2211.12572 , 2022. 2, 4 - · arXiv
  50. [52][52] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations (ICLR) , 2023. 3 -
  51. [53][53] Andrey Voynov, Kfir Abernan, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. arXiv preprint arXiv:2211.13752 , 2022. 2, 3 - · arXiv
  52. [54][54] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv , 2022. 2 -
  53. [55][55] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. In Conference on Neural Information Processing Systems (NeurIPS) , 2019. 2 -
  54. [56][56] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-tovideo synthesis. In Conference on Neural Information Processing Systems (NeurIPS) , 2018. 2 -
  55. [57][57] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-A-Video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565 , 2022. 2, 3, 5, 7, 12, 13 - · arXiv
  56. [58][58] Yiran Xu, Badour AlBahar, and Jia-Bin Huang. Temporally consistent semantic video editing. arXiv preprint arXiv: 2206.10590 , 2022. 2 - · arXiv
  57. [59][59] Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos´ e Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, MingHsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: Masked generative video transformer. arXiv , 2022. 3 -
  58. [60][60] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Computer Vision and Pattern Recognition (CVPR) , 2023. 1, 3 -
  59. [61][61] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations (ICLR) , 2022. 2 -
  60. [62][62] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023. 2 - · arXiv
  61. [63][63] Qihang Zhang, Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Towards smooth video composition. International Conference on Learning Representations (ICLR) , 2023. 2

Cited by

Sorted by citations

Metadata

Authors
Ceylan, D., Huang, C.-H. P., & Mitra, N. J.
Affiliations
Adobe Systems (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
23149-23160
DOI
10.48550/arxiv.2303.12688
arXiv
2303.12688
OpenAlex
W4390871980
Cited by
171
References
0
Citations by year
2026: 32 · 2025: 62 · 2024: 73 · 2023: 4
FWCI
18.32
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Enhancement Techniques
Keywords
video editing, image diffusion models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review