CoChat
Sign Up
Cited by 12Generative Adversarial Networks and Image SynthesisDigital Media Forensic DetectionAdvanced Image Processing Techniques

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control

Open access
Bian, Yuxuan, Zhang, Zhaoyang, Ju, Xuan, Cao, Mingdeng, Xie, Liangbin, Shan, Ying, Xu, Qiang·2025·Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers
DOI 10.1145/3721238.3730673
Open this paper in CoChat — free account

Abstract

Video inpainting, which aims to restore corrupted video content, has experienced substantial progress. Despite these advances, existing methods, whether propagating unmasked region pixels through optical flow and receptive field priors, or extending image-inpainting models temporally, face challenges in generating fully masked objects or balancing the competing objectives of background context preservation and foreground generation in one model, respectively. To address these limitations, we propose a novel dual-stream paradigm VideoPainter that incorporates an efficient context encoder (comprising only 6% of the backbone parameters) to process masked videos and inject backbone-aware background contextual cues to any pre-trained video DiT, producing semantically consistent content in a plug-and-play manner. This architectural separation significantly reduces the model's learning complexity while enabling nuanced integration of crucial background context. We also introduce a novel target region ID resampling technique that enables any-length video inpainting, greatly enhancing our practical applicability. Additionally, we establish a scalable dataset pipeline leveraging current vision understanding models, contributing VPData and VPBench to facilitate segmentation-based inpainting training and assessment, the largest video inpainting dataset and benchmark to date with over 390K diverse clips. Using inpainting as a pipeline basis, we also explore downstream applications including video editing and video editing pair data generation, demonstrating competitive performance and significant practical potential. Extensive experiments demonstrate VideoPainter's superior performance in both any-length video inpainting and editing, across eight key metrics, including video quality, mask region preservation, and textual coherence.

Overview

Integrity signals

Registry

Crossref · 10.1145/3721238.3730673

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. Omri Avrahami Ohad Fried and Dani Lischinski. 2023. Blended latent diffusion. ACM transactions on graphics (TOG) 42 4 (2023) 1–11. · DOI · Paper page
  2. Jianhong Bai Tianyu He Yuchi Wang Junliang Guo Haoji Hu Zuozhu Liu and Jiang Bian. 2024. Uniedit: A unified tuning-free framework for video motion and appearance editing. arXiv preprint arXiv:https://arXiv.org/abs/2402.13185 (2024). · arXiv
  3. Yuxuan Bian Xuan Ju Jiangtong Li Zhijian Xu Dawei Cheng and Qiang Xu. 2024. Multi-patch prediction: Adapting llms for time series representation learning. arXiv preprint arXiv:https://arXiv.org/abs/2402.04852 (2024). · arXiv
  4. Minghong Cai Xiaodong Cun Xiaoyu Li Wenze Liu Zhaoyang Zhang Yong Zhang Ying Shan and Xiangyu Yue. 2024. DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation. arXiv preprint arXiv:https://arXiv.org/abs/2412.18597 (2024). · arXiv
  5. 10.1109/ICCV.2019.00916 · DOI
  6. Ya-Liang Chang Zhe Yu Liu Kuan-Ying Lee and Winston Hsu. 2019b. Learnable gated temporal shift module for deep video inpainting. arXiv preprint arXiv:https://arXiv.org/abs/1907.01131 (2019). · arXiv
  7. Junsong Chen Jincheng Yu Chongjian Ge Lewei Yao Enze Xie Yue Wu Zhongdao Wang James Kwok Ping Luo Huchuan Lu and Zhenguo Li. 2023. PixArt-α : Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. arxiv:https://arXiv.org/abs/2310.00426 [cs.CV] · arXiv
  8. Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. 2022. EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks.
  9. 10.1109/ICCV51070.2023.01850 · DOI
  10. Zixun Fang Wei Zhai Aimin Su Hongliang Song Kai Zhu Mao Wang Yu Chen Zhiheng Liu Yang Cao and Zheng-Jun Zha. 2024. ViViD: Video Virtual Try-on using Diffusion Models. arXiv preprint arXiv:https://arXiv.org/abs/2405.11794 (2024). · arXiv
  11. 10.1007/978-3-030-58610-2_42 · DOI
  12. 10.1145/3641519.3657407 · DOI
  13. Yuwei Guo Ceyuan Yang Anyi Rao Zhengyang Liang Yaohui Wang Yu Qiao Maneesh Agrawala Dahua Lin and Bo Dai. 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:https://arXiv.org/abs/2307.04725 (2023). · arXiv
  14. 10.1109/ICCV51070.2023.01240 · DOI
  15. 10.1007/978-3-030-58583-9_3 · DOI
  16. Xuan Ju Xian Liu Xintao Wang Yuxuan Bian Ying Shan and Qiang Xu. 2024. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. arXiv preprint arXiv:https://arXiv.org/abs/2403.06976 (2024). · arXiv
  17. 10.1109/CVPR.2019.00594 · DOI
  18. 10.1109/ICCV.2019.00451 · DOI
  19. 10.1007/978-3-030-58548-8_42 · DOI
  20. Yaowei Li Yuxuan Bian Xuan Ju Zhaoyang Zhang Ying Shan and Qiang Xu. 2024. BrushEdit: All-In-One Image Inpainting and Editing. arXiv preprint arXiv:https://arXiv.org/abs/2412.10316 (2024). · arXiv
  21. 10.1109/CVPR52688.2022.01704 · DOI
  22. Rui Liu Hanming Deng Yangyi Huang Xiaoyu Shi Lewei Lu Wenxiu Sun Xiaogang Wang Jifeng Dai and Hongsheng Li. 2021. Decoupled spatial-temporal transformer for video inpainting. arXiv preprint arXiv:https://arXiv.org/abs/2104.06637 (2021). · arXiv
  23. Shilong Liu Zhaoyang Zeng Tianhe Ren Feng Li Hao Zhang Jie Yang Chunyuan Li Jianwei Yang Hang Su Jun Zhu et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:https://arXiv.org/abs/2303.05499 (2023). · arXiv
  24. Chong Mou Mingdeng Cao Xintao Wang Zhaoyang Zhang Ying Shan and Jian Zhang. 2024. ReVideo: Remake a Video with Motion and Content Control. arXiv preprint arXiv:https://arXiv.org/abs/2405.13865 (2024). · arXiv
  25. OpenAI. 2024. Hello GPT-4. https://openai.com/index/hello-gpt-4o/
  26. 10.1109/CVPR.2016.85 · DOI
  27. Adam Polyak Amit Zohar Andrew Brown Andros Tjandra Animesh Sinha Ann Lee Apoorv Vyas Bowen Shi Chih-Yao Ma Ching-Yao Chuang et al. 2024. Movie gen: A cast of media foundation models. arXiv preprint arXiv:https://arXiv.org/abs/2410.13720 (2024). · arXiv
  28. Weize Quan Jiaxi Chen Yanli Liu Dong-Ming Yan and Peter Wonka. 2024. Deep learning-based image and video inpainting: A survey. International Journal of Computer Vision 132 7 (2024) 2367–2400. · DOI
  29. Nikhila Ravi Valentin Gabeur Yuan-Ting Hu Ronghang Hu Chaitanya Ryali Tengyu Ma Haitham Khedr Roman Rädle Chloe Rolland Laura Gustafson et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:https://arXiv.org/abs/2408.00714 (2024). · arXiv
  30. Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman Mehdi Cherti Theo Coombes Aarush Katta Clayton Mullis Mitchell Wortsman et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 (2022) 25278–25294. · DOI · Paper page
  31. 10.1145/3641519.3657497 · DOI
  32. Wenhao Sun Rong-Cheng Tu Jingyi Liao and Dacheng Tao. 2024. Diffusion model-based video editing: A survey. arXiv preprint arXiv:https://arXiv.org/abs/2407.07111 (2024). · arXiv
  33. Gemini Team Petko Georgiev Ving Ian Lei Ryan Burnell Libin Bai Anmol Gulati Garrett Tanzer Damien Vincent Zhufeng Pan Shibo Wang et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:https://arXiv.org/abs/2403.05530 (2024). · arXiv
  34. 10.1007/978-3-030-58536-5_24 · DOI
  35. Pavel Tokmakov, Jie Li, and Adrien Gaidon. 2023. Breaking the “Object” in Video Object Segmentation. In CVPR.
  36. 10.1609/aaai.v33i01.33015232 · DOI
  37. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. Video-to-Video Synthesis. In Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc.https://proceedings.neurips.cc/paper_files/paper/2018/file/d86ea612dec96096c5e0fcc8dd42ab6d-Paper.pdf
  38. Xiang Wang Hangjie Yuan Shiwei Zhang Dayou Chen Jiuniu Wang Yingya Zhang Yujun Shen Deli Zhao and Jingren Zhou. 2024. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems 36 (2024).
  39. Chenfei Wu Lun Huang Qianxi Zhang Binyang Li Lei Ji Fan Yang Guillermo Sapiro and Nan Duan. 2021. GODIVA: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:https://arXiv.org/abs/2104.14806 (2021). · arXiv
  40. 10.1007/978-3-030-01228-1_36 · DOI
  41. 10.1109/CVPR.2019.00384 · DOI
  42. Zhuoyi Yang Jiayan Teng Wendi Zheng Ming Ding Shiyu Huang Jiazheng Xu Yuanming Yang Wenyi Hong Xiaohan Zhang Guanyu Feng et al. 2024. CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. arXiv preprint arXiv:https://arXiv.org/abs/2408.06072 (2024). · arXiv
  43. 10.1007/978-3-030-58517-4_31 · DOI
  44. 10.1007/978-3-031-19797-0_5 · DOI
  45. 10.1007/978-3-031-19797-0_5 · DOI
  46. 10.1109/CVPR52688.2022.00589 · DOI
  47. 10.1109/CVPR.2018.00068 · DOI · Paper page
  48. 10.1109/CVPR52733.2024.00684 · DOI
  49. 10.1007/978-3-030-96530-3 · DOI
  50. Bojia Zi Shihao Zhao Xianbiao Qi Jianan Wang Yukai Shi Qianyu Chen Bin Liang Kam-Fai Wong and Lei Zhang. 2024. CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency Controllability and Compatibility. arXiv preprint arXiv:https://arXiv.org/abs/2403.12035 (2024). · arXiv
  51. 10.1109/CVPR46437.2021.01618 · DOI

Cited by

Sorted by citations

Metadata

Authors
Bian, Y., Zhang, Z., Ju, X., Cao, M., Xie, L., Shan, Y., & Xu, Q.
Affiliations
  • Chinese University of Hong Kong (HK)
  • Tencent (China) (CN)
  • The University of Tokyo (JP)
  • University of Macau (MO)
ORCID
Venue
Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers
Publisher
ACM
Published
2025-07-27
Pages
1-12
DOI
10.1145/3721238.3730673
OpenAlex
W4412587860
Cited by
12
References
6
Citations by year
2026: 8 · 2025: 4
FWCI
9.01
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM SIGGRAPH Conference (SIGGRAPH) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
Keywords
video inpainting, video editing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review