CoChat
Sign Up
PreprintCited by 17Generative Adversarial Networks and Image SynthesisAutonomous Vehicle Technology and SafetyAdvanced Vision and Imaging

MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control

Open access
Gao, Ruiyuan, Chen, Kai, Xiao, Bo, Hong, Lanqing, Li, Zhenguo, Xu, Qiang·2024·arXiv
DOI 10.48550/arxiv.2411.13807arXiv 2411.13807
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid advancement of diffusion models has greatly improved video synthesis, especially in controllable video generation, which is vital for applications like autonomous driving. Although DiT with 3D VAE has become a standard framework for video generation, it introduces challenges in controllable driving video generation, especially for geometry control, rendering existing control methods ineffective. To address these issues, we propose MagicDrive-V2, a novel approach that integrates the MVDiT block and spatial-temporal conditional encoding to enable multi-view video generation and precise geometric control. Additionally, we introduce an efficient method for obtaining contextual descriptions for videos to support diverse textual control, along with a progressive training strategy using mixed video data to enhance training efficiency and generalizability. Consequently, MagicDrive-V2 enables multi-view driving video synthesis with $3.3\times$ resolution and $4\times$ frame count (compared to current SOTA), rich contextual control, and geometric controls. Extensive experiments demonstrate MagicDrive-V2's ability, unlocking broader applications in autonomous driving.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.13807

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Diffusion score matching and Gaussian flow matching frameworks are theoretically equivalent, sharing fundamental similarities in their objectives and training procedures.

    “as shown by Gao et al. (2024), these frameworks are theoretically equivalent, sharing fundamental similarities in their objectives and training procedures.”

    From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Gao et al. (2024)

    The Related Work section says flow matching simplifies diffusion models, establishing a relationship between the frameworks. The Preliminary section gives a flow-matching training loss, but neither passage establishes theoretical equivalence with diffusion score matching or compares their objectives and training procedures. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Existing autonomous driving video generation approaches have been limited by constraints in data scale, resolution, and the number of camera views, restricting their effectiveness as comprehensive driving world simulators.

    “Despite their potential, existing approaches have been limited by constraints in data scale (Gao et al., 2024,; Jia et al., 2023; Lu et al., 2025; Wang et al., 2023, 2024), resolution (Hu et al., 2023; Yang et al., 2024), and the number of camera views (Blattmann et al., 2023; Gao et al., 2024), restricting their effectiveness as comprehensive driving world simulators.”

    From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Gao et al., 2024

    Table 1 (Introduction; page not provided) shows that several earlier driving-video methods generated only one camera view, supporting the camera-view limitation for those approaches. The Introduction and Related Work also describe limits in resolution and frame count, but the passages found do not identify data scale as a limitation of earlier methods. In the citing sentence, Gao et al. is attached to data scale and camera views, not resolution. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Video generative models have the potential to revolutionize autonomous driving simulation by enabling the generation of realistic driving videos conditioned on diverse input modalities.

    “Video generative models have the potential to revolutionize autonomous driving simulation by enabling the generation of realistic driving videos conditioned on diverse input modalities, such as text, images, trajectories, 3D data, or maps (Blattmann et al., 2023; Gao et al., 2024,,; Hu et al., 2023; Jia et al., 2023; Kim et al., 2021; Lu et al., 2025; Wang et al., 2023, 2024; Yang et al., 2024).”

    From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Gao et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [1][1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foun- dation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 3 · arXiv
  2. [2][2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 3 · arXiv
  3. [6][6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-\alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024. 2, 3
  4. [7][7] Kai Chen, Ruiyuan Gao, Lanqing Hong, Hang Xu, Xu Jia, Holger Caesar, Dengxin Dai, Bingbing Liu, Dzmitry Tsishkou, Songcen Xu, et al. ECCV 2024 W-CODA: 1st workshop on multimodal perception and comprehen- sion of corner cases in autonomous driving. arXiv preprint arXiv:2507.01735, 2025. 5 · arXiv
  5. [9][9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021. 2
  6. [10][10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In ICML, 2024. 2, 3, 7
  7. [11][11] Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive3d: Controllable 3d genera- tion for any-view rendering in street scenes. arXiv preprint arXiv:2405.14475, 2024. 2, 3, 5 · arXiv
  8. [12][12] Ruiyuan Gao, Kai Chen, Enze Xie, HONG Lanqing, Zhen- guo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. In ICLR, 2024. 1, 2, 3, 4, 5, 6
  9. [13][13] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398, 2024. 1, 2, 5 · arXiv
  10. [14][14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 2
  11. [15][15] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. In NeurIPS, 2022. 2
  12. [16][16] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for au- tonomous driving. arXiv preprint arXiv:2309.17080, 2023. 1, 2 · arXiv
  13. [17][17] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driv- ing. In CVPR, 2023. 2
  14. [18][18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5 · arXiv
  15. [19][19] Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Heng- tong Hu, Xia Zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, et al. Dive: Dit-based video generation with enhanced control. arXiv preprint arXiv:2409.01595, 2024. 2 · arXiv
  16. [23][23] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. 2
  17. [24][24] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. 3
  18. [25][25] Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023. 3
  19. [27][27] Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, et al. Unleashing generalization of end-to-end autonomous driving with controllable long video generation. arXiv preprint arXiv:2406.01349, 2024. 1, 2, 3, 5 · arXiv
  20. [28][28] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equa- tions. In ICLR, 2022. 4
  21. [32][32] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In ICLR, 2020. 2
  22. [35][35] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 4 · arXiv
  23. [36][36] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023. 1, 2 · arXiv
  24. [41][41] Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street- view elements with multi-perspective consistency via bev sketch layout. arXiv preprint arXiv:2308.01661, 2023. 5, 6 · arXiv
  25. [42][42] Xuemeng Yang, Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A closed-loop generative sim- ulation platform for autonomous driving. arXiv preprint arXiv:2408.00415, 2024. 1, 2 · arXiv
  26. [43][43] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1, 2, 3, 4, 5, 6 · arXiv
  27. [45][45] Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene rep- resentation. arXiv preprint arXiv:2410.13571, 2024. 2 · arXiv

Cited by

Sorted by citations

Related

Authors
Gao, R., Chen, K., Xiao, B., Hong, L., Li, Z., & Xu, Q.
Affiliations
  • Chinese University of Hong Kong (HK)
  • Hong Kong University of Science and Technology (HK)
  • Huawei Technologies (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
28135-28144
DOI
10.48550/arxiv.2411.13807
arXiv
2411.13807
OpenAlex
W7160087193
Cited by
16
References
20
Citations by year
2026: 16
FWCI
12.09
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Chinese University of Hong Kong (7106937,8601130,8601440)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
long video generation, high-resolution video, autonomous driving, adaptive control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review