CoChat
Sign Up
Cited by 72Reinforcement Learning in RoboticsAutonomous Vehicle Technology and SafetyGenerative Adversarial Networks and Image Synthesis

DriveGAN: Towards a Controllable High-Quality Neural Simulation

Open access
Kim, Seung Wook, Philion, Jonah, Torralba, Antonio, Fidler, Sanja·2021·2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr46437.2021.00576
Open this paper in CoChat — free accountOpen PDF

Abstract

Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are hand-crafted, a scaleable way to build simulators is to use machine learning to learn how the environment behaves in response to an action, directly from data. In this work, we aim to learn to simulate a dynamic environment directly in pixel-space, by watching unannotated sequences of frames and their associated actions. We introduce a novel high-quality neural simulator referred to as DriveGAN that achieves controllability by disentangling different components without supervision. In addition to steering controls, it also includes controls for sampling features of a scene, such as the weather as well as the location of non-player objects. Since DriveGAN is a fully differentiable simulator, it further allows for re-simulation of a given video sequence, offering an agent to drive through a recorded scene again, possibly taking different actions. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data. We showcase that our approach greatly surpasses the performance of previous data-driven simulators, and al-lows for new key features not explored before.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr46437.2021.00576

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [75]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. CoRR, abs/1710.11252, 2017. 2 · arXiv
  2. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013. 2
  3. Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.02254, 2017. 5 · arXiv
  4. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. 14 · arXiv
  5. Aidan Clark, Jeff Donahue, and Karen Simonyan. Ef- ficient video generation on complex datasets. CoRR, abs/1907.06571, 2019. · arXiv
  6. Marc Deisenroth and Carl E Rasmussen. Pilco: A model- based and data-efficient approach to policy search. In Pro- ceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472, 2011. 2
  7. Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. CoRR, abs/1802.07687, 2018. 2 · arXiv
  8. Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Advances in neural informa- tion processing systems, pages 4414-4423, 2017. 4
  9. Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta- sim2: Unsupervised learning of scene structure for synthetic data generation. arXiv preprint arXiv:2008.09092, 2020. 1, 2 · arXiv
  10. Vincent Dumoulin, Jonathon Shlens, and Manjunath Kud- lur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629, 2016. 3 · arXiv
  11. Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsuper- vised learning for physical interaction through video predic- tion. In Advances in neural information processing systems, pages 64-72, 2016. 2
  12. Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Exploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06830, 2017. 3 · arXiv
  13. David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Informa- tion Processing Systems, pages 2450-2462, 2018. 1, 2, 5
  14. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Ville- gas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019. 2
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997. 2, 4, 14
  16. Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. CoRR, abs/1806.04166, 2018. 2 · arXiv
  17. Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceed- ings of the IEEE International Conference on Computer Vi- sion, pages 1501-1510, 2017. 3
  18. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. arXiv preprint arXiv:1502.03167, 2015. 14 · arXiv
  19. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017. 3, 13, 15
  20. Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019. 2 · arXiv
  21. Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Ko- ray Kavukcuoglu. Video pixel networks. CoRR, abs/1610.00527, 2016. 2 · arXiv
  22. Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 4551-4560, 2019. 1, 2
  23. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improv- ing the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8119, 2020. 3, 8, 12, 13
  24. Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Tor- ralba, and Sanja Fidler. Learning to simulate dynamic envi- ronments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1231-1240, 2020. 1, 2, 4, 5, 7
  25. Yoon Kim. Convolutional neural networks for sentence clas- sification. arXiv preprint arXiv:1408.5882, 2014. 4 · arXiv
  26. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 13 · arXiv
  27. Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. CoRR, abs/1903.01434, 2019. 2 · arXiv
  28. Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. CoRR, abs/1512.09300, 2015. 2, 3 · arXiv
  29. Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018. 2, 5 · arXiv
  30. Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv:1705.02894, 2017. 15 · arXiv
  31. William Lotter, Gabriel Kreiman, and David Cox. Deep pre- dictive coding networks for video prediction and unsuper- vised learning. arXiv preprint arXiv:1605.08104, 2016. 2 · arXiv
  32. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Recti- fier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3, 2013. 12
  33. Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11167-11176, 2020. 1, 2
  34. Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801.04406, 2018. 15 · arXiv
  35. Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In Advances in Neural Information Processing Systems, pages 92-102, 2019. 2
  36. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative ad- versarial networks. arXiv preprint arXiv:1802.05957, 2018. 14 · arXiv
  37. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction us- ing deep networks in atari games. In Advances in neural information processing systems, pages 2863-2871, 2015. 2
  38. Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. arXiv preprint arXiv:2008.05711, 2020. 8, 15 · arXiv
  39. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018. 1
  40. Marc'Aurelio Ranzato, Arthur Szlam, Joan Bruna, Michaël Mathieu, Ronan Collobert, and Sumit Chopra. Video (lan- guage) modeling: a baseline for generative models of natural videos. CoRR, abs/1412.6604, 2014. 2 · arXiv
  41. Nataniel Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate. arXiv preprint arXiv:1810.02513, 2018. 1 · arXiv
  42. M. Saito and Shunta Saito. Tganv2: Efficient training of large models for video generation with multiple subsampling layers. ArXiv, abs/1811.09245, 2018. 2 · arXiv
  43. Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Sin- gan: Learning a generative model from a single natural im- age. In Proceedings of the IEEE International Conference on Computer Vision, pages 4570-4580, 2019. 3, 13, 15
  44. Richard S Sutton. Integrated architectures for learning, plan- ning, and reacting based on approximating dynamic pro- gramming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990. 2
  45. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. 2
  46. Dustin Tran, Rajesh Ranganath, and David Blei. Hierar- chical implicit models and likelihood-free variational infer- ence. In Advances in Neural Information Processing Sys- tems, pages 5523-5533, 2017. 15
  47. Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. CoRR, abs/1707.04993, 2017. 2 · arXiv
  48. Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6 · arXiv
  49. Dirk Weissenborn, Oscar Täckström, and Jakob Uszko- reit. Scaling autoregressive video models. CoRR, abs/1906.02634, 2019. · arXiv
  50. Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jiten- dra Malik, and Silvio Savarese. Gibson env: real-world per- ception for embodied agents. In Computer Vision and Pat- tern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018. 5
  51. Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and information-preserving future frame prediction and beyond. In ICLR, 2020. 2
  52. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586-595, 2018. 3, 13

Cited by

Sorted by citations

Authors
Kim, S. W., Philion, J., Torralba, A., & Fidler, S.
Affiliations
Nvidia (United Kingdom) (GB)
ORCID
Venue
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2021-06
Pages
5816-5825
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr46437.2021.00576
OpenAlex
W3158192271
Cited by
72
References
7
Citations by year
2026: 12 · 2025: 15 · 2024: 14 · 2023: 16 · 2022: 13 · 2021: 2
FWCI
5.20
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
neural simulation, differentiable simulation, driving simulation, autonomous driving, weather modification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review