The DOI matches its DataCite record; 74 of the 78 references it cites were found in a registry or an index.
Video-to-Video Synthesis
Open accessAbstract
We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image synthesis problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without understanding temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a novel video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generator and discriminator architectures, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our approach to future video prediction, outperforming several state-of-the-art competing systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Results again showed that the proposed algorithm overperformed the other algorithms, both in the PSNR and SSIM metrics.”
From A Review of Detection and Removal of Raindrops in Automotive Vision Systems (2021) · cites this paper as [67]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Self-citationImage-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Self-citationAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
(1981) · Cited by 11,963
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Research works printed but not linked: 10
- [7][7] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. -
- [16][16] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS) , 2016. -
- [25][25] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision (ICCV) , 2017. -
- [32][32] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. -
- [36][36] N. Kalchbrenner, A. v. d. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu. Video pixel networks. arXiv preprint arXiv:1610.00527 , 2016. - · arXiv
- [37][37] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations (ICLR) , 2018. -
- [42][42] X. Liang, L. Lee, W. Dai, and E. P. Xing. Dual motion GAN for future-flow embedded video prediction. In Advances in Neural Information Processing Systems (NIPS) , 2017. -
- [47][47] L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool. Pose guided person image generation. In Advances in Neural Information Processing Systems (NIPS) , 2017. -
- [50][50] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. In International Conference on Learning Representations (ICLR) , 2016. -
- [65][65] N. Srivastava, E. Mansimov, and R. Salakhudinov. Unsupervised learning of video representations using lstms. In International Conference on Machine Learning (ICML) , 2015. -
Cited by
Sorted by citations
Semantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974
Self-citationThies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,443
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231
Zhu, Y., Zabaras, N. J., Koutsourelakis, P., & Perdikaris, P. G. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.05.024 · Cited by 1,083
Media Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979
Metadata
- Authors
- Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B.
- Affiliations
- Nvidia (United Kingdom) (GB)
- University of California, Berkeley (US)
- Adobe Systems (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 31
- Pages
- 1144-1156
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1808.06601
- OpenAlex
- W2886748926
- Cited by
- 594
- References
- 78
- Citations by year
- 2026: 1 · 2025: 28 · 2024: 61 · 2023: 87 · 2022: 88 · 2021: 123 · 2020: 117 · 2019: 87 · 2018: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- semantic segmentation masks, generative adversarial networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
