Learning temporal coherence via self-supervision for GAN-based video generation
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Our work explores temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationships in the generated data are much less explored. Natural temporal changes are crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as L 2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temporal cycle consistencies. In contrast, we focus on improving learning objectives and propose a temporally self-supervised algorithm. For both tasks, we show that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effectively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. Additionally, we propose a first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirm the rankings computed with these metrics. Code, data, models, and results are provided at https://github.com/thunil/TecoGAN.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 13×
“Generative adversarial models (GANs) have been extremely successful at learning complex distributions such as natural images (Zhu et al., 2017; Isola et al., 2017).”
BackgroundInfluentialImage-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 2×
“…found adversarial training [Goodfel-low et al. 2014] to significantly improve the perceptual quality in multi-modal settings such as image generation [Brock et al. 2019], colorization [He et al. 2018], super-resolution [Ledig et al. 2016], and translation [Isola et al. 2017; Zhu et al. 2017] tasks.”
BackgroundThe Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870 · cited 2×
“While regular losses such as L2 [Kim et al. 2016; Lai et al. 2017] offer good performance for image super-resolution (SR) tasks in terms of PSNR metrics, previous work found adversarial training [Goodfellow et al.”
MethodResultMaking a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Deformable ConvNets V2: More Deformable, Better Results
Zhu, X., Hu, H., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00953 · Cited by 2,943
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
Research works printed but not linked: 45
- Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. 2018. Recycle-GAN: Unsupervised Video Retargeting. In The European Conference on Computer Vision (ECCV).
- Dina Bashkirova, Ben Usman, and Kate Saenko. 2018. Unsupervised video-to-video translation. arXiv preprint arXiv:1806.03698 (2018). · arXiv
- 10.1109/CVPR.2018.00652 · DOI · Paper page
- Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations. https://openreview.net/forum?id=B1xsqj09Fm
- Jose Caballero, Christian Ledig, Andrew P Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation.. In CVPR, Vol. 1. 7.
- (CC) Blender Foundation | mango.blender.org. 2011. Tears of Steel. https://mango.blender.org/. Online; accessed 15 Nov. 2018.
- 10.1109/ICCV.2017.126 · DOI · Paper page
- Yang Chen, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. 2019. Mocycle-GAN: Unpaired Video-to-Video Translation. arXiv preprint arXiv:1908.09514 (August 2019). · arXiv
- 10.1111/cgf.13511 · DOI
- 10.5555/1763974.1764031 · doi:10.5555/1763974.1764031
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
- Ishaan Gulrajani Faruk Ahmed Martin Arjovsky Vincent Dumoulin and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in neural information processing systems. 5767--5777.
- 10.1109/CVPR.2019.00402 · DOI · Paper page
- 10.1145/3197517.3201365 · DOI · Paper page
- 10.1145/3306346.3323006 · DOI
- 10.1109/CVPR.2018.00340 · DOI · Paper page
- 10.1145/3355089.3356557 · DOI
- Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017). · arXiv
- 10.1145/3272127.3275065 · DOI
- 10.1145/3355089.3356560 · DOI
- Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate superresolution. In IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2. 5.
- 10.1109/ICCV.2015.68 · DOI · Paper page
- 10.1109/CVPR.2011.5995614 · DOI · Paper page
- 10.1109/ICCV.2017.274 · DOI · Paper page
- 10.1109/CVPR.2019.00470 · DOI · Paper page
- 10.1109/ICCV.2017.304 · DOI · Paper page
- Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. 2019. NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
- Suraj Nair, Mohammad Babaeizadeh, Chelsea Finn, Sergey Levine, and Vikash Kumar. 2018. Time reversal as self-supervision. arXiv preprint arXiv:1810.01128 (2018). · arXiv
- 10.1145/3343031.3350864 · DOI
- Eduardo Pérez-Pellitero, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Schölkopf. 2018. Photorealistic Video Super Resolution. arXiv preprint arXiv:1807.07930 (2018). · arXiv
- 10.1007/978-3-319-45886-1_3 · DOI · Paper page
- 10.1109/ICCV.2017.481 · DOI · Paper page
- Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-Recurrent Video Super-Resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).
- 10.1109/CVPR.2016.207 · DOI · Paper page
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
- 10.1145/3197517.3201333 · DOI · Paper page
- Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. 2017. Detail-Revealing Deep Video Super-Resolution. In The IEEE International Conference on Computer Vision (ICCV).
- 10.1145/3072959.3073633 · DOI
- 10.1109/TIP.2018.2836316 · DOI · Paper page
- Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Guilin Liu Andrew Tao Jan Kautz and Bryan Catanzaro. 2018a. Video-to-Video Synthesis. In Advances in Neural Information Processing Systems (NIPS).
- 10.1109/CVPR.2019.00267 · DOI · Paper page
- 10.1145/3306346.3323024 · DOI · Paper page
- 10.1145/3306346.3322956 · DOI
- 10.1145/3197517.3201304 · DOI · Paper page
- 10.1109/CVPR.2019.00824 · DOI · Paper page
Cited by
Generative Adversarial Networks for Spatio-temporal Data: A Survey
Gao, N., Xue, H., Shao, W., Zhao, S., Qin, K. K., Prabowo, A., Rahaman, M. S., & Salim, F. D. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3474838 · Cited by 127
Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks
Lee, H.-P. (., Yang, Y.-J., Von Davier, T. S., Forlizzi, J., & Das, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642116 · Cited by 82
Detecting Deepfake Videos with Temporal Dropout 3DCNN
Zhang, D., Li, C., Lin, F., Zeng, D., & Ge, S. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/178 · Cited by 77
MoCHA-former: Moiré-Conditioned Hybrid Adaptive Transformer for Video Demoiréing
Sung, J., Roh, C., Eom, C., & Oh, J. (2026) · Neurocomputing · doi:10.1016/j.neucom.2025.132477
Metadata
- Authors
- Chu, M., Xie, Y., Mayer, J., Leal-Taixé, L., & Thuerey, N.
- Venue
- ACM Transactions on Graphics
- Published
- 2020-08-12
- Volume
- 39
- Issue
- 4
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2996880115
- Cited by
- 201
- References
- 12
- FWCI
- 7.98
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- European Research Council (realFlow (StG- 2015-637014))
- Alexander von Humboldt-Stiftung (Sofja Kovalevskaja Award)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Learning temporal coherence via self-supervision for GAN-based video generation. https://app.cochat.ai/papers/10.1145/3386569.3392457
- The paper
- Chu, M., Xie, Y., Mayer, J., Leal-Taixé, L., & Thuerey, N. (2020). Learning temporal coherence via self-supervision for GAN-based video generation. ACM Transactions on Graphics. https://doi.org/10.1145/3386569.3392457
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.