CoChat

Learning temporal coherence via self-supervision for GAN-based video generation

Open access
Chu, Mengyu, Xie, You, Mayer, Jonas, Leal-Taixé, Laura, Thuerey, Nils · 2020 · ACM Transactions on Graphics
Cited by 201

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Our work explores temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationships in the generated data are much less explored. Natural temporal changes are crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as L 2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temporal cycle consistencies. In contrast, we focus on improving learning objectives and propose a temporally self-supervised algorithm. For both tasks, we show that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effectively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. Additionally, we propose a first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirm the rankings computed with these metrics. Code, data, models, and results are provided at https://github.com/thunil/TecoGAN.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3386569.3392457

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 45

  1. Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. 2018. Recycle-GAN: Unsupervised Video Retargeting. In The European Conference on Computer Vision (ECCV).
  2. Dina Bashkirova, Ben Usman, and Kate Saenko. 2018. Unsupervised video-to-video translation. arXiv preprint arXiv:1806.03698 (2018). · arXiv
  3. 10.1109/CVPR.2018.00652 · DOI · Paper page
  4. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations. https://openreview.net/forum?id=B1xsqj09Fm
  5. Jose Caballero, Christian Ledig, Andrew P Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation.. In CVPR, Vol. 1. 7.
  6. (CC) Blender Foundation | mango.blender.org. 2011. Tears of Steel. https://mango.blender.org/. Online; accessed 15 Nov. 2018.
  7. 10.1109/ICCV.2017.126 · DOI · Paper page
  8. Yang Chen, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. 2019. Mocycle-GAN: Unpaired Video-to-Video Translation. arXiv preprint arXiv:1908.09514 (August 2019). · arXiv
  9. 10.1111/cgf.13511 · DOI
  10. 10.5555/1763974.1764031 · doi:10.5555/1763974.1764031
  11. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
  12. Ishaan Gulrajani Faruk Ahmed Martin Arjovsky Vincent Dumoulin and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in neural information processing systems. 5767--5777.
  13. 10.1109/CVPR.2019.00402 · DOI · Paper page
  14. 10.1145/3197517.3201365 · DOI · Paper page
  15. 10.1145/3306346.3323006 · DOI
  16. 10.1109/CVPR.2018.00340 · DOI · Paper page
  17. 10.1145/3355089.3356557 · DOI
  18. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017). · arXiv
  19. 10.1145/3272127.3275065 · DOI
  20. 10.1145/3355089.3356560 · DOI
  21. Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate superresolution. In IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2. 5.
  22. 10.1109/ICCV.2015.68 · DOI · Paper page
  23. 10.1109/CVPR.2011.5995614 · DOI · Paper page
  24. 10.1109/ICCV.2017.274 · DOI · Paper page
  25. 10.1109/CVPR.2019.00470 · DOI · Paper page
  26. 10.1109/ICCV.2017.304 · DOI · Paper page
  27. Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. 2019. NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  28. Suraj Nair, Mohammad Babaeizadeh, Chelsea Finn, Sergey Levine, and Vikash Kumar. 2018. Time reversal as self-supervision. arXiv preprint arXiv:1810.01128 (2018). · arXiv
  29. 10.1145/3343031.3350864 · DOI
  30. Eduardo Pérez-Pellitero, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Schölkopf. 2018. Photorealistic Video Super Resolution. arXiv preprint arXiv:1807.07930 (2018). · arXiv
  31. 10.1007/978-3-319-45886-1_3 · DOI · Paper page
  32. 10.1109/ICCV.2017.481 · DOI · Paper page
  33. Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-Recurrent Video Super-Resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).
  34. 10.1109/CVPR.2016.207 · DOI · Paper page
  35. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
  36. 10.1145/3197517.3201333 · DOI · Paper page
  37. Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. 2017. Detail-Revealing Deep Video Super-Resolution. In The IEEE International Conference on Computer Vision (ICCV).
  38. 10.1145/3072959.3073633 · DOI
  39. 10.1109/TIP.2018.2836316 · DOI · Paper page
  40. Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Guilin Liu Andrew Tao Jan Kautz and Bryan Catanzaro. 2018a. Video-to-Video Synthesis. In Advances in Neural Information Processing Systems (NIPS).
  41. 10.1109/CVPR.2019.00267 · DOI · Paper page
  42. 10.1145/3306346.3323024 · DOI · Paper page
  43. 10.1145/3306346.3322956 · DOI
  44. 10.1145/3197517.3201304 · DOI · Paper page
  45. 10.1109/CVPR.2019.00824 · DOI · Paper page

Cited by

See all 201

Metadata

Authors
Chu, M., Xie, Y., Mayer, J., Leal-Taixé, L., & Thuerey, N.
Venue
ACM Transactions on Graphics
Published
2020-08-12
Volume
39
Issue
4
ISSN
  • 0730-0301 (ISSN-L)
  • 1557-7368
OpenAlex
W2996880115
Cited by
201
References
12
FWCI
7.98
Type
journal article
Language
en
Open access
green
Funders
  • European Research Council (realFlow (StG- 2015-637014))
  • Alexander von Humboldt-Stiftung (Sofja Kovalevskaja Award)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning temporal coherence via self-supervision for GAN-based video generation. https://app.cochat.ai/papers/10.1145/3386569.3392457
The paper
Chu, M., Xie, Y., Mayer, J., Leal-Taixé, L., & Thuerey, N. (2020). Learning temporal coherence via self-supervision for GAN-based video generation. ACM Transactions on Graphics. https://doi.org/10.1145/3386569.3392457

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.