CoChat
Sign Up
PreprintCited by 204Generative Adversarial Networks and Image SynthesisArtificial Intelligence in GamesDigital Media Forensic Detection

Towards Accurate Generative Models of Video: A New Metric & Challenges

Open access
Unterthiner, Thomas, van Steenkiste, Sjoerd, Kurach, Karol, Marinier, Raphael, Michalski, Marcin, Gelly, Sylvain·2018·arXiv
DOI 10.48550/arxiv.1812.01717arXiv 1812.01717
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent attempts at formulating generative models of video have had some success, current progress is hampered by (1) the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples, and (2) the wide gap between purely synthetic video data sets and challenging real-world data sets in terms of complexity. To this extent we propose Fréchet Video Distance (FVD), a new metric for generative models of video, and StarCraft 2 Videos (SCV), a benchmark of game play from custom starcraft 2 scenarios that challenge the current capabilities of generative models of video. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos, and provide initial benchmark results on SCV.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1812.01717

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In an evaluation of various video generative models trained on the TikTok dataset, video sample (a) of the poorest quality cannot be effectively distinguished from samples (b), (c), and (d) based on FID or VBench metrics, while FVMD aligns better with assessed video quality and motion consistency.

    “For instance, video sample (a), which is of the poorest quality, cannot be effectively distinguished from samples (b), (c), and (d) based on the FID or VBench metrics. In contrast, the FVMD metric aligns better with the assessed video quality and motion consistency.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Unterthiner et al., 2018

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • FVD aligns better with human judgments than KVD.

    “The authors found that FVD aligns better with human judgments than KVD.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Unterthiner et al. 2018

  • FVD is sensitive not only to spatial degradation but also to temporal aberrations such as swapping of video frames.

    “The authors demonstrate that the FVD measure is not only sensitive to spatial degradation (different kinds of noise), but also to temporal aberrations such as swapping of video frames.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Unterthiner et al. 2018

  • Human studies can be used to measure how well automated metrics align with human preferences.

    “Human studies can also be used to measure how well the automated metrics align with human preferences, checking if human judgments agree with metric results or differ when assessing similar videos (Unterthiner et al., 2018; Huang et al., 2024; Liu et al., 2024).”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Unterthiner et al., 2018

  • FVMD is more aligned with human preferences for temporal consistency compared to FVD.

    “We find this metric to be more aligned with human preferences for temporal consistency compared to FVD [31].”

    From GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving (2025) · cites this paper as [31]

    The Human Evaluation section (p. 7) reports that FVD agrees well with human judgments and outperforms the other metrics tested. A search for FVMD and a comparison of temporal-consistency preferences found no such comparison in our copy, so it cannot establish whether FVMD aligns better than FVD. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. [1][1] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine. Stochastic variational video prediction. International Conference on Learning Representations (ICLR) , 2017. 2, 4, 5, 6, 13 -
  2. [2][2] M. Bi´ nkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD GANs. International Conference on Learning Representations (ICLR) , 2018. 3, 6 -
  3. [3][3] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv , 2018. 1 -
  4. [4][4] W. Byeon, Q. Wang, R. K. Srivastava, P. Koumoutsakos, P. Vlachas, Z. Wan, T. Sapsis, F. Raue, S. Palacio, T. Breuel, et al. Contextvp: Fully context-aware video prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 1122-1126, 2018. 2 -
  5. [5][5] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. 3, 6 -
  6. [6][6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2009. 2, 6 -
  7. [7][7] E. Denton and R. Fergus. Stochastic video generation with a learned prior. International Conference on Machine Learning (ICML) , 2018. 2, 6 -
  8. [8][8] D. Dowson and B. Landau. The frchet distance between multivariate normal distributions. Journal of Multivariate Analysis , 12(3):450 - 455, 1982. 2 -
  9. [9][9] F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning , pages 344-356, 2017. 1, 6, 11 -
  10. [10][10] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in Neural Information Processing Systems (NIPS) , 2016. 1, 3, 6 -
  11. [11][11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems (NIPS) , 2014. 1 -
  12. [12][12] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch¨ olkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research , 13(Mar):723-773, 2012. 3 -
  13. [13][13] E. Haller and M. Leordeanu. Unsupervised object segmentation in video by efficient selection of highly probable positive features. IEEE International Conference on Computer Vision (ICCV) , 2017. 1 -
  14. [14][14] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems (NIPS) , 2017. 2, 3, 6 -
  15. [15][15] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. 1 -
  16. [17][17] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Imageto-image translation with conditional adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. 1 -
  17. [18][18] H. Jiang, D. Sun, V. Jampani, M.-H. Yang, E. LearnedMiller, and J. Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 1 -
  18. [19][19] N. Kalchbrenner, A. van den Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu. Video pixel networks. International Conference on Machine Learning (ICML) , 2017. 2 -
  19. [20][20] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. International Conference on Learning Representations (ICLR) , 2018. 1 -
  20. [21][21] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset. arXiv , 2017. 3, 11 -
  21. [24][24] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. arXiv , 2018. 2, 6 -
  22. [25][25] A. Lerer, S. Gross, and R. Fergus. Learning physical intuition of block towers by example. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48 , pages 430-438. JMLR. org, 2016. 1 -
  23. [26][26] B. Lotter, G. Kreiman, and D. Cox. Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning. International Conference on Learning Representations (ICLR) , 2017. 1 -
  24. [27][27] P. Luc, C. Couprie, S. Chintala, and J. Verbeek. Semantic segmentation using adversarial networks. arXiv , 2016. 1 -
  25. [28][28] M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are gans created equal? a large-scale study. Advances in Neural Information Processing Systems (NIPS) , 2018. 6 -
  26. [29][29] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. International Conference on Learning Representations (ICLR) , 2016. 1 -
  27. [31][31] K. Preuer, P. Renz, T. Unterthiner, S. Hochreiter, and G. Klambauer. Frchet chemnet distance: A metric for generative models for molecules in drug discovery. Journal of Chemical Information and Modeling , 58(9):1736-1741, 2018. 2 -
  28. [32][32] I. Radosavovic, P. Dollr, R. Girshick, G. Gkioxari, and K. He. Data distillation: Towards omni-supervised learning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 1 -
  29. [33][33] M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv , 2014. 1 -
  30. [34][34] M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. International Conference on Computer Vision (ICCV) , 2017. 2 -
  31. [35][35] M. S. Sajjadi, B. Sch¨ olkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. International Conference on Computer Vision (ICCV) , 2017. 1 -
  32. [36][36] C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: a local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition (ICPR) , volume 3, pages 32-36. IEEE, 2004. 3 -
  33. [37][37] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv , 2012. 3 -
  34. [39][39] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. 2 -
  35. [40][40] L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. International Conference on Learning Representations (ICLR) , 2016. 6 -
  36. [41][41] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 2 -
  37. [42][42] T. Unterthiner, B. Nessler, C. Seward, G. Klambauer, M. Heusel, H. Ramsauer, and S. Hochreiter. Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields. International Conference on Learning Representations (ICLR) , 2018. 6 -
  38. [43][43] A. Vaswani, S. Bengio, E. Brevdo, F. Chollet, A. N. Gomez, S. Gouws, L. Jones, L. Kaiser, N. Kalchbrenner, N. Parmar, R. Sepassi, N. Shazeer, and J. Uszkoreit. Tensor2tensor for neural machine translation. arXiv , 2018. 13 -
  39. [44][44] R. Villegas, D. Erhan, H. Lee, et al. Hierarchical long-term video prediction without supervision. In International Conference on Machine Learning , pages 6033-6041, 2018. 2 -
  40. [45][45] O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. K¨ uttler, J. Agapiou, J. Schrittwieser, et al. StarCraft II: A new challenge for reinforcement learning. arXiv , 2017. 2, 3, 4 -
  41. [46][46] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS) , 2016. 2 -
  42. [47][47] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, N. Yakovenko, A. Tao, J. Kautz, and B. Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems , pages 1152-1164, 2018. 2 -
  43. [48][48] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 2004. 2, 3 -
  44. [49][49] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 3

Cited by

Sorted by citations

Authors
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., & Gelly, S.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1812.01717
arXiv
1812.01717
OpenAlex
W4297750695
Cited by
204
References
2
Citations by year
2026: 1 · 2025: 20 · 2024: 47 · 2023: 56 · 2022: 17 · 2021: 35 · 2020: 16 · 2019: 11 · 2018: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
video quality metrics, temporal coherence, sample diversity, video generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review