CoChat
Sign Up
PreprintCited by 1Image and Video Quality AssessmentGenerative Adversarial Networks and Image SynthesisDigital Media Forensic Detection

The DOI matches its DataCite record; the references it cites are not checked yet.

Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos

Open access
Liu, Jiahe, Qu, Youran, Yan, Qi, Zeng, Xiaohui, Wang, Lele, Liao, Renjie·2024·arXiv
DOI 10.48550/arxiv.2407.16124arXiv 2407.16124
Open this paper in CoChat — free accountOpen PDF

Abstract

Significant advancements have been made in video generative models recently. Unlike image generation, video generation presents greater challenges, requiring not only generating high-quality frames but also ensuring temporal consistency across these frames. Despite the impressive progress, research on metrics for evaluating the quality of generated videos, especially concerning temporal and motion consistency, remains underexplored. To bridge this research gap, we propose Fréchet Video Motion Distance (FVMD) metric, which focuses on evaluating motion consistency in video generation. Specifically, we design explicit motion features based on key point tracking, and then measure the similarity between these features via the Fréchet distance. We conduct sensitivity analysis by injecting noise into real videos to verify the effectiveness of FVMD. Further, we carry out a large-scale human study, demonstrating that our metric effectively detects temporal noise and aligns better with human perceptions of generated video quality than existing metrics. Additionally, our motion features can consistently improve the performance of Video Quality Assessment (VQA) models, indicating that our approach is also applicable to unary video quality evaluation. Code is available at https://github.com/ljh0v0/FMD-frechet-motion-distance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.16124

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • FVMD is more aligned with human preferences for temporal consistency compared to FVD.

    “We find this metric to be more aligned with human preferences for temporal consistency compared to FVD [31].”

    From GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving (2025) · cites this paper as [30]

    The Results section reports that FVMD correlates more closely with human judgments than the comparison metrics, including FVD. However, the Human Study (Section 4.4) describes ratings of generated videos, not ratings specifically of temporal consistency, so that narrower description of human preferences is not established. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Human studies can be used to measure how well automated metrics align with human preferences.

    “Human studies can also be used to measure how well the automated metrics align with human preferences, checking if human judgments agree with metric results or differ when assessing similar videos (Unterthiner et al., 2018; Huang et al., 2024; Liu et al., 2024).”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Liu et al., 2024

  • FVMD assesses motion consistency by analyzing speed and acceleration patterns, assuming smooth motions should follow physical laws and avoid abrupt changes.

    “FVMD assesses motion consistency by analyzing speed and acceleration patterns, assuming smooth motions should follow physical laws and avoid abrupt changes.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Liu et al. 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 1

Research works printed but not linked: 50

  1. Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S. Stochastic variational video prediction. arXiv preprint arXiv:1710.11252, 2017. · arXiv
  2. Balaji, Y., Min, M. R., Bai, B., Chellappa, R., and Graf, H. P. Conditional gan with discriminative filter generation for text-to-video synthesis. In IJCAI, volume 1, pp. 2, 2019.
  3. Blattmann, A., Milbich, T., Dorkenwald, M., and Ommer, B. ipoke: Poking a still image for controlled stochastic video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14707-14717, 2021.
  4. Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models as world simulators. preprint, 2024.
  5. Carreira, J. and Zisserman, A. Quo vadis, action recogni- tion? a new model and the kinetics dataset. In proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
  6. Castrejon, L., Ballas, N., and Courville, A. Improved condi- tional vrnns for video prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7608-7617, 2019.
  7. Dalal, N. and Triggs, B. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp. 886-893. Ieee, 2005.
  8. Denton, E. and Fergus, R. Stochastic video generation with a learned prior. In International conference on machine learning, pp. 1174-1183. PMLR, 2018.
  9. Dorkenwald, M., Milbich, T., Blattmann, A., Rombach, R., Derpanis, K. G., and Ommer, B. Stochastic image- to-video synthesis using cinns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3742-3753, 2021.
  10. Dowson, D. and Landau, B. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455, 1982.
  11. Ebert, F., Finn, C., Lee, A. X., and Levine, S. Self- supervised visual planning with temporal skip connec- tions. CoRL, 12:16, 2017.
  12. Franceschi, J.-Y., Delasalles, E., Chen, M., Lamprier, S., and Gallinari, P. Stochastic latent residual video prediction. In International Conference on Machine Learning, pp. 3233-3246. PMLR, 2020.
  13. Ge, S., Hayes, T., Yang, H., Yin, X., Pang, G., Jacobs, D., Huang, J.-B., and Parikh, D. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
  14. He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. La- tent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221, 2022. · arXiv
  15. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
  16. Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a. · arXiv
  17. Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022b.
  18. Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205.15868, 2022. · arXiv
  19. Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., and Bo, L. Animate anyone: Consistent and controllable image-to- video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023. · arXiv
  20. Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Com- prehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982, 2023. · arXiv
  21. Jafarian, Y. and Park, H. S. Self-supervised 3d representa- tion learning of dressed humans from social media videos. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2022.
  22. Lee, A. X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., and Levine, S. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018. · arXiv
  23. Li, D., Jiang, T., and Jiang, M. Quality assessment of in-the-wild videos. In Proceedings of the 27th ACM international conference on multimedia, pp. 2351-2359, 2019.
  24. Liu, X., Min, X., Zhai, G., Li, C., Kou, T., Sun, W., Wu, H., Gao, Y., Cao, Y., Zhang, Z., et al. Ntire 2024 quality assessment of ai-generated content challenge. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6337-6362, 2024.
  25. Lu, Y., Li, X., Pei, Y., Yuan, K., Xie, Q., Qu, Y., Sun, M., Zhou, C., and Chen, Z. Kvq: Kaleidoscope video quality assessment for short-form videos. arXiv preprint arXiv:2402.07220, 2024. · arXiv
  26. Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., and Ritter, H. Video diffusion models: A survey. arXiv preprint arXiv:2405.03150, 2024. · arXiv
  27. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
  28. Saito, M., Matsumoto, E., and Saito, S. Temporal generative adversarial nets with singular value clipping. In Proceed- ings of the IEEE international conference on computer vision, pp. 2830-2839, 2017.
  29. Saito, M., Saito, S., Koyama, M., and Kobayashi, S. Train sparsely, generate densely: Memory-efficient unsuper- vised training of high-resolution temporal gan. Interna- tional Journal of Computer Vision, 128(10):2586-2606, 2020.
  30. Sand, P. and Teller, S. Particle video: Long-range motion estimation using point trajectories. International journal of computer vision, 80:72-91, 2008.
  31. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022. · arXiv
  32. Skorokhodov, I., Tulyakov, S., and Elhoseiny, M. Stylegan- v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3626-3636, 2022.
  33. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
  34. Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
  35. Sun, W., Min, X., Lu, W., and Zhai, G. A deep learn- ing based no-reference quality assessment model for ugc videos. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 856-865, 2022.
  36. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
  37. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. Learning spatiotemporal features with 3d convolu- tional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497, 2015.
  38. Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
  39. Voleti, V., Jolicoeur-Martineau, A., and Pal, C. Mcvd- masked conditional video diffusion for prediction, gener- ation, and interpolation. Advances in neural information processing systems, 35:23371-23385, 2022.
  40. Vondrick, C., Pirsiavash, H., and Torralba, A. Generating videos with scene dynamics. Advances in neural infor- mation processing systems, 29, 2016.
  41. Wang, T., Li, L., Lin, K., Lin, C.-C., Yang, Z., Zhang, H., Liu, Z., and Wang, L. Disco: Disentangled control for referring human dance generation in real world. arXiv e-prints, pp. arXiv-2307, 2023.
  42. Wang, T.-C., Liu, M.-Y., Tao, A., Liu, G., Kautz, J., and Catanzaro, B. Few-shot video-to-video synthesis. arXiv preprint arXiv:1910.12713, 2019. · arXiv
  43. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
  44. Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., and Duan, N. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on computer vision, pp. 720-736. Springer, 2022a.
  45. Wu, H., Chen, C., Hou, J., Liao, L., Wang, A., Sun, W., Yan, Q., and Lin, W. Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling. In European conference on computer vision, pp. 538-554. Springer, 2022b.
  46. Xu, Z., Zhang, J., Liew, J. H., Yan, H., Liu, J.-W., Zhang, C., Feng, J., and Shou, M. Z. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16498, 2023. · arXiv
  47. Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021. · arXiv
  48. Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
  49. Zheng, Y., Harley, A. W., Shen, B., Wetzstein, G., and Guibas, L. J. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19855-19865, 2023.
  50. Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., and Feng, J. Magicvideo: Efficient video generation with latent diffu- sion models. arXiv preprint arXiv:2211.11018, 2022. · arXiv

Metadata

Authors
Liu, J., Qu, Y., Yan, Q., Zeng, X., Wang, L., & Liao, R.
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.16124
arXiv
2407.16124
OpenAlex
W6948038273
Cited by
1
References
0
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research
  • Government of Canada
  • Vector Institute
  • Alliance de recherche numérique du Canada
  • Natural Sciences and Engineering Research Council of Canada (CRDPJ 543676-19, rgpin-2022-04636, rgpin-2019-05448, RGPIN-2019)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
motion consistency, temporal coherence, video quality assessment, keypoint tracking, motion features, Fréchet distance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review