CoChat
Sign Up
PreprintCited by 238Human Pose and Action RecognitionDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

The DOI matches its DataCite record.

VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples

Open access
Pan, Tian, Song, Yibing, Yang, Tianyu, Jiang, Wenhao, Liu, Wei·2021·arXiv
DOI 10.48550/arxiv.2103.05905arXiv 2103.05905
Open this paper in CoChat — free accountOpen PDF

Abstract

MoCo is effective for unsupervised image representation learning. In this paper, we propose VideoMoCo for unsupervised video representation learning. Given a video sequence as an input sample, we improve the temporal feature representations of MoCo from two perspectives. First, we introduce a generator to drop out several frames from this sample temporally. The discriminator is then learned to encode similar feature representations regardless of frame removals. By adaptively dropping out different frames during training iterations of adversarial learning, we augment this input sample to train a temporally robust encoder. Second, we use temporal decay to model key attenuation in the memory queue when computing the contrastive loss. As the momentum encoder updates after keys enqueue, the representation ability of these keys degrades when we use the current input sample for contrastive learning. This degradation is reflected via temporal decay to attend the input sample to recent keys in the queue. As a result, we adapt MoCo to learn video representations without empirically designing pretext tasks. By empowering the temporal robustness of the encoder and modeling the temporal decay of the keys, our VideoMoCo improves MoCo temporally based on contrastive learning. Experiments on benchmark datasets including UCF101 and HMDB51 show that VideoMoCo stands as a state-of-the-art video representation learning method.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2103.05905

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The drop mechanism can be used for contrastive learning and can effectively solve the problem of over-fitting.

    “Studies (Pan et al. 2021; Cheng et al. 2020) have shown that the drop mechanism can be used for contrastive learning and can effectively solve the problem of over-fitting.”

    From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Pan et al. 2021

    The Pretext Contrastive Learning section describes dropping video frames to create queries for contrastive training. The Configuration Evaluation section reports improved recognition performance with adversarial frame dropout, but does not show that it solves overfitting; it also reports that random dropout worsened performance. Page numbers are unavailable in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 49

  1. Uta Buchler, Biagio Brattoli, and Bjorn Ommer. Improving spatiotemporal self-supervision by deep reinforcement learn- ing. In European Conference on Computer Vision, 2018.
  2. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learn- ing of visual representations. In International Conference on Machine Learning, 2020.
  3. Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle- consistency learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  4. Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learn- ing with odd-one-out networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
  5. Chuang Gan, Boqing Gong, Kun Liu, Hao Su, and Leonidas J Guibas. Geometry guided convolutional neural networks for self-supervised video representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  6. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Neural Information Processing Systems, 2020.
  7. Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial net- works for diverse and limited data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
  8. Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensional- ity reduction by learning an invariant mapping. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2006.
  9. Tengda Han, Weidi Xie, and Andrew Zisserman. Video rep- resentation learning by dense predictive coding. In Proceed- ings of the IEEE International Conference on Computer Vi- sion Workshops, 2019.
  10. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  11. Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Jun- yuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2019.
  12. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversar- ial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
  13. Simon Jenni, Givi Meishvili, and Paolo Favaro. Video repre- sentation learning by recognizing temporal transformations. In European Conference on Computer Vision, 2020.
  14. H Jhuang, H Garrote, E Poggio, T Serre, and T Hmdb. A large video database for human motion recognition. In IEEE/CVF International Conference on Computer Vision, 2011.
  15. Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video ro- tation prediction. arXiv preprint arXiv:1811.11387, 2018. · arXiv
  16. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950. · arXiv
  17. Dahun Kim, Donghyeon Cho, and In So Kweon. Self- supervised video representation learning with space-time cu- bic puzzles. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, 2019.
  18. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European Conference on Computer Vision, 2020.
  19. Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Re- visiting self-supervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  20. Bruno Korbar, Du Tran, and Lorenzo Torresani. Coopera- tive learning of audio and video models from self-supervised synchronization. In Neural Information Processing Systems, 2018.
  21. Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised representation learning by sort- ing sequences. In IEEE/CVF International Conference on Computer Vision, 2017.
  22. Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):105-117, 1988.
  23. Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via a mutual encoder- decoder with feature equalizations. In European Conference on Computer Vision, 2020.
  24. Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma, Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. arXiv preprint arXiv:2001.00294, 2020. · arXiv
  25. SC Martin Arjovsky and Leon Bottou. Wasserstein gener- ative adversarial networks. In International Conference on Machine Learning, 2017.
  26. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  27. Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuf- fle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, 2016.
  28. Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f- gan: Training generative neural samplers using variational divergence minimization. In Neural Information Processing Systems, 2016.
  29. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. · arXiv
  30. Mohammad Sabokrou, Mohammad Khalooei, and Ehsan Adeli. Self-supervised representation learning via neighborhood-relational encoding. In IEEE/CVF Inter- national Conference on Computer Vision, 2019.
  31. Yibing Song, Chao Ma, Lijun Gong, Jiawei Zhang, Ryn- son WH Lau, and Ming-Hsuan Yang. Crest: Convolutional residual learning for visual tracking. In IEEE/CVF Interna- tional Conference on Computer Vision, 2017.
  32. Yibing Song, Chao Ma, Xiaohe Wu, Lijun Gong, Linchao Bao, Wangmeng Zuo, Chunhua Shen, Rynson WH Lau, and Ming-Hsuan Yang. Vital: Visual tracking via adversarial learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  33. Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative ad- versarial network. In IEEE/CVF International Conference on Computer Vision, 2017.
  34. Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudi- nov. Unsupervised learning of video representations using lstms. In International Conference on Machine Learning, 2015.
  35. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In IEEE/CVF Interna- tional Conference on Computer Vision, 2019.
  36. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. In European Conference on Com- puter Vision, 2020.
  37. Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by col- orizing videos. In European Conference on Computer Vision, 2018.
  38. Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  39. Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self- supervised video representation learning by pace prediction. In European Conference on Computer Vision, 2020.
  40. Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li. Unsupervised deep tracking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  41. Ning Wang, Wengang Zhou, Yibing Song, Chao Ma, Wei Liu, and Houqiang Li. Unsupervised deep representation learning for real-time tracking. International Journal of Computer Vision, 2020.
  42. Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learn- ing correspondence from the cycle-consistency of time. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  43. Yinglong Wang, Yibing Song, Chao Ma, and Bing Zeng. Re- thinking image deraining via rain streaks and vapors. In Eu- ropean Conference on Computer Vision, 2020.
  44. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2018.
  45. Shi Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Neural Information Processing Systems, 2015.
  46. Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  47. Ceyuan Yang, Yinghao Xu, Bo Dai, and Bolei Zhou. Video representation learning with visual tempo consistency. arXiv preprint arXiv:2006.15489, 2020. · arXiv
  48. Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixi- ang Ye. Video playback rate perception for self-supervised spatio-temporal representation learning. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2020.
  49. Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2020.

Cited by

Sorted by citations

Metadata

Authors
Pan, T., Song, Y., Yang, T., Jiang, W., & Liu, W.
Affiliations
Tencent (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
11200-11209
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.2103.05905
arXiv
2103.05905
OpenAlex
W3176780013
Cited by
238
References
0
Citations by year
2026: 12 · 2025: 31 · 2024: 44 · 2023: 82 · 2022: 57 · 2021: 12
FWCI
15.64
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
contrastive learning, temporal robustness, temporal decay, momentum encoder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review