Unsupervised Video Representation Learning by Bidirectional Feature Prediction
Open accessAbstract
This paper introduces a novel method for self-supervised video representation learning via feature prediction. In contrast to the previous methods that focus on future feature prediction, we argue that a supervisory signal arising from unobserved past frames is complementary to one that originates from the future frames. The rationale behind our method is to encourage the network to explore the temporal structure of videos by distinguishing between future and past given present observations. We train our model in a contrastive learning framework, where joint encoding of future and past provides us with a comprehensive set of temporal hard negatives via swapping. We empirically show that utilizing both signals enriches the learned representations for the downstream task of action recognition. It out-performs independent prediction of future and past.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current perceptual hashing schemes often require manual annotation of video samples and their similarity relationships, and these methods may not have enough generalization capability to deal with massive video and emerging video manipulations.
“But, current perceptual hashing schemes often require manual annotation of video samples and their similarity relationships, and these methods may not have enough generalization capability to deal with massive video and emerging video manipulations [16].”
From Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning (2023) · cites this paper as [16]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Learning Representations for Automatic Colorization
Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_35 · Cited by 946 · see citing works
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · see citing works
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 40
- Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015.
- Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020.
- Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
- Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In CVPR, 2020.
- Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. arXiv, abs/2002.05709, 2020. · arXiv
- Dave Epstein, Boyuan Chen, and Carl Vondrick. Oops! pre- dicting unintentional action in video. In CVPR, 2020.
- Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learn- ing with odd-one-out networks. In CVPR, 2017.
- David F. Fouhey, Weicheng Kuo, Alexei A. Efros, and Jiten- dra Malik. From lifestyle vlogs to everyday interactions. In CVPR, 2018.
- Tengda Han, Weidi Xie, and Andrew Zisserman. Video rep- resentation learning by dense predictive coding. In ICCV Workshop, 2019.
- Tengda Han, Weidi Xie, and Andrew Zisserman. Memory- augmented dense predictive coding for video representation learning. In ECCV, 2020.
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. arXiv, abs/1911.05722, 2019. · arXiv
- Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson. Learning visual groups from co-occurrences in space and time. In ICLR Workshop, 2016.
- Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to egomotion. In ICCV, 2015.
- Dinesh Jayaraman and Kristen Grauman. Slow and steady feature analysis: Higher order temporal coherence in video. In CVPR, 2016.
- Longlong Jing and Yingli Tian. Self-supervised spatiotem- poral feature learning via video rotation prediction. arXiv, abs/1811.11387, 2018. · arXiv
- Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv, abs/1705.06950, 2017. · arXiv
- Dahun Kim, Donghyeon Cho, and In So Kweon. Self- supervised video representation learning with space-time cu- bic puzzles. In AAAI, 2019.
- Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
- Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In ICCV, 2011.
- Zihang Lai and Weidi Xie. Self-supervised learning for video correspondence flow. In BMVC, 2019.
- Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised representation learning by sort- ing sequences. In ICCV, 2017.
- William Lotter, Gabriel Kreiman, and David D. Cox. Deep predictive coding networks for video prediction and unsuper- vised learning. In ICLR, 2017.
- Michaël Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv, abs/1511.05440, 2016. · arXiv
- Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv, abs/1912.01991, 2019. · arXiv
- Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
- Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
- AJ Piergiovanni, Anelia Angelova, and Michael S. Ryoo. Evolving losses for unsupervised video representation learn- ing. In CVPR, 2020.
- Ben Poole, Sherjil Ozair, Aaron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual infor- mation. In ICML, 2019.
- Khurram Soomro, Amir Roshan Zamir, Mubarak Shah, Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv, abs/1212.0402, 2012. · arXiv
- Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. arXiv, abs/1906.05849, 2019. · arXiv
- Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In ICLR, 2020.
- Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv, abs/1807.03748, 2018. · arXiv
- Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. An- ticipating visual representations from unlabeled video. In CVPR, 2016.
- Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurIPS, 2016.
- Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by col- orizing videos. In ECCV, 2018.
- Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learn- ing correspondence from the cycle-consistency of time. In CVPR, 2019.
- Donglai Wei, Joseph J. Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In CVPR, 2018.
- Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, 2019.
- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014.
- Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, 2016.
Cited by
Sorted by citations
Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning
Zhao, G., Qin, C., Luo, X., Zhang, X., & Chang, C.-C. (2023) · Proceedings of the 2023 ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3577163.3595110 · Cited by 1 · see citing works
“But, current perceptual hashing schemes often require manual annotation of video samples and their similarity relationships, and these methods may not have enough generalization capability to deal with massive video and emerging video manipulations [16].”
Background
- Authors
- Behrmann, N., Gall, J., & Noroozi, M.
- Affiliations
- Robert Bosch (United States) (US)
- ORCID
- Noroozi, Mehdi
- Venue
- 2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2021-01
- Pages
- 1669-1678
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W3099331408
- Cited by
- 28
- References
- 5
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 11 · 2022: 10 · 2021: 4
- FWCI
- 2.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (EXC-2070, 390732324)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- video representation learning, self-supervised learning, future frame prediction, contrastive learning, temporal structure, human action recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
