CoChat
Sign Up
PreprintCited by 1,861Human Pose and Action RecognitionMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning

Unsupervised Learning of Video Representations using LSTMs

Open access
Srivastava, Nitish, Mansimov, Elman, Salakhutdinov, Ruslan·2015·arXiv
DOI 10.48550/arxiv.1502.04681arXiv 1502.04681
Open this paper in CoChat — free accountOpen PDF

Abstract

We use multilayer Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We try to visualize and interpret the learned features. We stress test the model by running it on longer time scales and on out-of-domain data. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only a few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1502.04681

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The proposed solution is the first time motion prediction with cross-dimensional learning has been used for self-supervised learning in the context of HAR.

    “This is the first time that motion prediction with cross-dimensional learning has been used for self-supervised learning in the context of HAR.”

    From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [23]

  • The cross-dimensional aspect of the proposed motion prediction pretext task results in more effective learning than standard motion prediction.

    “While motion prediction has been used in the field of computer vision for self-supervised learning [23, 26], our model is novel in that the cross-dimensional aspect results in more effective learning.”

    From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [23]

Cites

Flagged first, then by citations

1 of the 30 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Srivastava, N., Mansimov, E., & Salakhutdinov, R.
Affiliations
University of Toronto (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
Pages
843-852
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1502.04681
arXiv
1502.04681
OpenAlex
W2116435618
Cited by
1,861
References
30
Citations by year
2026: 3 · 2025: 56 · 2024: 128 · 2023: 148 · 2022: 183 · 2021: 279 · 2020: 308 · 2019: 286 · 2018: 238 · 2017: 146 · 2016: 62 · 2015: 23 · 2012: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Funders
  • Raytheon Company
  • Nvidia
  • Samsung
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
video representation learning, unsupervised learning, long short-term memory networks, LSTM encoder-decoder, video reconstruction, future frame prediction, perceptual features, feature visualization, human action recognition, transfer learning, few-shot learning, out-of-domain generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review