Real-Time Action Recognition With Deeply Transferred Motion Vector CNNs
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The two-stream CNNs prove very successful for video based action recognition. However the classical two-stream CNNs are time costly, mainly due to the bottleneck of calculating optical flows. In this paper, we propose a two-stream based real-time action recognition approach by using motion vector to replace optical flow. Motion vectors are encoded in video stream and can be extracted directly without extra calculation. However directly training CNN with motion vectors degrades accuracy severely due to the noise and the lack of fine details in motion vectors. In order to relieve this problem, we propose four training strategies which leverage the knowledge learned from optical flow CNN to enhance the accuracy of motion vector CNN. Our insight is that motion vector and optical flow share inherent similar structures which allows us to transfer knowledge from one domain to another. To fully utilize the knowledge learned in optical flow domain, we develop deeply transferred motion vector CNN. Experimental results on various datasets show the effectiveness of our training strategies. Our approach is significantly faster than optical flow based approaches and achieves processing speed of 390.7 frames per second, surpassing real-time requirement. We release our model and code to facilitate further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Video Google: a text retrieval approach to object matching in videos
Sivic, & Zisserman (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238663 · Cited by 6,367 · cited 3×
“Further researches show its potential for video retrieval [6] and action recognition [2].”
BackgroundMethodLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965 · cited 3×
“posed real-time action recognition algorithm: UCF101 [9], HMDB51 [10], and THUMOS14 [15].”
BackgroundAction Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Self-citationRepresenting and Recognizing the Visual Appearance of Materials Using Three-Dimensional Textons
Leung, T., & Malik, J. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011126920638 · Cited by 1,498 · cited 1×
“BoVW framework is originally proposed for image classification [16].”
MethodImage Classification with the Fisher Vector: Theory and Practice
Sánchez, J., Perronnin, F., Mensink, T., & Verbeek, J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0636-x · Cited by 1,498 · cited 3×
“One popular method along this line is the improved-dense-trajectories (iDT) [2] and Fisher Vector encoding [7].”
MethodAction recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299059 · Cited by 1,194
Self-citationCross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530 · cited 2×
“Among these approaches, the most relevant works to us are [13], [40], and [32].”
BackgroundMethodUnifying distillation and privileged information
Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015) · arXiv · doi:10.48550/arxiv.1511.03643 · Cited by 270 · cited 1×
“[37] unified two style of knowledge transfer, privileged information and knowledge distillation, into generalized distillation.”
Background
Cited by
A Comprehensive Survey of Vision-Based Human Action Recognition Methods
Zhang, H.-B., Zhang, Y.-X., Zhong, B., Lei, Q., Yang, L., Du, J.-X., & Chen, D.-S. (2019) · Sensors · doi:10.3390/s19051005 · Cited by 503
“Many deep learning-based action representation methods have been proposed in the field of human action recognition [19,20,42,45,46,82].”
BackgroundFine-grained activity classification in assembly based on multi-visual modalities
Chen, H., Zendehdel, N., Leu, M. C., & Yin, Z. (2024) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02152-x · Cited by 47
Metadata
- Authors
- Zhang, B., Wang, L., Wang, Z., Qiao, Y., & Wang, H.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2018-05
- Volume
- 27
- Issue
- 5
- Pages
- 2326-2339
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W2782776028
- Cited by
- 178
- References
- 15
- FWCI
- 10.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.