The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Beyond Frame-level CNN: Saliency-Aware 3-D CNN With LSTM for Video Action Recognition
Abstract
Human activity recognition in videos with convolutional neural network (CNN) features has received increasing attention in multimedia understanding. Taking videos as a sequence of frames, a new record was recently set on several benchmark datasets by feeding frame-level CNN sequence features to long short-term memory (LSTM) model for video activity recognition. This recurrent model-based visual recognition pipeline is a natural choice for perceptual problems with time-varying visual input or sequential outputs. However, the above-mentioned pipeline takes frame-level CNN sequence features as input for LSTM, which may fail to capture the rich motion information from adjacent frames or maybe multiple clips. Furthermore, an activity is conducted by a subject or multiple subjects. It is important to consider attention that allows for salient features, instead of mapping an entire frame into a static representation. To tackle these issues, we propose a novel pipeline, saliency-aware three-dimensional (3-D) CNN with LSTM, for video action recognition by integrating LSTM with salient-aware deep 3-D CNN features on videos shots. Specifically, we first apply saliency-aware methods to generate saliency-aware videos. Then, we design an end-to-end pipeline by integrating 3-D CNN with LSTM, followed by a time series pooling layer and a softmax layer to predict the activities. Noticeably, we set a new record on two benchmark datasets, i.e., UCF101 with 13 320 videos and HMDB-51 with 6766 videos. Our method outperforms the state-of-the-art end-to-end methods of action recognition by 3.8% and 3.2%, respectively on above two datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
Learning realistic human actions from movies
Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587756 · Cited by 3,444
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Behavior Recognition via Sparse Spatio-Temporal Features
Dollar, P., Rabaud, V., Cottrell, G., & Belongie, S. (2006) · 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance · doi:10.1109/vspets.2005.1570899 · Cited by 2,460
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,861
A 3-dimensional sift descriptor and its application to action recognition
Scovanner, P., Ali, S., & Shah, M. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291311 · Cited by 1,596
Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks
Sun, L., Jia, K., Yeung, D.-Y., & Shi, B. E. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.522 · Cited by 577
Saliency-aware geodesic video object segmentation
Wang, W., Shen, J., & Porikli, F. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298961 · Cited by 524
Robust Joint Graph Sparse Coding for Unsupervised Spectral Feature Selection
Zhu, X., Li, X., Zhang, S., Ju, C., & Wu, X. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2521602 · Cited by 339
Block-Row Sparse Multiview Multilabel Learning for Image Classification
Zhu, X., Li, X., & Zhang, S. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2403356 · Cited by 324
Efficient Motion and Disparity Estimation Optimization for Low Complexity Multiview Video Coding
Pan, Z., Zhang, Y., & Kwong, S. (2015) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2015.2419824 · Cited by 315
Missing Value Estimation for Mixed-Attribute Data Sets
Zhu, X., Zhang, S., Jin, Z., Zhang, Z., & Xu, Z. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2010.99 · Cited by 286
A Sparse Embedding and Least Variance Encoding Approach to Hashing
Zhu, X., Zhang, L., & Huang, Z. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2332764 · Cited by 220
Fast Motion Estimation Based on Content Property for Low-Complexity H.265/HEVC Encoder
Pan, Z., Lei, J., Zhang, Y., Sun, X., & Kwong, S. (2016) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2016.2580920 · Cited by 214
Liu, Q., Cai, W., Shen, J., Fu, Z., Liu, X., & Linge, N. (2016) · Security and Communication Networks · doi:10.1002/sec.1582 · Cited by 204
Pooled motion features for first-person videos
Ryoo, M. S., Rothrock, B., & Matthies, L. H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298691 · Cited by 183
Learning in high-dimensional multimedia data: the state of the art
Gao, L., Song, J., Liu, X., Shao, J., Liu, J., & Shao, J. (2015) · Multimedia Systems · doi:10.1007/s00530-015-0494-1 · Cited by 148
Self-citationOptimized Graph Learning Using Partial Tags and Multiple Features for Image and Video Annotation
Song, J., Gao, L., Nie, F., Shen, H. T., Yan, Y., & Sebe, N. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2601260 · Cited by 129
Self-citationOptimal graph learning with partial tags and multiple features for image and video annotation
Gao, L., Song, J., Nie, F., Yan, Y., Sebe, N., & Shen, H. T. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299066 · Cited by 79
Self-citationGraph-without-cut: An Ideal Graph Learning for Image Segmentation
Gao, L., Song, J., Nie, F., Zou, F., Sebe, N., & Shen, H. T. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10177 · Cited by 38
Self-citationSelf-representation nearest neighbor search for classification
Zhang, S., Cheng, D., Zong, M., & Gao, L. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.08.115 · Cited by 22
Self-citation
Cited by
A survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
CNN Variants for Computer Vision: History, Architecture, Application, Challenges and Future Scope
Bhatt, D., Patel, C., Talsania, H., Patel, J., Vaghela, R., Pandya, S., Modi, K., & Ghayvat, H. (2021) · Electronics · doi:10.3390/electronics10202470 · Cited by 890
Artificial intelligence for the metaverse: A survey
Huynh-The, T., Pham, Q.-V., Pham, X.-Q., Nguyen, T. T., Han, Z., & Kim, D.-S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105581 · Cited by 710
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 601
Dropout vs. batch normalization: an empirical study of their impact to deep learning
Garbin, C., Zhu, X., & Marques, O. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08453-9 · Cited by 571
Metadata
- Authors
- Wang, X., Gao, L., Song, J., & Shen, H.
- Venue
- IEEE Signal Processing Letters
- Published
- 2017-04
- Volume
- 24
- Issue
- 4
- Pages
- 510-514
- ISSN
- 1070-9908 (ISSN-L)
- 1558-2361
- OpenAlex
- W2522485430
- Cited by
- 265
- References
- 33
- FWCI
- 20.08
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.