The DOI matches its Crossref record; 44 of the 46 references it cites were found in a registry or an index.
Two-Stream 3-D convNet Fusion for Action Recognition in Videos With Arbitrary Size and Length
Abstract
3-D convolutional neural networks (3-D-convNets) have been very recently proposed for action recognition in videos, and promising results are achieved. However, existing 3-D-convNets has two “artificial” requirements that may reduce the quality of video analysis: 1) It requires a fixed-sized (e.g., 112 $\times$ 112) input video; and 2) most of the 3-D-convNets require a fixed-length input (i.e., video shots with fixed number of frames). To tackle these issues, we propose an end-to-end pipeline named Two-stream 3-D-convNet Fusion, which can recognize human actions in videos of arbitrary size and length using multiple features. Specifically, we decompose a video into spatial and temporal shots. By taking a sequence of shots as input, each stream is implemented using a spatial temporal pyramid pooling (STPP) convNet with a long short-term memory (LSTM) or CNN-E model, softmax scores of which are combined by a late fusion. We devise the STPP convNet to extract equal-dimensional descriptions for each variable-size shot, and we adopt the LSTM/CNN-E model to learn a global description for the input video using these time-varying descriptions. With these advantages, our method should improve all 3-D CNN-based video analysis methods. We empirically evaluate our method for action recognition in videos and the experimental results show that our method outperforms the state-of-the-art methods (both 2-D and 3-D based) on three standard benchmark datasets (UCF101, HMDB51 and ACT datasets).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is also worth pointing out that DH [35], another deep-based hashing method in those baselines, yields unsatisfactory performance on all datasets, where three speculations are illustrated for those results as follows.”
From Unsupervised Deep Video Hashing via Balanced Code for Large-Scale Video Retrieval (2019) · cites this paper as [35]
Citations
Cites
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,313
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,232
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601
Learning realistic human actions from movies
Ivan Laptev, A., Marszałek, M., Schmid, C., & Rozenfeld, B. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587756 · Cited by 3,481
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Cited by
Sorted by citations
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 719
“Instead of using 2D CNN, some other works [95], [96], [97], [98] used 3D CNNs as their feature extractors.”
MethodDropout vs. batch normalization: an empirical study of their impact to deep learning
Garbin, C., Xingquan Zhu, A., & Marques, O. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08453-9 · Cited by 620
“…as a new machine learning paradigms that have shown tremendous performance gain compared to conventional machine learning algorithms, especially for applications such as voice recognition [4], image classification [13], video action recognition [32, 33], and spatio-temporal data applications [14].”
BackgroundEnhancing transportation systems via deep learning: A survey
Yuan, W., Zhang, D., Liu, Y., Dai, B., & Lee, L. H. (2018) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2018.12.004 · Cited by 322
Self-Supervised Video Hashing With Hierarchical Binary Auto-Encoder
Song, J., Zhang, H., Li, X., Gao, L., Wang, M., & Hong, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2814344 · Cited by 283
“But it is beyond a set of frames and the temporal information in video is important for video content analysis [33].”
BackgroundSelf-citationFrom Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning
Song, J., Guo, Y., Gao, L., Li, X., Hanjalić, A., & Shen, H. T. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2851077 · Cited by 260
Self-citationHierarchical LSTMs with Adaptive Attention for Visual Captioning
Gao, L., Li, X., Song, J., & Shen, H. T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894139 · Cited by 249
“C3D [29], [30] model is proposed to explore temporal information and good at extracting motion information.”
BackgroundSelf-citationMiCT: Mixed 3D/2D Convolutional Tube for Human Action Recognition
Zhou, Y., Sun, X., Zha, Z.-J., & Zeng, W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00054 · Cited by 247
“The out-of-the-box C3D feature has since been widely employed in many subsequent works on action recognition and detection [31, 21, 6, 39, 37, 4].”
BackgroundMethodSelf-citationDeep adversarial metric learning for cross-modal retrieval
Xu, X., He, L., Lu, H., Gao, L., & Ji, Y. (2018) · World Wide Web · doi:10.1007/s11280-018-0541-x · Cited by 210
Self-citationSpatio-Temporal Attention Networks for Action Recognition and Detection
Li, J., Liu, X., Zhang, W., Zhang, M., Song, J., & Sebe, N. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2965434 · Cited by 181
“To avoid the expensive computation while still capturing the spatio-temporal correlations, a number of 3D CNNs methods have been proposed in the past few years [1], [6], [7], [12]–[17].”
MethodSelf-citationUllah, A., Muhammad, K., Ding, W., Palade, V., Haq, I. U., & Baik, S. W. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107102 · Cited by 169
“Later, the problem of expensive computation of the spatio-temporal features is solved by utilizing 3D CNNs. Recently, a variety of 3D CNNs with different architectures have been deployed for action recognition, such as the two-stream 3D Con-vNet [8], pseudo-3D CNNs [9], MiCT-Net [10], etc.”
BackgroundCOCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval
Li, X., Xu, C., Wang, X., Lan, W., Jia, Z., Yang, G., & Xu, J. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896494 · Cited by 166
“[43] introduce two-stream 3-D ConvNet fusion that handles video clips with arbitrary size and length by spatial temporal pyramid techniques.”
BackgroundSelf-citationUnsupervised Deep Video Hashing via Balanced Code for Large-Scale Video Retrieval
Wu, G., Han, J., Guo, Y., Liu, L., Ding, G., Ni, Q., & Shao, L. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2882155 · Cited by 148
“The contents within those frames are less correlated, especially for the cases of long videos, implying the features in terms of distributions are quite different [22], [23].”
BackgroundSpatio-temporal deformable 3D ConvNets with attention for action recognition
Li, J., Liu, X., Zhang, M., & Wang, D. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107037 · Cited by 138
Self-citationMulti-scale deep context convolutional neural networks for semantic segmentation
Zhou, Q., Yang, W., Gao, G., Ou, W., Lu, H., Chen, J., & Latecki, L. J. (2018) · World Wide Web · doi:10.1007/s11280-018-0556-3 · Cited by 132
“In the future, we hope to demonstrate the generality of our method for other visual tasks and applications, such as video analysis [36, 44], image segmentation [9], video retrieval [37, 38] and cross-modal retrieval [45, 46].”
BackgroundMethodAn Overview of Image Caption Generation Methods
Wang, H., Zhang, Y., & Yu, X. (2020) · Computational Intelligence and Neuroscience · doi:10.1155/2020/3062706 · Cited by 124
Deep learning approaches for video-based anomalous activity detection
Pawar, K., & Attar, V. (2018) · World Wide Web · doi:10.1007/s11280-018-0582-1 · Cited by 128
“State-of-the-art results have been achieved for image classification [47], object detection [69, 70, 76], activity recognition [88, 108] egocentric activity recognition [109], video hashing [91] and video captioning [29] using deep learning.”
BackgroundContinuous Gesture Segmentation and Recognition Using 3DCNN and Convolutional LSTM
Zhu, G., Zhang, L., Shen, P., Song, J., Shah, S. A. A., & Bennamoun, M. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2869278 · Cited by 117
“Multi-modality fusion is widely used to improve the final accuracy, such as the score fusion [32], [55] or the feature fusion [8], [56].”
MethodEnd-to-End Automatic Image Annotation Based on Deep CNN and Multi-Label Data Augmentation
Ke, X., Zou, J., & Niu, Y. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2895511 · Cited by 103
“According to the mathematics adopted by each annotation method, annotation methods can be classified into three categories, which are based on probability and statistics [1]–[4], graphs [5]–[12], and machine learning [13]–[26].”
MethodVision Transformer and Deep Sequence Learning for Human Activity Recognition in Surveillance Videos
Hussain, A., Hussain, T., Ullah, W., & Baik, S. W. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3454167 · Cited by 102
“Similarly, Hussain et al. [16] have proposed a lightweight 3D CNN model for anomaly activity recognition and camera prioritization in surveillance environments. variants of 3D CNNs include two-stream 3D CNN [17], pseudo-3D CNN [18], and MiCT-Net [19].”
BackgroundHuman Action Recognition from Various Data Modalities: A ReviewPreprint
Rahmani, H., Bennamoun, M., & Ke, Q. (2021) · doi:10.36227/techrxiv.13708270 · Cited by 99
Action recognition based on RGB and skeleton data sets: A survey
Yue, R., Tian, Z., & Du, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.071 · Cited by 95
Video Activity Recognition: State-of-the-Art
Rodríguez-Moreno, I., Martínez‐Otzeta, J. M., Sierra, B., Rodríguez, I., & Jauregi, E. (2019) · Sensors · doi:10.3390/s19143160 · Cited by 93
“…66.1% - Two-Stream SVMP(ResNet) [143] - 71.0% - Two-Stream SVMP(+ iDT) [143] - 72.6% - Two-Stream SVMP(I3D conf) [143] - 83.1% - STPP + CNN-E (RGB) [145] 85.6% 62.1% - STPP + LSTM (RGB) [145] 85.0% 62.5% - STPP + CNN-E (FLOW) [145] 83.2% 55.4% - STPP + LSTM (FLOW) [145] 83.8% 54.7% - STPP + CNN-E…”
BackgroundMethodInfluentialViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos
Wensel, J., Ullah, H., & Munir, A. (2023) · IEEE Access · doi:10.1109/access.2023.3293813 · Cited by 100
“The rest of comparative methods that include STPP+LSTM [47], Optical flow + multi-layer LSTM [48], TSN [49], IP-LSTM [50], Deep autoencoder [51], TS-LSTM + temporal-inception [52], HATNet [53], Correlational CNN+LSTM [54], STDAN [55], DB-LSTM+SSPF [56], DS-GRU [57], TCLC [58], Vit+LSTM [23], Semi-supervised temporal g…”
MethodEEG-based classification of emotions using empirical mode decomposition and autoregressive model
Zhang, Y., Zhang, S., & Ji, X. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-5885-9 · Cited by 85
“[38] proposed a two-stream 3DconvNetFusionmethod to recognize human actions in videos of arbitrary size and length using multiple features.”
BackgroundMethodA systematic review on content-based video retrieval
Spolaôr, N., Lee, H., Takaki, W. S. R., Ensina, L. A., Coy, C. S. R., & Wu, F. C. (2020) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2020.103557 · Cited by 87
End-to-End Multi-Channel Speech SeparationPreprint
Gu, R., Wu, J., Zhang, S., Chen, L., Xu, Y., Yu, M., Su, D., Zou, Y., & Yu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06286 · Cited by 81
Relation Attention for Temporal Action Localization
Chen, P., Gan, C., Shen, G., Huang, W., Zeng, R., & Tan, M. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2959977 · Cited by 74
“IN THE past few years, deep learning has been widely used to analyze visual content [1]–[7], especially for action recognition [8], [9].”
MethodGAIM: Graph Attention Interaction Model for Collective Activity Recognition
Lu, L.-H., Lu, Y., Yu, R., Di, H., Zhang, L., & Wang, S.-Z. (2020) · IEEE transactions on multimedia · doi:10.1109/tmm.2019.2930344 · Cited by 72
“[1]–[4] have been contributed.”
A 3D-CNN and LSTM Based Multi-Task Learning Architecture for Action Recognition
Ouyang, X., Xu, S., Zhang, C., Zhou, P., Yang, Y., Liu, G., & Li, X. (2019) · IEEE Access · doi:10.1109/access.2019.2906654 · Cited by 76
“Deep learning also have shown great performance on human action recognition due to its automatic extraction of features [24], [24]–[27].”
BackgroundVideo-Based Human Activity Recognition Using Deep Learning Approaches
Surek, G. A. S., Seman, L. O., Stefenon, S. F., Mariani, V. C., & Coelho, L. D. S. (2023) · Sensors · doi:10.3390/s23146384 · Cited by 80
“Sports 1M 7 41 [38–44] Kinetics 3 18 [21,37,42] Fudan-Columbia Video Dataset 1 6 [45] Without pre-workout 6 35 [27,46–50]”
Detection of anomaly in surveillance videos using quantum convolutional neural networks
Amin, J., Anjum, M. A., Ibrar, K., Sharif, M. I., Kadry, S., & Crespo, R. A. G. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104710 · Cited by 66
Improved human action recognition approach based on two-stream convolutional neural network model
Liu, C., Ying, J., Yang, H., Hu, X., & Liu, J. (2020) · The Visual Computer · doi:10.1007/s00371-020-01868-8 · Cited by 65
“Wang et al. [24] used a 3D CNN instead of a 2D-CNN and adopted the STPP (spatiotemporal pyramid pooling) in the last convolution layer to achieve consistent feature dimensions of output.”
MethodTemporal–Spatial Mapping for Action Recognition
Song, X., Lan, C., Zeng, W., Xing, J., Sun, X., & Yang, J. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2896029 · Cited by 66
“Wang et al. [33] propose two-stream 3D Con-vNets Fusion to recognize actions of arbitrary size and length by using spatial temporal pyramid pooling (STPP) with a LSTM or CNN model to extract multi-size descriptions and learn global representation for each input video.”
MethodSelf-citationLiu, K., Gao, L., Khan, N., Qi, L., & Guan, L. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2974323 · Cited by 64
“In this paper, motivated by classic two-stream CNN for action recognition in [22], [42], which takes appearance information as static feature stream and takes optical flow as dynamic feature stream.”
MethodSelf-citationAutomated TB classification using ensemble of deep architectures
Hooda, R., Mittal, A., & Sofat, S. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07984-5 · Cited by 69
Deep learning approaches for seizure video analysis: A review
Ahmedt‐Aristizabal, D., Armin, M. A., Hayder, Z., Garcia‐Cairasco, N., Petersson, L., Fookes, C., Denman, S., & McGonigal, A. (2024) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2024.109735 · Cited by 65
Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework
Ullah, H., & Munir, A. (2023) · Journal of Imaging · doi:10.3390/jimaging9070130 · Cited by 64
“The rest of comparative methods including STPP+LSTM [67], optical flow + multi-layer LSTM [38], TSN [68], IP-LSTM [69], deep autoencoder [59], TS-LSTM + temporal-inception [70], HATNet [71], correlational CNN+LSTM [72], STDN [56], DB-LSTM+SSPF [39], DS-GRU [43], TCLC [73], and semi-supervised…”
BackgroundMethodInfluentialTwo-Stream Action Recognition-Oriented Video Super-Resolution
Zhang, H., Liu, D., & Xiong, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00889 · Cited by 62
“Later on, C3D features and 3D CNN architectures [4, 35, 36, 39] appeared.”
BackgroundYogaNet: 3-D Yoga Asana Recognition Using Joint Angular Displacement Maps With ConvNets
Maddala, T. K. K., Kishore, P. V. V., Kumar, E. K., & Dande, A. K. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2904880 · Cited by 60
“[22] used an end-to-end, twostream CNN architecture for action videos of arbitrary length and size, which can be used on raw video data.”
MethodFused GRU with semantic-temporal attention for video captioning
Gao, L., Wang, X., Song, J., & Liu, Y. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.06.096 · Cited by 56
Self-citationPredictively encoded graph convolutional network for noise-robust skeleton-based action recognition
Yoon, Y., Yu, J., & Jeon, M. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02487-z · Cited by 56
3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition
Arif, S., Wang, J., Hassan, T. U., & Fei, Z. (2019) · Future Internet · doi:10.3390/fi11020042 · Cited by 56
Ravi, S., Maloji, S., Kishore, P. V. V., Kumar, E. K., Kumar, M. T. K., & Kumar, D. A. (2019) · Journal of Computer Languages · doi:10.1016/j.cola.2019.04.002 · Cited by 53
Serpush, F., & Rezaei, M. (2021) · SN Computer Science · doi:10.1007/s42979-021-00484-0 · Cited by 53
LSTM with bio inspired algorithm for action recognition in sports videos
Chen, J., Samuel, R. D. J., & Poovendran, P. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104214 · Cited by 52
Exposing Deepfake Face Forgeries With Guided Residuals
Guo, Z., Yang, G., Chen, J., & Sun, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3237169 · Cited by 55
“For image forensics, fusing two or more types of features can improve detection accuracy and robustness [39].”
MethodSelf-citationAhmad, T., & Wu, J. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3249152 · Cited by 52
“D CNNs, pooling schemes, and temporal modeling [31], [32], [33], [34], [35].”
BackgroundDense Semantics-Assisted Networks for Video Action Recognition
Luo, H., Lin, G., Yao, Y., Tang, Z., Wu, Q., & Hua, X. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3100842 · Cited by 50
Deep Self-Taught Hashing for Image Retrieval
Liu, Y., Song, J., Zhou, K., Yan, L., Liu, L., Zou, F., & Shao, L. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2822781 · Cited by 48
A Psychologically Inspired Fuzzy Cognitive Deep Learning Framework to Predict Crowd Behavior
Varghese, E. B., Thampi, S. M., & Berretti, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2987021 · Cited by 47
Bidirectional Posture-Appearance Interaction Network for Driver Behavior Recognition
Tan, M., Ni, G., Liu, X., Zhang, S. S., Wu, X., Wang, Y., & Zeng, R. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3123127 · Cited by 47
“Here, we follow [40]–[42] to use the outputs of two”
MethodSelf-citationHussain, A., Khan, S. U., Khan, N., Shabaz, M., & Baik, S. W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107218 · Cited by 55
Hussain, A., Khan, S. U., Rida, I., Khan, N., & Baik, S. W. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102211 · Cited by 44
Residual LSTM Attention Network for Object Tracking
Kim, H., & Park, R. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2835768 · Cited by 45
“as video analysis including action recognition [12], [13], video captioning [14], [15], and video hashing [16].”
MethodLearning Composite Latent Structures for 3D Human Action Representation and Recognition
Wei, P., Sun, H., & Zheng, N. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2897902 · Cited by 42
“In recent years, neural network and deep learning methods have been extensively used in action recognition and achieved impressive results [36]–[46].”
Generalized zero-shot learning for action recognition with web-scale video data
Liu, K., Liu, W., Ma, H., Huang, W., & Dong, X. (2018) · World Wide Web · doi:10.1007/s11280-018-0642-6 · Cited by 41
“Many works [48, 53, 58] explore to utilize 3D convolutional operator to extract the motion features of videos.”
BackgroundEfficient Image Hashing with Geometric Invariant Vector Distance for Copy Detection
Liu, S., & Huang, Z. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3355394 · Cited by 41
“More recently, some learning-based hashing methods have been proposed, which includes machine learning hashing [33, 42–45] and deep learning hashing [46–49].”
BackgroundMethodSpatial–temporal pooling for action recognition in videos
Wang, J., Shao, Z., Huang, X., Lü, T., Zhang, R., & Lv, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.071 · Cited by 41
Aligning Correlation Information for Domain Adaptation in Action Recognition
Xu, Y., Cao, H., Mao, K., Chen, Z., Xie, L., & Yang, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3212909 · Cited by 46
“One of which utilizes the two-stream structure [30], [31], [32], [33], [34], [35], [36], extracting video features through CNNs from both optical flow and RGB inputs.”
MethodText-instance graph: Exploring the relational semantics for text-based visual question answering
Li, X., Wu, B., Song, J., Gao, L., Zeng, P., & Gan, C. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108455 · Cited by 39
Self-citationDastbaravardeh, E., Askarpour, S., Anari, M. S., & Rezaee, K. (2024) · International Journal of Intelligent Systems · doi:10.1155/2024/1052344 · Cited by 43
“Feature extraction is sometimes achieved by creating a skeleton from the video; however, the information gained from the skeleton is sometimes discarded, making the method less robust and accurate [37, 45, 89, 90].”
BackgroundDevaraj, J., Ganesan, S., Elavarasan, R. M., & Subramaniam, U. (2021) · Applied Sciences · doi:10.3390/app11094129 · Cited by 39
“A deep convolutional neural network called visual geometry group VGG 19, a successor of AlexNet, is used to classify the weather events [53].”
MethodHussain, A., Khan, S. U., Khan, N., Ullah, W., Alkhayyat, A. H., Alharbi, M., & Baik, S. W. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2023.11.017 · Cited by 39
Temporal Action Localization in Untrimmed Videos Using Action Pattern Trees
Song, H., Wu, X., Zhu, B., Wu, Y., Chen, M., & Jia, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2866370 · Cited by 36
“HUMAN action understanding has been an active topic in computer vision with many applications including video surveillance, human computer interaction, sports video analysis, and video retrieval [7], [15], [38], [41], [47].”
BackgroundOn fusing the latent deep CNN feature for image classification
Liu, X., Zhang, R., Meng, Z., Hong, R., & Liu, G. (2018) · World Wide Web · doi:10.1007/s11280-018-0600-3 · Cited by 35
“The deep learning based video understanding methods [14, 52, 55] utilize novel fusion frameworks combining both the early fusion and late fusion strategies, and show great performance comparing to state of the art approaches.”
MethodSelf-citationA Hybrid Transformer Framework for Efficient Activity Recognition Using Consumer Electronics
Hussain, A., Khan, S. U., Khan, N., Bhatt, M. W., Farouk, A., Bhola, J., & Baik, S. W. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3373824 · Cited by 34
“…[13] 73.7 92.2 - T-Inception [28] 67.5 93.9 - OPLSTM [16] 72.21 94.45 94.9 LTF [29] 68.2 91.4 - STDAN + RGB [30] 60.4 91.0 AESVM [31] 70.3 94.33 96.4 3DCNN+Fusion 3DCF [32] 70.3 92.6 - Pseudo-3DCNN P3DCNN [33] - 93.7 - MiCT-3DCNN M3DCT [34] 70.5 94.7 - ViT-LSTM [12] 73.714 - 96.144 CDG [14] 72.”
InfluentialContent-based image retrieval via a hierarchical-local-feature extraction scheme
Jian, M., Yin, Y., Dong, J., & Lam, K. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6122-2 · Cited by 34
“In the big data era, content-based image retrieval (CBIR) has been a hot research topic in the last few years, because of its many practical applications, including in the fields of multimedia management and analysis [5, 40], biomedical image retrieval [43], dermatological lesions analysis [21], intelligent video surv…”
Action Recognition in Videos Using Pre-Trained 2D Convolutional Neural Networks
Kim, J.-H., & Won, C. S. (2020) · IEEE Access · doi:10.1109/access.2020.2983427 · Cited by 34
“In [17], the two-stream CNN structure was updated to support the videos with arbitrary size and length.”
MethodGan, H., Li, S., Ou, M., Yang, X., Huang, B., Liu, K., & Xue, Y. (2021) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2021.106384 · Cited by 34
Duan, C., & Zhang, T. (2020) · IEEE Access · doi:10.1109/access.2020.3025165 · Cited by 34
“Through the comparison, we selected the appropriate fusion strategy and fusion layer, which can help us achieve better accuracy [32], [33].”
MethodLearning Semantic-Aware Spatial-Temporal Attention for Interpretable Action Recognition
Fu, J., Gao, J., & Xu, C. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3137023 · Cited by 35
Question-Led object attention for visual question answering
Gao, L., Cao, L., Xu, X., Shao, J., & Song, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.102 · Cited by 32
Self-citationCascade multi-head attention networks for action recognition
Wang, J., Peng, X., & Qiao, Y. (2020) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.102898 · Cited by 34
“Two-stream 3D-convNet Fusion is proposed to recognize actions in arbitrary size and length videos with multiple features (Wang et al., 2018a).”
BackgroundRen, Z., Zhang, Q., Cheng, J. S., Hao, F., & Gao, X. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.12.020 · Cited by 31
Deep learning methods for single camera based clinical in-bed movement action recognition
Karácsony, T., Jeni, L. A., la Torre, F. D., & Cunha, J. P. S. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.104928 · Cited by 31
“To further enhance the capture of long-term connections a 3D CNN-LSTM architecture was proposed [162].”
BackgroundFully Convolutional Network for Multiscale Temporal Action Proposals
Guo, D., Li, W., & Fang, X. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2839534 · Cited by 30
“In recent years, we have witnessed the rapid growth of various deep learning methods applied to video recognition [19]–[21].”
BackgroundSpatial–Temporal Context-Aware Online Action Detection and Prediction
Huang, J., Li, N., Li, T., Liu, S., & Li, G. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2923712 · Cited by 32
“[21] and [22] utilize a two-stream structue,”
BackgroundDeep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos
Agethen, S., & Hsu, W. H. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2932564 · Cited by 31
“by many works for action recognition [12][13].”
Deep Gesture Video Generation With Learning on Regions of Interest
Cui, R., Cao, Z. L., Pan, W., Zhang, C., & Wang, J. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2960700 · Cited by 30
Weakly Semantic Guided Action Recognition
Yu, T., Wang, L., Da, C., Gu, H., Xiang, S., & Pan, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2907060 · Cited by 28
Order-aware convolutional pooling for video based action recognition
Wang, P., Liu, L., Shen, C., & Shen, H. T. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.03.002 · Cited by 30
Self-citationLiang, C., Liu, D., Qi, L., & Guan, L. (2020) · IEEE Access · doi:10.1109/access.2020.2976496 · Cited by 28
“Recent work [56] a 3D CNN fusion strategy is proposed by combing the softmax scores for action recognition with arbitrary length.”
MethodIntelligent Video Analytics for Human Action Recognition: The State of Knowledge
Kulbacki, M., Segen, J., Chaczko, Z., Rozenblit, J. W., Kulbacki, M., Klempous, R., & Wojciechowski, K. W. (2023) · Sensors · doi:10.3390/s23094258 · Cited by 31
“Several publicly available datasets have also been widely used in the research community to evaluate the performance of IVA algorithms.”
MethodModeling transformer architecture with attention layer for human activity recognition
Pareek, G., Nigam, S., & Singh, R. K. (2024) · Neural Computing and Applications · doi:10.1007/s00521-023-09362-7 · Cited by 28
“Wang et al. [20] introduced a two-stream 3D convNet fusion for action recognition in videos of arbitrary size and length, emphasizing the importance of flexibility in model design.”
Srihari, D., Kishore, P. V. V., Kumar, E. K., Kumar, D. A., Kumar, M. T. K., Prasad, M. V. D., & Prasad, C. R. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08588-9 · Cited by 26
“In this era of deep nets, multi-channel CNNs were introduced with RGB stream and depth stream separately as input to a CNN [44].”
MethodA New Robust Reference Image Hashing System
Singh, S. P., Bhatnagar, G., & Singh, A. K. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3050435 · Cited by 26
Attention-based multi-modal fusion for improved real estate appraisal: a case study in Los Angeles
Bin, J., Gardiner, B., Liu, Z., & Li, E. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07895-5 · Cited by 26
Adaptation-Oriented Feature Projection for One-shot Action Recognition
Zou, Y., Shi, Y., Shi, D., Wang, Y., Liang, Y., & Tian, Y. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972128 · Cited by 25
“Furthermore, [30] improved this model to recognize actions with arbitrary image sizes and video lengths.”
BackgroundSpatiotemporal Saliency Representation Learning for Video Action Recognition
Kong, Y., Wang, Y., & Li, A. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3066775 · Cited by 25
A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection
Korban, M., Youngs, P. A., & Acton, S. T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3377192 · Cited by 25
Large scale image tamper detection and restoration
Sarkar, D., Palit, S., Som, S., & Dey, K. N. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08669-0 · Cited by 24
Zhang, Q., Zhou, L., Zhang, T., & Zhang, D.-H. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7180-9 · Cited by 23
“In [24], an end-to-end framework of Two-stream 3D convNet Fusion was proposed, which can recognize human actions in videos of arbitrary size and length using multiple features, and the proposed scheme enhanced the performance of CNNbased video analysis algorithm.”
MethodVideo sketch: A middle-level representation for action recognition
Zhang, X., Huang, Y., Mi, Y., Pei, Y., Zou, Q., & Wang, S. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01905-y · Cited by 23
“Wang et al. [28] present a two-stream 3D ConvNet fusion framework by considering both appearance and optical flows, which solves the issues of fixed-sized and fixed-length input video.”
BackgroundSpiking neural network based on joint entropy of optical flow features for human action recognition
Berlin, S. J., & John, M. (2020) · The Visual Computer · doi:10.1007/s00371-020-02012-2 · Cited by 25
Kasban, H., & Salama, D. H. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08100-3 · Cited by 22
Unsupervised densely attention network for infrared and visible image fusion
Li, Y., Wang, J., Miao, Z., & Wang, J. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09301-x · Cited by 22
Attention-Driven Appearance-Motion Fusion Network for Action Recognition
Liu, S., & Ma, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3148588 · Cited by 22
“…havebeendevelopedforhumanactionrecognition[6]:(1)2D ConvNets with temporal models on top such as LSTM [7], [8], temporal convolution [9], sparse sampling and aggregation [10], and attention modeling [11], [12], (2) 3D ConvNets [13], and (3) two-stream ConvNets [14], [15].”
BackgroundA Multi-Modal Transformer network for action detection
Korban, M., Youngs, P. A., & Acton, S. T. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109713 · Cited by 23
“[21] introduced a 3-”
Semantic-filtered Soft-Split-Aware video captioning with audio-augmented feature
Xu, Y., Yang, J., & Mao, K. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.027 · Cited by 22
Development of Compound Fault Diagnosis System for Gearbox Based on Convolutional Neural Network
Lin, M.-C., Han, P.-Y., Fan, Y.-H., & Li, C.-H. G. (2020) · Sensors · doi:10.3390/s20216169 · Cited by 23
“After that, CNN related research began to flourish in various fields, including object recognition [7,8], portrait recognition [9,10], action recognition [11,12], etc.”
Background
Metadata
- Authors
- Wang, X., Gao, L., Wang, P., Sun, X., & Liu, X.
- Affiliations
- University of Electronic Science and Technology of China (CN)
- The University of Adelaide (AU)
- Harbin Institute of Technology (CN)
- Beihang University (CN)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2018-03
- Volume
- 20
- Issue
- 3
- Pages
- 634-644
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2751445731
- Cited by
- 263
- References
- 46
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 12 · 2023: 24 · 2022: 23 · 2021: 44 · 2020: 47 · 2019: 60 · 2018: 42
- FWCI
- 17.03
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Funders
- Fundamental Research Funds for the Central Universities (ZYGX2014J063, ZYGX2014Z007)
- National Natural Science Foundation of China (61502080, 61632007)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- 3D convolutional neural networks, long short-term memory, decision fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
