A Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Abstract
We present a multi-stream bi-directional recurrent neural network for fine-grained action detection. Recently, twostream convolutional neural networks (CNNs) trained on stacked optical flow and image frames have been successful for action recognition in videos. Our system uses a tracking algorithm to locate a bounding box around the person, which provides a frame of reference for appearance and motion and also suppresses background noise that is not within the bounding box. We train two additional streams on motion and appearance cropped to the tracked bounding box, along with full-frame streams. Our motion streams use pixel trajectories of a frame as raw features, in which the displacement values corresponding to a moving scene point are at the same spatial position across several frames. To model long-term temporal dynamics within and between actions, the multi-stream CNN is followed by a bi-directional Long Short-Term Memory (LSTM) layer. We show that our bi-directional LSTM network utilizes about 8 seconds of the video sequence to predict an action label. We test on two action detection datasets: the MPII Cooking 2 Dataset, and a new MERL Shopping Dataset that we introduce and make available to the community with this paper. The results demonstrate that our method significantly outperforms state-of-the-art action detection methods on both datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current approaches decouple action segmentation into extracting low-level spatiotemporal features and applying high-level temporal classifiers.
“Current approaches decouple this task into extracting low-level spatiotemporal features and applying high-level temporal classifiers.”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [27]
Recurrent models have a limited span of attention.
“have been found empirically to have a limited span of attention [27]”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [27]
Sliding window action detectors typically do not capture long-range temporal patterns.
“recent temporal models have been limited to sliding window action detectors [25, 27, 20], which typically do not capture long-range temporal patterns”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [27]
mAP is very sensitive to the confidence score assigned to each segment prediction.
“The key issue is that mAP is very sensitive to a confidence score assigned to each segment prediction.”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [27]
Recurrent models are difficult to interpret.
“recurrent models [27, 9], which can capture latent temporal patterns but are difficult to interpret”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [27]
Our approach outperforms Singh et al.'s LSTM-based detection model on MERL Shopping.
“We use Singh's proposed dataset, MERL Shopping, and show our approach outperforms their LSTM-based detection model.”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [27]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407 · see citing works
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · see citing works
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,121 · see citing works
Detecting Pedestrians Using Patterns of Motion and Appearance
Viola, P. A., Jones, M., & Snow, D. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-6644-8 · Cited by 2,098 · see citing works
Self-citationHierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., & Wang, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298714 · Cited by 1,962 · see citing works
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669 · see citing works
Action recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299059 · Cited by 1,194 · see citing works
Detecting pedestrians using patterns of motion and appearance
Viola, Jones, & Snow (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238422 · Cited by 1,002 · see citing works
Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition
Graves, A., Fernández, S., & Schmidhuber, J. (2005) · Lecture Notes in Computer Science · doi:10.1007/11550907_126 · Cited by 925 · see citing works
Sequential Deep Learning for Human Action Recognition
Baccouche, M., Mamalet, F., Wolf, C., García, C., & Baskurt, A. M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25446-8_4 · Cited by 793 · see citing works
EpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298720 · Cited by 757 · see citing works
Robust real-time periodic motion detection, analysis, and applications
Cutler, R., & Davis, L. S. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.868681 · Cited by 701 · see citing works
Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice
Peng, X., Wang, L., Wang, X., & Qiao, Y. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.03.013 · Cited by 681 · see citing works
The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · see citing works
Gkioxari, G., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298676 · Cited by 545 · see citing works
A database for fine grained activity detection of cooking activities
Rohrbach, M., Amin, S., Andriluka, M., & Schiele, B. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247801 · Cited by 521 · see citing works
Towards Good Practices for Very Deep Two-Stream ConvNetsPreprint
Wang, L., Xiong, Y., Wang, Z., & Qiao, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02159 · Cited by 381 · see citing works
Discriminative subvolume search for efficient action detection
Yuan, J., Liu, Z., & Wu, Y. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206671 · Cited by 270 · see citing works
Action Localization with Tubelets from Motion
Jain, M., van Gemert, J. C., Jeǵou, H., Bouthémy, P., & Snoek, C. G. M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.100 · Cited by 250 · see citing works
Spatiotemporal Deformable Part Models for Action Detection
Tian, Y., Sukthankar, R., & Shah, M. A. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.341 · Cited by 243 · see citing works
Fast action proposals for human action detection and search
Yu, G., & Yuan, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298735 · Cited by 190 · see citing works
Recognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data
Rohrbach, M., Rohrbach, A., Regneri, M., Amin, S., Andriluka, M., Pinkal, M., & Schiele, B. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0851-8 · Cited by 182 · see citing works
Parsing Videos of Actions with Segmental Grammars
Pirsiavash, H., & Ramanan, D. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.85 · Cited by 161 · see citing works
Action detection in complex scenes with spatial and temporal ambiguities
Hu, Y., Cao, L., Lv, F., et al. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459153 · Cited by 150 · see citing works
Representing Videos Using Mid-level Discriminative Patches
Jain, A., Gupta, A. K., Rodriguez, M. D., & Davis, L. S. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.332 · Cited by 138 · see citing works
Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
Sun, C., Shetty, S., Sukthankar, R., & Nevatia, R. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806226 · Cited by 121 · see citing works
(2016) · MPG.PuRe (Max Planck Society) · Cited by 107 · see citing works
Multiple Granularity Analysis for Fine-Grained Action Detection
Ni, B., Paramathayalan, V. R., & Moulin, P. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.102 · Cited by 59 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · see citing works
“We use MERL Shopping which was designed for action detection, Georgia Tech Egocentric Activities which was designed for action segmentation, and 50 Salads which has been used for both.”
BackgroundMethodInfluentialSpatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition
Liu, J., Shahroudy, A., Xu, D., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_50 · Cited by 1,353 · see citing works
Deep Learning for IoT Big Data and Streaming Analytics: A Survey
Mohammadi, M., Al-Fuqaha, A., Sorour, S., & Guizani, M. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2844341 · Cited by 1,471 · see citing works
“) [163] and providing a list of related items for the detected action.”
BackgroundTemporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_7 · Cited by 1,059 · see citing works
“It has been shown that in practice LSTM only keeps a memory of about 4 seconds on some video-based action segmentation datasets [15].”
BackgroundTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892 · see citing works
TALL: Temporal Activity Localization via Language Query
Gao, J., Sun, C., Yang, Z., & Nevatia, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.563 · Cited by 884 · see citing works
“For traditional temporal action localization, most current approaches [30, 15, 26, 34, 35] apply activity classifiers trained with optical flow-based methods [33, 28] or Convolutional Neural Networks (CNNs) [29, 32] in a sliding window fashion.”
BackgroundMethodR-C3D: Region Convolutional 3D Network for Temporal Activity Detection
Xu, H., Das, A., & Saenko, K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.617 · Cited by 780 · see citing works
“Alternative activity detection approaches [4, 17, 18, 29, 39] use a recurrent neural network (RNN) to encode a sequence of frame or video chunk features ( e.g. VGG [28], C3D [32]) and predict the activity label at each time step.”
MethodInfluentialRethinking the Faster R-CNN Architecture for Temporal Action Localization
Chao, Y.-W., Vijayanarasimhan, S., Seybold, B., Ross, D. A., Deng, J., & Sukthankar, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00124 · Cited by 744 · see citing works
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673 · see citing works
“[23] use a two-stream network to learn representations of short video chunks.”
Going deeper into action recognition: A survey
Herath, S., Harandi, M. T., & Porikli, F. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.010 · Cited by 646 · see citing works
“Therefore, fine-grained action recognition tasks (Ni et al., 2016; Singh et al., 2016; Lea et al., 2016) that have been already receiving growing attention from the community, can shape the future solutions and associated problems.”
BackgroundShou, Z., Chan, J., Zareian, A., Miyazawa, K., & Chang, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.155 · Cited by 580 · see citing works
“Various methods have been developed, from the perspective of supervoxel merging [20, 54, 55], tracking [69, 42, 62, 52], object detection and linking [28, 14, 75, 42, 62], spatio-temporal segmentation [31, 70], and leveraging still images [21, 58, 22].”
MethodTowards Automatic Learning of Procedures From Web Instructional Videos
Zhou, L., Xu, C., & Corso, J. J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12342 · Cited by 663 · see citing works
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates
Liu, J., Shahroudy, A., Xu, D., Kot, A. C., & Wang, G. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2771306 · Cited by 535 · see citing works
“…(RNNs) which can handle the sequential data with variable lengths [10], [11], have shown their strength in language modeling [12], [13], [14], image captioning [15], [16], video analysis [17], [18], [19], [20], [21], [22], [23], [24], [25], and RGB-based activity recognition [26], [27], [28], [29].”
BackgroundTURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
Gao, J., Yang, Z., Sun, C., Chen, K., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.392 · Cited by 498 · see citing works
“[23] extend two-stream [21] framework to multi-stream bi-directional LSTM networks and achieved state-of-the-art performance on MPII-Cooking dataset [19].”
BackgroundSingle Shot Temporal Action Detection
Lin, T., Zhao, X., & Shou, Z. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123343 · Cited by 475 · see citing works
“However, it is di cult for RNNs to keep a long time period memory in practice [32].”
BackgroundMethodSST: Single-Stream Temporal Action Proposals
Buch, S., Escorcia, V. A., Shen, C., Ghanem, B., & Niebles, J. C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.675 · Cited by 440 · see citing works
“A number of existing methods approach the problem of temporal action localization [19, 22, 26, 27, 31, 32, 35, 40, 24, 30, 38].”
BackgroundWeakly Supervised Action Localization by Sparse Temporal Pooling Network
Nguyen, P., Han, B., Liu, T., & Prasad, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00706 · Cited by 401 · see citing works
RGB-D-based human motion recognition with deep learning: A survey
Wang, P., Li, W., Ogunbona, P., Wan, J., & Escalera, S. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.04.007 · Cited by 405 · see citing works
“[118] presented a multi-stream bi-directional recurrent neural network for fine-grained action detection.”
BackgroundMS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349 · see citing works
A Multi-Stream Feature Fusion Approach for Traffic Prediction
Li, Z., Xiong, G., Tian, Y., Lv, Y., Chen, Y., Hui, P., & Su, X. J. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3026836 · Cited by 336 · see citing works
Gaussian Temporal Awareness Networks for Action Localization
Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., & Mei, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00043 · Cited by 380 · see citing works
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
Xu, H., He, K., Plummer, B. A., Sigal, L., Sclaroff, S., & Saenko, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019062 · Cited by 318 · see citing works
Attentive Moment Retrieval in Videos
Liu, M., Wang, X., Nie, L., He, X., Chen, B., & Chua, T. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210003 · Cited by 313 · see citing works
“And Singh et al. [30] presented a multi-stream bi-directional RNN network for fine-grained action detection.”
BackgroundOnline Real-Time Multiple Spatiotemporal Action Localisation and Prediction
Singh, G., Saha, S., Sapienza, M., Torr, P. H. S., & Fabio Cuzzolin, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.393 · Cited by 283 · see citing works
“LSTMs are also increasingly being used [56, 3, 37, 57] to address the problem.”
MethodTemporal Context Network for Activity Localization in Videos
Dai, X., Singh, B., Zhang, G., Davis, L. S., & Chen, Y. Q. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.610 · Cited by 270 · see citing works
“In [31], a multi-stream RNN was employed along with tracking to generate frame level predictions to which simple grouping was applied at multiple detection thresholds for obtaining detections.”
BackgroundMethodInfluentialSelf-citationBagautdinov, T., Alahi, A., Fleuret, F., Fua, P., & Savarese, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.365 · Cited by 261 · see citing works
“More recently, data-driven approaches based on deep learning have started to emerge, including methods based on 3D CNNs [23] and multi-stream networks [15, 36].”
BackgroundMethodCross-modal Moment Localization in Videos
Liu, M., Wang, X., Nie, L., Tian, Q. C., Chen, B., & Chua, T. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240549 · Cited by 262 · see citing works
“Meanwhile, Singh et al. [30] extended two-stream framework [28] via augmenting full-frame image features with features from a bounding box surrounding the actor.”
BackgroundVideo Super-Resolution with Recurrent Structure-Detail Network
Isobe, T., Jia, X., Gu, S., Li, S., Wang, S., & Tian, Q. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_38 · Cited by 260 · see citing works
“In [24], a bi-directional LSTM is applied after a multi-stream CNN to fully explore temporal information in a sequence for fine-grained action detection.”
MethodMulti-stream CNN: Learning representations based on human-related regions for action recognition
Tu, Z., Xie, W. C., Qin, Q., Poppe, R. W., Veltkamp, R. C., Li, B., & Yuan, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.01.020 · Cited by 250 · see citing works
Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
Zhang, Z., Lin, Z., Zhao, Z., & Xiao, Z. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331235 · Cited by 239 · see citing works
“[33] propose a multi-stream bi-directional RNN with two additional streams on motion and appearance to achieve the fine-grained action detection.”
BackgroundDeep Temporal Convolutional Networks for Short-Term Traffic Flow Forecasting
Zhao, W., Gao, Y., Ji, T., Wan, X., Ye, F., & Bai, G. (2019) · IEEE Access · doi:10.1109/access.2019.2935504 · Cited by 223 · see citing works
Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos
Yuan, Y., Ma, L., Wang, J., Liu, W., & Zhu, W. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3038993 · Cited by 219 · see citing works
“Detecting or localizing activities in videos [18, 30, 26, 34, 11, 29, 21, 9, 8] is a prominent while fundamental problem for video understanding.”
BackgroundKumar, P., Chauhan, S., & Awasthi, L. K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09986-x · Cited by 220 · see citing works
“For this reason, the works of Singh et al. [188], Wu et al. [189], Baccouche et al. [190], Ng et al. [191], Li et al. [192], Wang et al. [193], and Chen et al. The primary goal of these works is to extract motion features from input video using industry-standard CNN models like AlexNet [194],…”
QuaterNet: A Quaternion-based Recurrent Model for Human MotionPreprint
Pavllo, D., Grangier, D., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06485 · Cited by 206 · see citing works
“Modeling human motion is essential for many applications, including action recognition [13, 34], action detection [49] and computer graphics [23].”
BackgroundWeakly Supervised Video Moment Retrieval From Text Queries
Mithun, N. C., Paul, S., & Roy–Chowdhury, A. K. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01186 · Cited by 205 · see citing works
“The text to video moment retrieval task is more challenging than the task of localizing categorical activities in videos, which is a comparatively well-studied field [20, 38, 44, 39, 28, 31].”
BackgroundOkutama-Action: An Aerial View Video Dataset for Concurrent Human Action Detection
Barekatain, M., Rabadán, M. M. I., Shih, H.-F., Murray, S., Nakayama, K., Matsuo, Y., & Prendinger, H. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.267 · Cited by 204 · see citing works
“In recent years, various datasets for human action recognition have been constructed, such as ActivityNet [1], Sports-1M [7], HMDB51 [8], MPII Cooking [18, 17], Olympic Sports [12], Hollywood [11, 9] and MERL Shopping [23].”
BackgroundDeep phenotyping: deep learning for temporal phenotype/genotype classification
Namin, S. T., Esmaeilzadeh, M., Najafi, M., Brown, T. B., & Borevitz, J. (2018) · Plant Methods · doi:10.1186/s13007-018-0333-4 · Cited by 200 · see citing works
“More specifically, [15, 16, 17, 18, 19] used LSTM in conjunction with CNN for action and activity recognition were shown to provide a significant improvement in performance over previous studies of video data.”
BackgroundMethodPapadomanolaki, M., Vakalopoulou, M., & Karantzalos, K. G. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3055584 · Cited by 196 · see citing works
“tracking [41] and action recognition [42].”
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · see citing works
Bayesian neural networks for flight trajectory prediction and safety assessment
Zhang, X., & Mahadevan, S. (2020) · Decision Support Systems · doi:10.1016/j.dss.2020.113246 · Cited by 195 · see citing works
Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks
Liao, Y., Xiong, P., Min, W., Min, W., & Lu, J. (2019) · IEEE Access · doi:10.1109/access.2019.2904749 · Cited by 193 · see citing works
“According to that, this paper will present a newmodified method, BLSTM-3D Residual Network (B3D ResNet), to instead of the original method based on video sequence, which can get an optimal result inspired by [9] and [10].”
MethodA Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences
Asadi, M., Clapés, A., Bellantonio, M., Escalante, H. J., Ponce-López, V., Baró, X., Guyon, I., Shohreh Kasaei, B., & Escalera, S. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.150 · Cited by 176 · see citing works
Bidirectional Convolutional LSTM for the Detection of Violence in Videos
Hanson, A., Pnvr, K., Krishnagopal, S., & Davis, L. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11012-3_24 · Cited by 170 · see citing works
“In this work, we propose a Bidirectional Convolutional LSTM (BiConvLSTM) [39, 17, 30] architecture, called the Spatiotemporal Encoder, to detect violence in videos.”
BackgroundMethodEncouraging LSTMs to Anticipate Actions Very Early
Aliakbarian, M. S., Saleh, F. S., Salzmann, M., Fernando, B., Petersson, L., & Andersson, L. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.39 · Cited by 169 · see citing works
“While great progress has been made to recognize actions from complete sequences [16, 38, 7, 9, 30, 41, 31, 10, 2] in the past decade, action anticipation [25, 26, 24, 45, 32, 33, 21], which aims to predict the observed action as early as possible, has become a popular research problem only recently.”
BackgroundAction-Stage Emphasized Spatiotemporal VLAD for Video Action Recognition
Tu, Z., Li, H., Zhang, D., Dauwels, J., Li, B., & Yuan, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2890749 · Cited by 168 · see citing works
“[42] employed a state-based tracker to detect the bounding box of the person, and two”
MethodAlleviating Over-segmentation Errors by Detecting Action Boundaries
Ishikawa, Y., Kasai, S., Aoki, Y., & Kataoka, H. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00237 · Cited by 171 · see citing works
“Other approaches model the temporal action sequence with a Markov model [23, 36] or an RNN [39] to classify framewise actions.”
BackgroundMethodInfluentialHe, D., Zhao, X. H., Huang, J., Li, F., Liu, X., & Wen, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018393 · Cited by 161 · see citing works
“In the past few years, studies on detecting a particular object in image (Pirinen and Sminchisescu 2018) or activity in video (Yeung et al. 2016; Singh et al. 2016; Zhao et al. 2017; Shou et al. 2017) have been extensively exploited.”
Temporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202 · see citing works
“[37] present a multi-stream bi-directional recurrent neural network for fine-grained action detection.”
BackgroundInfluentialImproving Action Segmentation via Graph-Based Temporal Reasoning
Huang, Y., Sugano, Y., & Sato, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01404 · Cited by 160 · see citing works
“Since MSTCN is based on temporal convolution networks, we further test the model performance combined with a recurrent backbone Bi-LSTM [53].”
BackgroundMethodInfluentialTemporal Action Localization by Structured Maximal Sums
Yuan, Z., Stroud, J., Lü, T., & Deng, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.342 · Cited by 148 · see citing works
“applied frame-wise CNN feature detectors, connected by recurrent LSTM modules [21, 18].”
MethodBoundary-Aware Cascade Networks for Temporal Action Segmentation
Wang, Z., Gao, Z., Wang, L., Li, Z., & Wu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_3 · Cited by 146 · see citing works
“[25] used a multi-stream bi-directional recurrent neural network for fine-grained action detection.”
MethodVision-based Fight Detection from Surveillance Cameras
Aktı, Ş., Tataroglu, G. A., & Ekenel, H. K. (2019) · International Conference on Image Processing Theory, Tools and Applications (IPTA) · doi:10.1109/ipta.2019.8936070 · Cited by 142 · see citing works
“Bi-LSTMs show better performance than regular LSTMs in action recognition, as also stated in related studies in [8], [9].”
Rodríguez-Opazo, C., Marrese-Taylor, E., Saleh, F. S., Li, H., & Gould, S. J. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093328 · Cited by 136 · see citing works
“Singh et al. [45] extended the two-stream [44] framework with person detection and bi-directional LSTMs and achieved state-of-the-art performance on the MPII-Cooking dataset [38].”
BackgroundA Review on Deep Learning Techniques for IoT Data
Lakshmanna, K., Kaluri, R., Gundluru, N., Alzamil, Z. S., Rajput, D. S., Khan, A. A., Haq, M. A., & Alhussen, A. (2022) · Electronics · doi:10.3390/electronics11101604 · Cited by 139 · see citing works
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Richard, A., Kuehne, H., Iqbal, A., & Gall, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00771 · Cited by 131 · see citing works
“Most existing work on temporal action segmentation focuses on a fully supervised task [15, 26, 31, 19, 32, 22, 6, 37, 18, 36, 28].”
BackgroundVideo Moment Retrieval With Cross-Modal Neural Architecture Search
Yang, X., Wang, S., Dong, J., Dong, J., Wang, M., & Chua, T. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3140611 · Cited by 131 · see citing works
A Survey of mmWave Radar-Based Sensing in Autonomous Vehicles, Smart Homes and Industry
Kong, H., Huang, C., Yu, J., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3409556 · Cited by 129 · see citing works
“Among RNNs, long short-term memory (LSTM)-based RNNs [108], gated recurrent unit (GRU)-based RNNs [109], and bi-directional RNNs [110] have become the focus of solving temporal tasks.”
A Comprehensive Study of Deep Video Action RecognitionPreprint
Zhu, Y., Li, X., Liu, C., Zolfaghari, M., Xiong, Y., Wu, C., Zhang, Z., Tighe, J., Manmatha, R., & Li, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06567 · Cited by 172 · see citing works
“RNNs are also intensively studied in video action localization [189] and video question answering [274], but these are beyond the scope of this survey.”
BackgroundSemantic Cues Enhanced Multimodality Multistream CNN for Action Recognition
Tu, Z., Xie, W. C., Dauwels, J., Li, B., & Yuan, J. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2830102 · Cited by 118 · see citing works
“Semantic cues, such as scene context [5], hand regions [20], human pose [8], human body [3], [19], and interacting”
BackgroundMethodInfluentialTemporal Action Segmentation: An Analysis of Modern Techniques
Ding, G., Şener, F., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327284 · Cited by 124 · see citing works
“Compared to previous works [65], [98], TCNs are”
MethodResultWeakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
Xu, C., & Ding, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00681 · Cited by 147 · see citing works
“[17] present a multi-stream bi-directional recurrent neural network for fine-grained action detection task.”
BackgroundASFormer: Transformer for Action Segmentation
Yi, F., Wen, H., & Jiang, T. (2021) · Proceedings of the British Machine Vision Conference 2021 · doi:10.5244/c.35.49 · Cited by 170 · see citing works
SCC: Semantic Context Cascade for Efficient Action Detection
Heilbron, F. C., Barrios, W., Escorcia, V. A., & Ghanem, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.338 · Cited by 96 · see citing works
“[39] also requires expensive optical flow measurements to describe the video sequence, but instead of using proposals, they rely on an LSTM to encode the temporal evolution of the video.”
MethodWeakly Supervised Energy-Based Learning for Action Segmentation
Li, J., Lei, P., & Todorović, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00634 · Cited by 92 · see citing works
“While a review of fully supervised action segmentation [25, 14, 18, 16] is beyond our scope, it is worth mentioning that our approach uses the same recurrent deep models for frame labeling as in [23, 25, 6].”
MethodA Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Kuehne, H., Richard, A., & Gall, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884469 · Cited by 144 · see citing works
“On the fine grained level, we use a discriminative representation of subactions, modeled by a recurrent neural network as e.g. used by [12], [13], [14], [15].”
MethodDeep Learning-Based Action Detection in Untrimmed Videos: A Survey
Vahdani, E., & Tian, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3193611 · Cited by 93 · see citing works
“In RNN-based methods such as [48], [49], [50], [51], [52], [53], [54], [55], the hidden state encodes the information from previous time steps which is useful to capture temporal dependencies.”
MethodCross-Modal Video Moment Retrieval with Spatial and Language-Temporal Attention
Jiang, B., Huang, X., Yang, C., & Yuan, J. (2019) · International Conference on Multimedia Retrieval · doi:10.1145/3323873.3325019 · Cited by 89 · see citing works
“[36] extended a multi-stream bidirectional RNN network [34] to predict activity labels or activity segments at each time step.”
BackgroundMonitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks
Hajj, H. A., Lamard, M., Conze, P.-H., Cochener, B., & Quellec, G. (2018) · Medical Image Analysis · doi:10.1016/j.media.2018.05.001 · Cited by 86 · see citing works
Video benchmarks of human action datasets: a review
Singh, T., & Vishwakarma, D. K. (2018) · Artificial Intelligence Review · doi:10.1007/s10462-018-9651-1 · Cited by 84 · see citing works
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 80 · see citing works
“tivities [58], which contain action detection [62], [63] (an activity may consist of multiple shorter sequences of actions), gaze direction estimation [64], [65], gesture recognition [66], [67],”
BackgroundAnalyzing human–human interactions: A survey
Stergiou, A., & Poppe, R. W. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.102799 · Cited by 80 · see citing works
“The idea of using multiple independent or dependent regions for various cues, and using separate streams to encode the input, also allows to focus on discriminative regions such as a hand of one person that touches the body of another [128, 88, 142, 156].”
BackgroundAction Recognition With Spatial-Temporal Discriminative Filter Banks
Martínez, B., Modolo, D., Xiong, Y., & Tighe, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00558 · Cited by 75 · see citing works
“Understanding human activities as a fine-grained recognition problem has been explored for some domain specific tasks [17, 21].”
BackgroundObject-Region Video Transformers
Herzig, R., Avraham, E. B., Mangalam, K., Bar, A., Chechik, G., Rohrbach, A., Darrell, T., & Globerson, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00315 · Cited by 77 · see citing works
Filtjens, B., Vanrumste, B., & Slaets, P. (2022) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2022.3230912 · Cited by 81 · see citing works
“Withinthe generic domain of action segmentation, i.e. approaches that are not speci fi cally designed for skeleton data, earlier methods mainly utilized a sliding-window scheme [10], [11].”
MethodTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01211-2 · Cited by 90 · see citing works
Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model
Hakim, N. L., Shih, T. K., Arachchi, S. P. K., Aditya, W., Chen, Y.-C., & Lin, C. (2019) · Sensors · doi:10.3390/s19245429 · Cited by 76 · see citing works
“LSTM has become an important part of deep learning models for image sequence modeling for human action/gesture recognition [35,36].”
BackgroundMulti-Scale Sparse Graph Convolutional Network For the Assessment of Parkinsonian Gait
Guo, R., Shao, X., Zhang, C., & Qian, X. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3068609 · Cited by 74 · see citing works
Coarse-to-Fine Semantic Alignment for Cross-Modal Moment Localization
Hu, Y., Nie, L., Liu, M., Wang, K., Wang, Y., & Hua, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3090521 · Cited by 74 · see citing works
“In 2016, Singh et al. [11] presented a multi-stream Bi-LSTM to effectively obtain the long-term temporal correlations for fine-grained action detection.”
BackgroundGlobal2Local: Efficient Structure Search for Video Action Segmentation
Gao, S., Han, Q., Li, Z., Peng, P., Wang, L., & Cheng, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01653 · Cited by 74 · see citing works
Deep Learning based Motion Prediction for Exoskeleton Robot Control in Upper Limb Rehabilitation
Ren, J.-L., Chien, Y.-H., Chia, E.-Y., Fu, L., & Lai, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794187 · Cited by 72 · see citing works
“neural network was proposed in [17] to predict the action”
BackgroundFine-grained action recognition using dynamic kernels
Yenduri, S., Perveen, N., Chalavadi, V., & Mohan, C. K. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108282 · Cited by 72 · see citing works
Towards Generalized mmWave-based Human Pose Estimation through Signal Augmentation
Xue, H., Cao, Q., Miao, C., Ju, Y., Hu, H., Zhang, A., & Su, L. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3613302 · Cited by 72 · see citing works
“, BiConvLSTM [10, 35, 49]) module to leverage the information from the adjacent frames to improve the generation of the current frame.”
BackgroundMethodInfluentialCompositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
Li, J., Xie, J., Qian, L., Zhu, L., Tang, S., Wu, F. Q., Yang, Y., Zhuang, Y., & Wang, X. E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00304 · Cited by 72 · see citing works
Temporal Alignment Networks for Long-term Video
Han, T., Xie, W., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00292 · Cited by 72 · see citing works
“Research has focused on designing effective modules to capture dependencies between different video chunks [22, 38, 39, 64].”
Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
Liu, J., Li, Y., Song, S., Xing, J., Lan, C., & Zeng, W. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2799968 · Cited by 62 · see citing works
Football Action Recognition Using Hierarchical LSTM
Tsunoda, T., Komori, Y., Matsugu, M., & Harada, T. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.25 · Cited by 68 · see citing works
Generic Event Boundary Detection: A Benchmark for Event Segmentation
Shou, M. Z., Lei, S. W., Wang, W., Ghadiyaram, D., & Feiszli, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00797 · Cited by 65 · see citing works
Gupta, S., & Dinesh, D. A. (2017) · IEEE International Conference on Advanced Networks and Telecommunications Systems · doi:10.1109/ants.2017.8384098 · Cited by 65 · see citing works
Evaluating deep learning models for sentiment classification
Ay, B., Talo, M., Hallaç, İ. R., & Galip Aydin, B. (2018) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.4783 · Cited by 64 · see citing works
Adaptive CU Mode Selection in HEVC Intra Prediction: A Deep Learning Approach
Kuanar, S. P., Rao, K. R., Bilas, M., & Bredow, J. W. (2019) · Circuits Systems and Signal Processing · doi:10.1007/s00034-019-01110-4 · Cited by 64 · see citing works
Fast Weakly Supervised Action Segmentation Using Mutual Consistency
Souri, Y., Fayyaz, M., Minciullo, L., Francesca, G., & Gall, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3089127 · Cited by 69 · see citing works
“Although these networks on their own struggle to learn a temporal model for long input sequences [36], [37], the temporal backbone, which encodes temporal relations at higher resolution, makes it easier for the segment generation subnetwork to learn temporal dependencies as shown by our ablation experiments.”
Deep learning approaches for seizure video analysis: A review
Ahmedt‐Aristizabal, D., Armin, M. A., Hayder, Z., Garcia‐Cairasco, N., Petersson, L., Fookes, C., Denman, S., & McGonigal, A. (2024) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2024.109735 · Cited by 63 · see citing works
Zhang, L., Lim, C. P., & Yu, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106918 · Cited by 59 · see citing works
C2F-TCN: A Framework for Semi- and Fully-Supervised Temporal Action Segmentation
Singhania, D., Rahaman, R., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284080 · Cited by 60 · see citing works
Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey
Asadi, M., Clapés, A., Bellantonio, M., Escalante, H. J., Ponce-López, V., Baró, X., Guyon, I., Shohreh Kasaei, B., & Escalera, S. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_19 · Cited by 58 · see citing works
“Caffe is preferred to other frameworks for its speed and efficiency, especially in ”fused” architectures for action recognition (Singh et al., 2016b; Deng et al., 2015; Diba et al., 2016; Peng and Schmid, 2016).”
BackgroundMethodInfluentialGraph Attention Based Proposal 3D ConvNets for Action Detection
Li, J., Liu, X., Zong, Z., Zhao, W., Zhang, M., & Song, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5893 · Cited by 58 · see citing works
“In the past decade, because of the stronger modeling capabilities of convolutional neural networks, more studies have focused on deep features based action detection (Singh et al. 2016; Gao et al. 2019; Lin et al. 2019; Liu et al. 2019).”
BackgroundRF-Next: Efficient Receptive Field Search for Convolutional Neural Networks
Gao, S., Li, Z., Qi, H., Cheng, M., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183829 · Cited by 57 · see citing works
“Some researchers [63], [64], [65], [66] utilize multi-streammodels to model dependencies from long and short term.”
BackgroundStructural Recurrent Neural Network (SRNN) for Group Activity Analysis
Biswas, S., & Gall, J. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00180 · Cited by 57 · see citing works
“In recent years, deep learning approaches such as [4, 15, 24, 26, 16] have been proposed to recognize activities in videos.”
BackgroundASFormer: Transformer for Action SegmentationPreprint
Yi, F., Wen, H., & Jiang, T. (2021) · British Machine Vision Conference · doi:10.48550/arxiv.2110.08568 · Cited by 319 · see citing works
Continuous Human Action Recognition for Human-machine Interaction: A Review
Gammulle, H., Ahmedt‐Aristizabal, D., Denman, S., Tychsen-Smith, L., Petersson, L., & Fookes, C. (2023) · ACM Computing Surveys · doi:10.1145/3587931 · Cited by 59 · see citing works
“In their empirical comparisons, the authors demonstrate that this architecture is not only capable of outperforming recurrent architectures such as Bi-LSTM [4], but it is also faster to train.”
BackgroundMethodInfluential
- Authors
- Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M.
- Affiliations
- University of Maryland, College Park (US)
- Mitsubishi Electric Research Laboratories (United States) (US)
- Northeastern University (US)
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 1961-1970
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2461621749
- Semantic Scholar
- 260d0b7e94777dcca1cd1a9651fb6c42cba2823f
- Cited by
- 461
- References
- 35
- Citations by year
- 2026: 12 · 2025: 25 · 2024: 36 · 2023: 48 · 2022: 51 · 2021: 65 · 2020: 63 · 2019: 79 · 2018: 41 · 2017: 38 · 2016: 3
- FWCI
- 18.62
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- bidirectional LSTM, human tracking, bounding box tracking, motion features, appearance features, video action recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
