Temporal Action Detection Using a Statistical Language Model
Abstract
While current approaches to action recognition on presegmented video clips already achieve high accuracies, temporal action detection is still far from comparably good results. Automatically locating and classifying the relevant action segments in videos of varying lengths proves to be a challenging task. We propose a novel method for temporal action detection including statistical length and language modeling to represent temporal and contextual structure. Our approach aims at globally optimizing the joint probability of three components, a length and language model and a discriminative action model, without making intermediate decisions. The problem of finding the most likely action sequence and the corresponding segment boundaries in an exponentially large search space is addressed by dynamic programming. We provide an extensive evaluation of each model component on Thumos 14, a large action detection dataset, and report state-of-the-art results on three datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Richard et al. [24] achieved 37.9 mAP@10 and 22.9 mAP@50 on 50 Salads mid-level.
“Their approach achieved 37.9 mAP@10 and 22.9 mAP@50.”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [24]
Cites
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407 · see citing works
The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · see citing works
Combining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · see citing works
Temporal Sequence Modeling for Video Event Detection
Cheng, Y., Fan, Q., Pankanti, S., & Choudhary, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.286 · Cited by 80 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · see citing works
“In the literature, our task goes by the name action segmentation [7, 8, 6, 15, 29, 16, 10] or action detection [28, 20, 21, 25].”
BackgroundMethodInfluentialTemporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_7 · Cited by 1,059 · see citing works
“[14] evaluated their model on the mid-level action granularity of 50 Salads which has 17 action classes.”
MethodResultInfluentialTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01211-2 · Cited by 90 · see citing works
“On THUMOS14, we compare with the contest results (Wang et al. 2014a; Oneata et al. 2014; Richard and Gall 2016) and those from recent works, including the methods that use segment-based 3D CNN (Shou et al.”
MethodThe THUMOS challenge on action recognition for videos “in the wild”
Idrees, H., Zamir, A., Jiang, Y., Gorban, A., Laptev, I., Sukthankar, R., & Shah, M. A. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.10.018 · Cited by 616 · see citing works
“…has been studied in Bojanowski et al. (2014) , Duchenne et al. (2009) , Hoai et al. (2011) , Pirsiavash and Ramanan (2014) , Shou et al. (2016) and Richard and Gall (2016) , while spatiotemporal localization of actions has been addressed in Ke et al. (2007) , Klaser et al. (2010) , Laptev and…”
Temporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892 · see citing works
R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
Xu, H., Das, A., & Saenko, K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.617 · Cited by 780 · see citing works
“The framework proposed in [22] explores the uses of a language model and an activity length model for detection.”
MethodRethinking the Faster R-CNN Architecture for Temporal Action Localization
Chao, Y.-W., Vijayanarasimhan, S., Seybold, B., Ross, D. A., Deng, J., & Sukthankar, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1804.07667 · Cited by 694
“A few other methods add a separate contextual reasoning stage on top of the frame-wise or snippet-wise prediction scores to explicitly model action durations or temporal transitions [34, 55, 20].”
MethodMS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673 · see citing works
“[20] propose a model for temporal action detection that consists of three components: an”
BackgroundSelf-citationShou, Z., Chan, J., Zareian, A., Miyazawa, K., & Chang, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.155 · Cited by 580 · see citing works
“In order to design a model specific to temporal localization, Richard and Gall [43] proposed using statistical length and language modeling to represent temporal and contextual structures.”
BackgroundMethodInfluentialUntrimmedNets for Weakly Supervised Action Recognition and Detection
Wang, L., Xiong, Y., Lin, D., & Gool, L. V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.678 · Cited by 554 · see citing works
“[35]∗ 15.”
Graph Convolutional Networks for Temporal Action Localization
Zeng, R., Huang, W., Gan, C., Tan, M., Rong, Y., Zhao, P., & Huang, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00719 · Cited by 559 · see citing works
“[31] 39.”
G-TAD: Sub-Graph Localization for Temporal Action DetectionPreprint
Xu, M., Zhao, C., Rojas, D., Thabet, A. K., & Ghanem, B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.11462 · Cited by 524 · see citing works
Single Shot Temporal Action Detection
Lin, T., Zhao, X., & Shou, Z. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123343 · Cited by 475 · see citing works
Weakly Supervised Action Localization by Sparse Temporal Pooling Network
Nguyen, P., Han, B., Liu, T., & Prasad, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1712.05080 · Cited by 440
“and is even competitive with many fully supervised approaches. Supervision Method AP@IoU 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fully supervised Heilbron et al. [15] – – – – 13.5 – – – – Richard et al. [24] 39.7 35.7 30.0 23.2 15.2 – – – – Shou et al. [28] 47.7 43.5 36.3 28.7 19.0 10.305.3 – – Yeung et al. [46] 48.9 44.0…”
InfluentialConvNet Architecture Search for Spatiotemporal Feature LearningPreprint
Tran, D., Ray, J., Zheng, S.-F., Chang, S.-F., & Paluri, M. (2017) · arXiv.org · doi:10.48550/arxiv.1708.05038 · Cited by 421 · see citing works
“Others are based on iDT with Fisher Vector [9, 27] or RNN/LSTM [47, 48].”
MethodW-TALC: Weakly-supervised Temporal Activity Localization and Classification
Paul, S., Roy, S., & Roy-Chowdhury, A. (2018) · European Conference on Computer Vision · doi:10.48550/arxiv.1807.10418 · Cited by 400
“tions, but without using their temporal annotations. Supervision IoU ! 0.1 0.2 0.3 0.4 0.5 0.7 Strong Saliency-Pool [26] 04.6 03.4 02.1 01.4 00.9 00.1 FV-DTF [36] 36.6 33.6 27.0 20.8 14.4 - SLM-mgram [39] 39.7 35.7 30.0 23.2 15.2 - S-CNN [44] 47.7 43.5 36.3 28.7 19.0 05.3 Glimpse [64] 48.9 44.0 27.0 20.8 14.4 - PSDF […”
InfluentialGaussian Temporal Awareness Networks for Action Localization
Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., & Mei, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00043 · Cited by 380 · see citing works
“[30] 39.”
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349 · see citing works
Self-citationEnd-to-End Temporal Action Detection With Transformer
Liu, X., Wang, Q., Hu, Y., Tang, X., Zhang, S., Bai, S., & Bai, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3195321 · Cited by 298 · see citing works
ASFormer: Transformer for Action SegmentationPreprint
Yi, F., Wen, H., & Jiang, T. (2021) · British Machine Vision Conference · doi:10.48550/arxiv.2110.08568 · Cited by 319 · see citing works
“IDT+LM [32] 44.4 38.9 27.8 45.8 48.7 - - - - - - ST-CNN [22] 55.9 It is worth noting that, our ASFormer can serve as a strong backbone model and thus can be easily adapted by other works.”
BackgroundInfluentialAutoLoc: Weakly-supervised Temporal Action LocalizationPreprint
Shou, Z., Gao, H., Zhang, L., Miyazawa, K., & Chang, S.-F. (2018) · European Conference on Computer Vision · doi:10.48550/arxiv.1807.08333 · Cited by 290 · see citing works
“Impressive improvement has been made in the past two years to address Temporal Action Localization (TAL) in untrimmed videos [23,15,46,67,68,51,50,36,22,73,17,66,11,10,16,7,6,69,52].”
BackgroundBackground Suppression Network for Weakly-supervised Temporal Action Localization
Lee, P., Uh, Y., & Byun, H. (2020) · arXiv.org · doi:10.48550/arxiv.1911.09963 · Cited by 238
“Full Richard et al. (2016) 39.”
Weakly-Supervised Action Localization With Background Modeling
Nguyen, P., Ramanan, D., & Fowlkes, C. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00560 · Cited by 182 · see citing works
“hich is difficult to do without the actual boundary supervision. Supervision Method AP@IoU 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fully supervised Heilbron et al. [14] – – – – 13.5 – – – – Richard et al. [23] 39.7 35.7 30.0 23.2 15.2 – – – – Shou et al. [26] 47.7 43.5 36.3 28.7 19.0 10.305.3 – – Yeung et al. [34] 48.9 44.…”
InfluentialTemporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202 · see citing works
“[32] resort to a statistical language model for representing temporal and contextual structure in videos of varying lengths.”
BackgroundMethodRevisiting Anchor Mechanisms for Temporal Action Localization
Yang, L., Peng, H., Zhang, D., Fu, J., & Han, J. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3016486 · Cited by 216 · see citing works
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · see citing works
“the exact start and end time of each action in the training set needs to be provided [24, 23, 34].”
BackgroundSelf-citationTemporal Aggregate Representations for Long-Range Video Understanding
Şener, F., Singhania, D., & Yao, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_10 · Cited by 139 · see citing works
“As such, temporal modelling is usually not necessary for simple action recognition [13], but more relevant for understanding complex activities [28, 30].”
BackgroundBoundary-Aware Cascade Networks for Temporal Action Segmentation
Wang, Z., Gao, Z., Wang, L., Li, Z., & Wu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_3 · Cited by 146 · see citing works
Temporal Action Segmentation: An Analysis of Modern Techniques
Ding, G., Şener, F., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327284 · Cited by 124 · see citing works
“[113] combines a visual model that maps visual cues to action probability with a language model applied to the action sequence and captures the segment durations with a length model.”
BackgroundMethodA Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences
Asadi, M., Clapés, A., Bellantonio, M., Escalante, H. J., Ponce-López, V., Baró, X., Guyon, I., Shohreh Kasaei, B., & Escalera, S. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.150 · Cited by 176 · see citing works
“2014 THUMOS-14 A ECCV [44] [46] [95] [111] [88]”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Richard, A., Kuehne, H., Iqbal, A., & Gall, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1805.06875 · Cited by 158
“Richard and Gall [23] propose the use of an explicit statistical language model as well as a length model in a fully supervised formulation.”
BackgroundMethodInfluentialSelf-citationProgressive Boundary Refinement Network for Temporal Action Detection
Liu, Q., & Wang, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6829 · Cited by 141 · see citing works
Temporal Action Localization by Structured Maximal Sums
Yuan, Z., Stroud, J., Lü, T., & Deng, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.342 · Cited by 148 · see citing works
“adopt language models applied to traditional motion features [12].”
MethodA Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Kuehne, H., Richard, A., & Gall, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884469 · Cited by 144 · see citing works
“[40],which introduces a lengthmodel that depends on the overall length of an action with respect to the mean length of the recognized class.”
BackgroundSelf-citationBehrmann, N., Golestaneh, S., Kolter, Z., Gall, J., & Noroozi, M. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2209.00638 · Cited by 138
“[28] use a language and length model to model the probability of action sequences and convert the frame-wise probabilities into action segments using dynamic programming.”
MethodSelf-citationA Pursuit of Temporal Accuracy in General Activity DetectionPreprint
Xiong, Y., Zhao, Y., Wang, L., Lin, D., & Tang, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02716 · Cited by 133 · see citing works
“We first compare with the challenge results [35, 20, 23]. as well as recent works, including methods using segment-based 3D CNN [25], score pyramids [40], and recurrent reinforcement learning [39].”
ResultUnsupervised Learning and Segmentation of Complex Activities from Video
Sener, F., & Yao, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1803.09490 · Cited by 128
“ECTC [9] is based on discriminative clustering, while OCDC [2] and Fine2Coarse [18] are both RNN-based methods.”
BackgroundMethodWeakly supervised learning of actions from transcripts
Kuehne, H., Richard, A., & Gall, J. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.004 · Cited by 119 · see citing works
“classical HMM based speech recognizer [23]. Recently, a combination of CNNs with recurrent neural networks has successfully been applied to a fully supervised segmentation task [24]. Richard and Gall [25] propose to use a length model and a statistical language model in order to capture context information between die…”
MethodSelf-citationWeakly Supervised Energy-Based Learning for Action Segmentation
Li, J., Lei, P., & Todorović, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00634 · Cited by 92 · see citing works
“Contributions: Prior work shows that a statistical language model is useful for weakly supervised learning and modeling of video sequences [17, 9, 19, 22, 3].”
BackgroundMethodObject-Region Video Transformers
Herzig, R., Avraham, E. B., Mangalam, K., Bar, A., Chechik, G., Rohrbach, A., Darrell, T., & Globerson, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00315 · Cited by 77 · see citing works
“A wide variety of methods have been proposed for it, such as actions modeling [1, 61, 95], temporal convolutions [51, 72], boundaries modeling [57, 58], attention [71, 91], structure utilization [92, 97], detection based methods [12, 87], end-to-end approaches [9, 15, 24, 39], recurrent neural networks [62, 74, 89], a…”
Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints
Richard, A., Kuehne, H., & Gall, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00627 · Cited by 89 · see citing works
“We would like to emphasize that our model only shares the factorization with the work of [24].”
BackgroundMethodInfluentialSelf-citationMulti-shot Temporal Event Localization: a Benchmark
Liu, X., Hu, Y. G., Bai, S., Ding, F., Bai, X., & Torr, P. H. S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01241 · Cited by 92 · see citing works
“, two-stage methods [7, 25, 48, 51, 81, 86] and one-stage methods [3, 33, 40, 44, 45, 56, 82], according to whether a standalone proposal generation stage is used.”
MethodStep-by-step Erasion, One-by-one Collection
Zhong, J.-X., Li, N., Kong, W., Zhang, T., Li, T. H., & Li, G. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240511 · Cited by 103 · see citing works
Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation
Ghosh, P., Yao, Y., Davis, L., & Divakaran, A. (2018) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093361 · Cited by 99 · see citing works
“Language models and RNNs/LSTMs are also employed to leverage dependencies/correlations among actions to produce a long sequence of detected actions [37].”
MethodASFormer: Transformer for Action Segmentation
Yi, F., Wen, H., & Jiang, T. (2021) · Proceedings of the British Machine Vision Conference 2021 · doi:10.5244/c.35.49 · Cited by 170 · see citing works
“IDT+LM [32] 44.4 38.9 27.8 45.8 48.7 - - - - - - ST-CNN [22] 55.9 It is worth noting that, our ASFormer can serve as a strong backbone model and thus can be easily adapted by other works.”
BackgroundInfluentialGraph Convolutional Module for Temporal Action Localization in Videos
Zeng, R., Huang, W., Tan, M., Rong, Y., Zhao, P., Huang, J., & Gan, C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3090167 · Cited by 89 · see citing works
Global2Local: Efficient Structure Search for Video Action Segmentation
Gao, S., Han, Q., Li, Z., Peng, P., Wang, L., & Cheng, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01653 · Cited by 74 · see citing works
“Some researchers [10, 49, 57, 58] utilize multi-stream models to model dependencies from both the long and short term.”
BackgroundZeng, R., Gan, C., Chen, P., Huang, W., Wu, Q., & Tan, M. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2922108 · Cited by 96 · see citing works
Activity Graph Transformer for Temporal Action LocalizationPreprint
Nawhal, M., & Mori, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08540 · Cited by 67 · see citing works
“…local-then-global paradigm where the video-level predictions are obtained by postprocessing of local ( i.e . frame-level or snippet-level) predictions using sequence modeling techniques such as recurrent neural networks, temporal convolutions and temporal pooling [11,25,28,32,39,39,41,47, 63,65].”
Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization
Alwassel, H., Heilbron, F. C., & Ghanem, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01240-3_16 · Cited by 80 · see citing works
“Other researchers have used language models [32], action progression analysis [26,34], and high-level semantics [5] to produce high-fidelity action detections.”
BackgroundMarginalized Average Attentional Network for Weakly-Supervised LearningPreprint
Yuan, Y., Lyu, Y., Shen, X., Tsang, I. W.-H., & Yeung, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08586 · Cited by 85 · see citing works
“Supervision Methods AP@IoU 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Richard et al. (Richard & Gall, 2016) 39.7 35.7 30.0 23.2 15.2 - - - - Shou et al. (Shou et al., 2016) 47.7 43.5 36.3 28.7 19.0 10.3 5.3 - - Yeung et al. (Yeung et al., 2016) 48.9 44.0 36.0 26.4 17.1 - - - - Yuan et al. (Yuan et al.,…”
InfluentialSegregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection
Xu, Y.-L., Zhang, C., Cheng, Z., Xie, J., Niu, Y., Pu, S.-L., & Wu, F. (2018) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1811.07460 · Cited by 84
Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering
Kumar, S., Haresh, S., Ahmed, A., Konin, A., Zia, M. Z., & Tran, Q.-H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01954 · Cited by 72 · see citing works
“, transcript supervision [12,18,35,42,50,61,62,64], and the set of actions occurring in a video, i.”
BackgroundSCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation
Fayyaz, M., & Gall, J. (2020) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2003.14266 · Cited by 81
“[26] models context and length information.”
Self-citationA Perceptual Prediction Framework for Self Supervised Event Segmentation
Aakur, S. N., & Sarkar, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00129 · Cited by 79 · see citing works
“Full SVM [19] 15.8 - HTK(64)[20] 56.3 - ED-TCN[27] 43.3 42.0 TCFPN[10] 52.0 54.9 GRU[29] 60.6 - Weak OCDC[6] 8.9 23.4 ECTC[16] 27.7 - Fine2Coarse[28] 33.3 47. proach [30], requires the number of clusters (from ground truth) to achieve the performance whereas our approach does not require such knowledge and is done in…”
MethodInfluentialZero-Shot Anticipation for Instructional Activities
Sener, F., & Yao, A. (2018) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2019.00095 · Cited by 75 · see citing works
“Understanding complex activities and their subactivities has been addressed typically as a supervised video segmentation and recognition problem [29, 40, 43].”
BackgroundHow Much Temporal Long-Term Context is Needed for Action Segmentation?
Bahrami, E., Francesca, G., & Gall, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00950 · Cited by 53 · see citing works
“Method Note Breakfast 50Salads F1@{10, 25, 50} Edit Acc F1@{10, 25, 50} Edit Acc IDT+LM [39] DF1 - - - - - 44.”
Self-citationAction Recognition From Single Timestamp Supervision in Untrimmed Videos
Moltisanti, D., Fidler, S., & Damen, D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.01015 · Cited by 71 · see citing works
“Some works [10, 18, 24] assume the transcript includes knowledge of ‘background’, specifying whether the actions occur in succession or with gaps.”
BackgroundOnline Detection of Action Start in Untrimmed, Streaming Videos
Shou, Z., Pan, J., Chan, J., Miyazawa, K., Mansour, H., Vetro, A., Giró-i-Nieto, X., & Chang, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_33 · Cited by 76 · see citing works
“This task has raised a lot of interest in recent [25,14,45,70,72,48,49,38,24,74,20,67,13,11,18,7,6,73,56,50,69,17].”
BackgroundC2F-TCN: A Framework for Semi- and Fully-Supervised Temporal Action Segmentation
Singhania, D., Rahaman, R., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284080 · Cited by 60 · see citing works
Learning Temporal Co-Attention Models for Unsupervised Video Action Localization
Gong, G., Wang, X., Mu, Y., & Tian, Q. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00984 · Cited by 71 · see citing works
“FS SLM-mgram [33] 30.”
Deep Reinforcement Learning for Surgical Gesture Segmentation and ClassificationPreprint
Liu, D.-C., & Jiang, T. (2018) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.48550/arxiv.1806.08089 · Cited by 70 · see citing works
“…t is the concatenation of current and future feature vectors extracted by TCN, gesture transition probabilities from a language and duration model [17], and a one-hot vector, which is formalized below: where s , s t + k s tcn , s t + k l tcn are respectively TCN features at the current frame, k s…”
BackgroundMethodJoint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences
VidalMata, R. G., Scheirer, W. J., Kukleva, A., Cox, D., & Kuehne, H. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00128 · Cited by 57 · see citing works
“While there has been plenty of work done in the area of action segmentation in video, a vast majority of the approaches rely on frame annotations –fully supervised methods– [9, 7, 13, 25], or some form of metadata –weakly supervised approaches– [16, 5, 24].”
BackgroundInfluentialFast Weakly Supervised Action Segmentation Using Mutual Consistency
Souri, Y., Fayyaz, M., Minciullo, L., Francesca, G., & Gall, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3089127 · Cited by 69 · see citing works
“Similar to [14], [40], we can factorize the term pð ~ LjY;A; LÞ yielding”
BackgroundSelf-citationExploring Temporal Preservation Networks for Precise Temporal Action Localization
Yang, K., Qiao, P., Li, D., Lv, S., & Dou, Y. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12234 · Cited by 77 · see citing works
“However, contextual information is very impor-1 tant in disambiguating local cues (Galleguillos and Belongie 2010).”
BackgroundCoarse to Fine Multi-Resolution Temporal Convolutional NetworkPreprint
Singhania, D., Rahaman, R., & Yao, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10859 · Cited by 43 · see citing works
“Initially, models for temporal video segmentation were based on statistical length modelling [37], RNNs [35, 41, 35], and RNNs combined with Hidden Markov models [27].”
MethodGraph Attention Based Proposal 3D ConvNets for Action Detection
Li, J., Liu, X., Zong, Z., Zhao, W., Zhang, M., & Song, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5893 · Cited by 58 · see citing works
TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action SegmentationPreprint
Ding, L., & Xu, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07818 · Cited by 50 · see citing works
“Here, we use the output features of their spatial CNN as the input to our TricorNet, and compare with their results obtained by the spatiotemporal CNN. Richard et al. [18] use a statistical language model to focus on localizing and classifying segments in videos of varying lengths.”
MethodWith a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition
Kazakos, E., Huh, J., Nagrani, A., Zisserman, A., & Damen, D. (2021) · British Machine Vision Conference · doi:10.48550/arxiv.2111.01024 · Cited by 60
“The most relevant work is [54], which utilises a statistical n-gram language model along with a length model and an action classifier.”
BackgroundAutomatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search
Gao, X., Jin, Y.-M., Dou, Q., & Heng, P.-A. (2020) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra40945.2020.9196674 · Cited by 59 · see citing works
“where strans are probabilities from a statistical language model [31] and shot is a one-hot vector indicating the gesture class given by the last action.”
BackgroundZhang, X., Shi, H.-C., Li, C., Zheng, K., Zhu, X., & Duan, L.-X. (2019) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1902.07370 · Cited by 58
Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization
Zhang, C., Xu, Y., Cheng, Z., Niu, Y. F., Pu, S., Wu, F. Q., & Zou, F. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351044 · Cited by 64 · see citing works
“, the improvement performance also verifies the common effectiveness on both datasets. Table 3: Comparison with state-of-the-arts on THUMOS’14. Method Label 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Richard et al. [18] strong 39.7 35.7 30.0 23.2 15.2 – – S-CNN [21] strong 47.7 43.5 36.3 28.7 19.0 10.3 5.3 CDC [19] strong – – 40.1 2…”
BackgroundInfluentialTwo-Stream Region Convolutional 3D Network for Temporal Activity Detection
Xu, H., Das, A., & Saenko, K. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2921539 · Cited by 49 · see citing works
Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey
Asadi, M., Clapés, A., Bellantonio, M., Escalante, H. J., Ponce-López, V., Baró, X., Guyon, I., Shohreh Kasaei, B., & Escalera, S. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_19 · Cited by 58 · see citing works
RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks
Gao, S., Li, Z., Qi, H., Cheng, M., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183829 · Cited by 57 · see citing works
“Richard and Gall employ [63] dynamic programming to inferencemodels composed of lengthmodel, language model, and action classifier.”
BackgroundMethodActivityNet Challenge 2017 SummaryPreprint
Ghanem, B., Niebles, J. C., Snoek, C. G. M., Heilbron, F. C., Alwassel, H., Krishna, R., Escorcia, V., Hata, K., & Buch, S. (2017) · arXiv.org · doi:10.48550/arxiv.1710.08011 · Cited by 51 · see citing works
Bargal, S. A., Zunino, A., Kim, D., Zhang, J.-M., Murino, V., & Sclaroff, S. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1711.06778 · Cited by 50
“[14] 39.7 Shou et al .”
InfluentialTemporal Tessellation: A Unified Approach for Video Analysis
Kaufman, D., Levi, G., Hassner, T., & Wolf, L. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.20 · Cited by 37 · see citing works
“9 Richard and Gall [43] 39.”
Fine-grained action segmentation using the semi-supervised action GAN
Gammulle, H., Denman, S., Sridharan, S., & Fookes, C. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107039 · Cited by 41 · see citing works
“For comparisons on the 50 salads dataset we use two models that use Improved Dense Trajectories (IDT) [47] together with a Language Model (LM) [48] and Conditional Random Field (CRF) [49] to segment the temporal actions.”
MethodZhang, X., Li, C., Shi, H., Zhu, X., Li, P., & Dong, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2962815 · Cited by 59 · see citing works
“of L adp and L dif are evaluated, the 8 TABLE III COMPARISON OF ACTION LOCALIZATION RESULTS ON THUMOS14. Supervision Method mAP@IoU (%) 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Full Richard and Gall, 2016 [54] 39.7 35.7 30.0 23.2 15.2 - - - - Shou et al., 2016 [24] 47.7 43.5 36.3 28.7 19.0 10.3 5.3 - - Yeung et al., 2016 […”
InfluentialAction Segmentation with Mixed Temporal Domain Adaptation
Chen, M.-H., Li, B., Bao, Y., & AlRegib, G. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093535 · Cited by 33 · see citing works
“IDT+LM [31] 44.”
A Proposal-Based Solution to Spatio-Temporal Action Detection in Untrimmed Videos
Gleason, J., Ranjan, R., Schwarcz, S., Castillo, C., Chen, J.-C., & Chellappa, R. (2018) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.48550/arxiv.1811.08496 · Cited by 40
“.4 0.5 0.6 0.7 Karaman et al. [24] 4.6 3.4 2.4 1.4 0.9 - - Oneata et al. [36] 36.6 33.6 27.0 20.8 14.4 - - Wang et al. [45] 18.2 17.0 14.0 11.7 8.3 - - Caba et al. [4] - - - - 13.5 - - Richard et al. [38] 39.7 35.7 30.0 23.2 15.2 - - Shou et al. [42] 47.7 43.5 36.3 28.7 19.0 10.3 5.3 Yeung et al. [48] 48.9 44.0 36.0 2…”
InfluentialGEARing smart environments for pediatric motor rehabilitation
Kokkoni, E., Mavroudi, E., Zehfroosh, A., Galloway, J. C., Vidal, R., Heinz, J., & Tanner, H. (2020) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-020-0647-0 · Cited by 39 · see citing works
“Recognizing sequences of actions performed by humans in long untrimmed videos (and determining their temporal boundaries) is an active research field [37, 38].”
BackgroundVo-Ho, V.-K., Truong, S., Yamazaki, K., Raj, B., Tran, M., & Le, N. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01702-9 · Cited by 35 · see citing works
“Inspired by anchor-based object detection in 2D images, anchor-based TAPGmethods (Richard and Gall, 2016; Chao et al., 2018;Heilbron et al., 2016; Shou et al., 2016;Gao et al., 2017) pre-define a set of anchor segments and try to fit them into groundtruth action segments in videos.”
MethodMavroudi, E., Bhaskara, D., Sefati, S., Ali, H., & Vidal, R. (2018) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.48550/arxiv.1801.09571 · Cited by 37
“Other types of temporal models include a duration model and language model recently proposed in [27] for modeling action durations and context.”
BackgroundOnline Action Detection in Untrimmed, Streaming Videos - Modeling and EvaluationPreprint
Shou, Z., Pan, J.-T., Chan, J., Miyazawa, K., Mansour, H., Vetro, A., Giró-i-Nieto, X., & Chang, S.-F. (2018) · arXiv.org · doi:10.48550/arxiv.1802.06822 · Cited by 35 · see citing works
“This task has raised a lot of interest in recent [25,14,45,70,72,48,49,38,24,74,20,67,13,11,18,7,6,73,56,50,69,17].”
BackgroundEfficient Action Detection in Untrimmed Videos via Multi-task Learning
Zhu, Y., & Newsam, S. D. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.29 · Cited by 37 · see citing works
“[26] is novel in that it includes a length and language model in addition to an action classifier.”
BackgroundMethodDeep Concept-wise Temporal Convolutional Networks for Action Localization
Li, X., Lin, T., Liu, X., Zuo, W., Li, C., Long, X., He, D., Li, F., Wen, S., & Gan, C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413860 · Cited by 32 · see citing works
“n localization mAP(%) on THUMOS’14. tIoU 0.1 0.2 0.3 0.4 0.5 Karaman et al. [22] 4.6 3.4 2.4 1.4 0.9 Oneata et al. [35] 36.6 33.6 27.0 20.8 14.4 Wang et al. [51] 18.2 17.0 14.0 11.7 8.3 Richard et al [38] 39.7 35.7 30.0 23.2 15.2 Shou et al. [44] 47.7 43.5 36.3 28.7 19.0 Yeung et al. [60] 48.9 44.0 36.0 26.4 17.1 Yuan…”
BackgroundInfluentialTemporal Action Localization in Untrimmed Videos Using Action Pattern Trees
Song, H., Wu, X.-X., Zhu, B., Wu, Y., Chen, M., & Jia, Y.-D. (2019) · IEEE transactions on multimedia · doi:10.1109/tmm.2018.2866370 · Cited by 33 · see citing works
“Different from the method [28], our method works on the videos with varied temporal length rather than limiting the length of videos to a constant.”
MethodPrecise Temporal Action Localization by Evolving Temporal Proposals
Qiu, H., Zheng, Y., Ye, H., Lu, Y., Wang, F., & He, L. (2018) · International Conference on Multimedia Retrieval · doi:10.1145/3206025.3206029 · Cited by 35 · see citing works
“oU thresholds α 0.3 0.4 0.5 0.6 0.7 Wang et al. [39] 14.6 12.1 8.5 4.7 1.5 Oneata et al. [26] 28.8 21.8 15.0 8.5 3.2 Heilbron et al. [3] - - 13.5 - - Escorcia et al. [5] - - 13.9 - - Richard and Gall [29] 30.0 23.2 15.2 - - Yeung et al. [46] 36.0 26.4 17.1 - - PSDF [47] 33.6 26.1 18.8 - - S-CNN [33] 36.3 28.7 19.0 10.…”
InfluentialCoarse-to-Fine Localization of Temporal Action Proposals
Long, F., Yao, T., Qiu, Z., Tian, X., Mei, T., & Luo, J. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2943204 · Cited by 34 · see citing works
Cascaded Pyramid Mining Network for Weakly Supervised Temporal Action LocalizationPreprint
Su, H., Zhao, X., & Lin, T. (2018) · Asian Conference on Computer Vision · doi:10.48550/arxiv.1810.11794 · Cited by 28 · see citing works
“tion, including both full supervision and weak supervision. Supervision Method mAP@tIoU () 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Fully supervised Oneata et al. [19] 36.6 33.6 27.0 20.8 14.4 - - Richard et al. [22] 39.7 35.7 30.0 23.2 15.2 - - Shou et al. [24] 47.7 43.5 36.3 28.7 19.0 10.3 5.3 Yuan et al. [39] 51.0 45.2 36.5 27.…”
InfluentialVideo Imprint Segmentation for Temporal Action Detection in Untrimmed Videos
Gao, Z., Wang, L., Zhang, Q., Niu, Z., Zheng, N., & Hua, G. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018328 · Cited by 25 · see citing works
Hierarchical Vector Quantization for Unsupervised Action Segmentation
Spurio, F., Bahrami, E., Francesca, G., & Gall, J. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2412.17640 · Cited by 26
“Richard and Gall (2016) model the probability of action sequences with language and length models.”
Self-citationSVIP: Sequence VerIfication for Procedures in Videos
Qian, Y., Luo, W., Lian, D., Tang, X., Zhao, P., & Gao, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01927 · Cited by 20 · see citing works
Do We Really Need Temporal Convolutions in Action Segmentation?
Du, D., Su, B., Li, Y., Qi, Z., Si, L., & Shan, Y. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00178 · Cited by 22 · see citing works
“The compared methods include IDT+LM (Richard and Gall 2016), Bi-LSTM (Singh et al.”
MethodTemporal Action Segmentation With High-Level Complex Activity Labels
Ding, G., & Yao, A. (2021) · IEEE transactions on multimedia · doi:10.1109/tmm.2022.3231099 · Cited by 24 · see citing works
Gated forward refinement network for action segmentation
Wang, D., Yuan, Y., & Wang, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.066 · Cited by 18 · see citing works
“50Salads F1@{10,25,50} Edit Acc IDT+LM [23] 44.”
BackgroundSelecting a text similarity measure for a content-based recommender system
Wijewickrema, M., Petras, V., & Dias, N. (2019) · Electronic library · doi:10.1108/el-08-2018-0165 · Cited by 24 · see citing works
Local Temporal Bilinear Pooling for Fine-Grained Action Parsing
Zhang, Y., Tang, S., Muandet, K., Jarvers, C., & Heiko Neumann, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01228 · Cited by 26 · see citing works
“[11] applies a statistical language model to capture action temporal dynamics.”
Background
- Authors
- Richard, A., & Gall, J.
- Affiliations
- University of Bonn (DE)
- ORCID
- Richard, Alexander
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 3131-3140
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2463824207
- Cited by
- 215
- References
- 0
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 8 · 2023: 11 · 2022: 23 · 2021: 35 · 2020: 27 · 2019: 38 · 2018: 38 · 2017: 23 · 2016: 6
- FWCI
- 12.15
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- temporal action localization, statistical language models, dynamic programming, video action recognition, THUMOS14
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
