A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Action recognition has become a rapidly developing research field within the last decade. But with the increasing demand for large scale data, the need of hand annotated data for the training becomes more and more impractical. One way to avoid frame-based human annotation is the use of action order information to learn the respective action classes. In this context, we propose a hierarchical approach to address the problem of weakly supervised learning of human actions from ordered action labels by structuring recognition in a coarse-to-fine manner. Given a set of videos and an ordered list of the occurring actions, the task is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. We address this problem by combining a framewise RNN model with a coarse probabilistic inference. This combination allows for the temporal alignment of long sequences and thus, for an iterative training of both elements. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes as well as by the introduction of a regularizing length prior. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 1×
“ct unseen actions in an untrimmed training set and networks are fine-tuned with respect to the detected instances. Note that classes in the datasets used for pretraining, such as UCF101 [26], Sports1M [27] or Kinetics [28], can overlap with the classes to search for in the untrimmed videos e.g. in case of the Thumos ac…”
MethodGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · cited 2×
“As recurrent neural networks are usually trained using back propagation through time (BPTT) [49], which requires to process the whole sequence in a forward and backward pass and a video can be very long and may easily exceed 10 , 000 frames, the computation time per minibatch can be extremely high.”
BackgroundUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Improving the Fisher Kernel for Large-Scale Image Classification
Perronnin, F., Sánchez, J., & Mensink, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_11 · Cited by 2,588
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669 · cited 1×
“Action recognition has come a long way within the last years, moving forward from highly tuned hand-crafted features as proposed by Wang et al. [1], [18] towards learned features and temporal connections as e.g. proposed by Simonyan and Zisserman [19], Feichtenhofer et al. [20] and Wang et al. [4].”
BackgroundImage Classification with the Fisher Vector: Theory and Practice
Sánchez, J., Perronnin, F., Mensink, T., & Verbeek, J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0636-x · Cited by 1,498 · cited 1×
“The Fisher vector representation [53] for each frame is computed over a sliding window of 20 frames.”
MethodThe Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · cited 5×
“HMM-BOW [16] 28:8 HMM-FV [52] 56:3 TCFPN [24] 52:0”
BackgroundMethodInfluentialSelf-citationA Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461 · cited 1×
“On the fine grained level, we use a discriminative representation of subactions, modeled by a recurrent neural network as e.g. used by [12], [13], [14], [15].”
MethodConnectionist Temporal Modeling for Weakly Supervised Action Labeling
Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_9 · Cited by 226
Temporal Action Detection Using a Statistical Language Model
Richard, A., & Gall, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.341 · Cited by 215 · cited 1×
“[40],which introduces a lengthmodel that depends on the overall length of an action with respect to the mean length of the recognized class.”
BackgroundSelf-citationWeakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · cited 6×
“GRU-RNN [17] 33:3 GRU + length prior 36:7”
BackgroundResultInfluentialSelf-citationWeakly-supervised action segmentation with iterative soft boundary assignment
Ding, L., & Xu, C. (2018) · arXiv · doi:10.48550/arxiv.1803.10699 · Cited by 124 · cited 7×
“HMM-BOW [16] 28:8 HMM-FV [52] 56:3 TCFPN [24] 52:0”
BackgroundMethodInfluentialWeakly supervised learning of actions from transcripts
Kuehne, H., Richard, A., & Gall, J. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.004 · Cited by 119 · cited 12×
“[11], third against the Extended Connectionist Temporal Classification (ECTC) by Huang et al.”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 36
- 10.1109/ICCV.2015.507 · DOI · Paper page
- 10.1109/CVPR.2015.7299065 · DOI · Paper page
- 10.1109/CVPR.2017.364 · DOI · Paper page
- 10.1109/CVPR.2016.412 · DOI · Paper page
- 10.1109/CVPR.2018.00706 · DOI · Paper page
- 10.1109/CVPR.2017.678 · DOI · Paper page
- 10.1109/CVPR.2017.115 · DOI
- 10.1145/2733373.2806226 · DOI · Paper page
- 10.1007/978-3-319-46487-9_52 · DOI
- malmaud, 2015, Whats cookin? Interpreting cooking videos using text, speech and vision, Proc Conf North Amer Chapter Assoc Comput Linguistics Human Language Technol, 143
- kay, 2017, The kinetics human action video dataset, arXiv 1705 06950
- peng, 2014, Action recognition with stacked fisher vectors, Proc Eur Conf Comput Vis, 581
- 10.1109/CVPR.2016.213 · DOI · Paper page
- 10.1109/ICCV.2009.5459279 · DOI · Paper page
- 10.1109/CVPR.2017.337 · DOI · Paper page
- 10.1109/ICASSP.1995.479396 · DOI
- 10.1109/79.790984 · DOI
- 10.1109/WACV.2016.7477701 · DOI · Paper page
- 10.1109/CVPR.2015.7298878 · DOI · Paper page
- 10.1109/CVPR.2015.7299101 · DOI · Paper page
- 10.1145/2733373.2806222 · DOI · Paper page
- simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc Int Conf Neural Inf Process, 568
- wang, 2016, Temporal segment networks: Towards good practices for deep action recognition, Proc Eur Conf Comput Vis, 20
- 10.1109/CVPR.2015.7298599 · DOI · Paper page
- 10.1109/CVPR.2008.4587756 · DOI · Paper page
- 10.1109/CVPR.2016.495 · DOI · Paper page
- 10.1109/CVPR.2009.5206557 · DOI · Paper page
- bojanowski, 2014, Weakly supervised action labeling in videos under ordering constraints, Proc Eur Conf Comput Vis, 628
- jozefowicz, 2015, An empirical exploration of recurrent network architectures, Proc Int Conf Mach Learn, 2342
- 10.3115/v1/W14-4012 · DOI · Paper page
- ballas, 2016, Delving deeper into convolutional networks for learning video representations, Proc Int Conf Learn Representations
- chung, 2014, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv 1412 3555
- 10.1109/LSP.2012.2184795 · DOI
- 10.1016/0165-1684(94)00088-H · DOI
- 10.1007/s10472-017-9561-y · DOI
- 10.1093/ietisy/e90-d.5.825 · DOI · Paper page
Cited by
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673
“6 GRU+length prior [23] - - - - 61.”
MethodSelf-citationMS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349
Self-citationIdentifying behavioral structure from deep variational embeddings of animal motion
Luxem, K., Mocellin, P., Fuhrmann, F., Kürsch, J., Miller, S. R., Palop, J. J., Remy, S., & Bauer, P. (2022) · Communications Biology · doi:10.1038/s42003-022-04080-7 · Cited by 218
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2020) · arXiv · Cited by 350
Self-citation
Metadata
- Authors
- Kuehne, H., Richard, A., & Gall, J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2020-04-01
- Volume
- 42
- Issue
- 4
- Pages
- 765-779
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2906270977
- Cited by
- 93
- References
- 17
- FWCI
- 3.49
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- DFG (KU 3396/2-1, GA 1927/4-1)
- ERC (ARCA (677650))
- AWS
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.