Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Abstract
We present an approach for weakly supervised learning of human actions. Given a set of videos and an ordered list of the occurring actions, the goal is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. To address this task, we propose a combination of a discriminative representation of subactions, modeled by a recurrent neural network, and a coarse probabilistic model to allow for a temporal alignment and inference over long sequences. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes. To this end, we adapt the number of subaction classes by iterating realignment and reestimation during training. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The proposed model achieves state-of-the-art results on 50Salads, GTEA, and the Breakfast dataset.
“Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.”
From MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019) · cites this paper as [25]
Cites
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · see citing works
Learning realistic human actions from movies
Ivan Laptev, A., Marszałek, M., Schmid, C., & Rozenfeld, B. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587756 · Cited by 3,481 · see citing works
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407 · see citing works
Improving the Fisher Kernel for Large-Scale Image Classification
Perronnin, F., Sánchez, J., & Mensink, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_11 · Cited by 2,588 · see citing works
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,121 · see citing works
Image Classification with the Fisher Vector: Theory and Practice
Sánchez, J., Perronnin, F., Mensink, T., & Verbeek, J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0636-x · Cited by 1,498 · see citing works
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459 · see citing works
Marszałek, M., Laptev, I., & Schmid, C. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206557 · Cited by 1,197 · see citing works
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · see citing works
Delving Deeper into Convolutional Networks for Learning Video RepresentationsPreprint
Ballas, N., Yao, L., Pal, C., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06432 · Cited by 715 · see citing works
End-to-End Learning of Action Detection from Frame Glimpses in Videos
Yeung, S., Russakovsky, O., Mori, G., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.293 · Cited by 585 · see citing works
The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · see citing works
Self-citationA database for fine grained activity detection of cooking activities
Rohrbach, M., Amin, S., Andriluka, M., & Schiele, B. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247801 · Cited by 521 · see citing works
Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification
Wu, Z., Wang, X., Jiang, Y., Ye, H., & Xue, X. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806222 · Cited by 483 · see citing works
A Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461 · see citing works
Learning latent temporal structure for complex event detection
Tang, K., Fei-Fei, L., & Koller, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247808 · Cited by 359 · see citing works
Unsupervised Learning from Narrated Instruction Videos
Alayrac, J.-B., Bojanowski, P., Agrawal, N., Šivic, J., Ivan Laptev, A., & Lacoste-Julien, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.495 · Cited by 289 · see citing works
Koller, O., Ney, H., & Bowden, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.412 · Cited by 288 · see citing works
Automatic annotation of human actions in video
Duchenne, O., Laptev, I., Sivic, J., Bach, F., & Ponce, J. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459279 · Cited by 275 · see citing works
Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation
Lea, C., Reiter, A., Vidal, R., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_3 · Cited by 246 · see citing works
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_9 · Cited by 226 · see citing works
Temporal Action Detection Using a Statistical Language Model
Richard, A., & Gall, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.341 · Cited by 215 · cited 2× · see citing works
“the exact start and end time of each action in the training set needs to be provided [24, 23, 34].”
BackgroundSelf-citationWeakly Supervised Action Labeling in Videos under Ordering Constraints
Bojanowski, P., Lajugie, R., Bach, F., Ivan Laptev, A., Ponce, J., Schmid, C., & Šivic, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_41 · Cited by 210 · see citing works
An end-to-end generative framework for video segmentation and recognition
Kuehne, H., Gall, J., & Serre, T. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477701 · Cited by 188 · see citing works
Self-citationWhat’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision
Malmaud, J., Huang, J., Rathod, V., Johnston, N., Rabinovich, A., & Murphy, K. P. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1015 · Cited by 144 · see citing works
Watch-n-patch: Unsupervised understanding of actions and relations
Wu, C., Zhang, J., Savarese, S., & Saxena, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299065 · Cited by 126 · see citing works
Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
Sun, C., Shetty, S., Sukthankar, R., & Nevatia, R. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806226 · Cited by 121 · see citing works
Weakly supervised learning of actions from transcripts
Kuehne, H., Richard, A., & Gall, J. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.004 · Cited by 119 · see citing works
Self-citationWebly-Supervised Video Recognition by Mutually Voting for Relevant Web Images and Web Video Frames
Gan, C., Sun, C., Duan, L., & Gong, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_52 · Cited by 95 · see citing works
Using a stochastic context-free grammar as a language model for speech recognition
Jurafsky, D., Wooters, C., Segal, J., Stolcke, A., Fosler, E., Tajchaman, G., & Morgan, N. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1995.479396 · Cited by 89 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673 · see citing works
“Earlier approaches for action segmentation can be grouped into two categories: sliding window approaches [22, 11, 19], that use temporal windows of different scales to detect action segments, and hybrid approaches that apply a coarse temporal modeling using Markov models on top of frame-wise classifiers [13, 16, 21].”
BackgroundMethodSelf-citationSkeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates
Liu, J., Shahroudy, A., Xu, D., Kot, A. C., & Wang, G. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2771306 · Cited by 535 · see citing works
Weakly Supervised Action Localization by Sparse Temporal Pooling Network
Nguyen, P., Han, B., Liu, T., & Prasad, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00706 · Cited by 401 · see citing works
W-TALC: Weakly-Supervised Temporal Activity Localization and Classification
Paul, S., Roy, S., & Roy–Chowdhury, A. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_35 · Cited by 347 · see citing works
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349 · see citing works
“Other approaches apply a coarse temporal modeling using Markov models on top of frame-wise classifiers [8], [9], [10].”
MethodSelf-citationAutoLoc: Weakly-Supervised Temporal Action Localization in Untrimmed Videos
Shou, Z., Gao, H., Zhang, L., Miyazawa, K., & Chang, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_10 · Cited by 289 · see citing works
Anticipative Video Transformer
Girdhar, R., & Grauman, K. L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01325 · Cited by 214 · see citing works
Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs
Koller, O., Zargaran, S., Ney, H., & Bowden, R. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1121-3 · Cited by 204 · see citing works
Cross-Task Weakly Supervised Learning From Instructional Videos
Zhukov, D., Alayrac, J.-B., Cinbiş, R. G., Fouhey, D. F., Laptev, I., & Šivic, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00365 · Cited by 208 · see citing works
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization
Narayan, S., Cholakkal, H., Khan, F. S., & Shao, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00877 · Cited by 179 · see citing works
When will you do what? - Anticipating Temporal Occurrences of Activities
Farha, Y. A., Richard, A., & Gall, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00560 · Cited by 180 · see citing works
Self-citationAlleviating Over-segmentation Errors by Detecting Action Boundaries
Ishikawa, Y., Kasai, S., Aoki, Y., & Kataoka, H. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00237 · Cited by 171 · see citing works
Temporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202 · see citing works
Improving Action Segmentation via Graph-Based Temporal Reasoning
Huang, Y., Sugano, Y., & Sato, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01404 · Cited by 160 · see citing works
Chang, C.-Y., Huang, D.-A., Sui, Y., Fei-Fei, L., & Niebles, J. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00366 · Cited by 150 · see citing works
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · see citing works
Boundary-Aware Cascade Networks for Temporal Action Segmentation
Wang, Z., Gao, Z., Wang, L., Li, Z., & Wu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_3 · Cited by 146 · see citing works
Temporal Aggregate Representations for Long-Range Video Understanding
Şener, F., Singhania, D., & Yao, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_10 · Cited by 139 · see citing works
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Richard, A., Kuehne, H., Iqbal, A., & Gall, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00771 · Cited by 131 · see citing works
Self-citationUnsupervised Learning of Action Classes With Continuous Temporal Embedding
Kukleva, A., Kuehne, H., Şener, F., & Gall, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01234 · Cited by 128 · see citing works
Self-citationWeakly-supervised action segmentation with iterative soft boundary assignment
Ding, L., & Xu, C. (2018) · arXiv · doi:10.48550/arxiv.1803.10699 · Cited by 124 · see citing works
Action Segmentation With Joint Self-Supervised Temporal Domain Adaptation
Chen, M.-H., Li, B., Bao, Y., AlRegib, G., & Kira, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00947 · Cited by 118 · see citing works
Unsupervised Learning and Segmentation of Complex Activities from Video
Şener, F., & Yao, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00873 · Cited by 114 · see citing works
Temporal Action Segmentation: An Analysis of Modern Techniques
Ding, G., Şener, F., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327284 · Cited by 124 · see citing works
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
Xu, C., & Ding, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00681 · Cited by 147 · see citing works
Temporal Structure Mining for Weakly Supervised Action Detection
Yu, T., Ren, Z., Li, Y., Yan, E., Xu, N., & Yuan, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00562 · Cited by 91 · see citing works
Weakly Supervised Energy-Based Learning for Action Segmentation
Li, J., Lei, P., & Todorović, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00634 · Cited by 92 · see citing works
A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Kuehne, H., Richard, A., & Gall, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884469 · Cited by 144 · see citing works
Self-citationAction Sets: Weakly Supervised Action Segmentation Without Ordering Constraints
Richard, A., Kuehne, H., & Gall, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00627 · Cited by 89 · see citing works
Self-citationTemporal Action Segmentation from Timestamp Supervision
Li, Z., Farha, Y. A., & Gall, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00826 · Cited by 90 · see citing works
Self-citationDeep Learning-Based Action Detection in Untrimmed Videos: A Survey
Vahdani, E., & Tian, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3193611 · Cited by 93 · see citing works
Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation
Sarfraz, M. S., Murray, N., Sharma, V., Diba, A., Gool, L. V., & Stiefelhagen, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01107 · Cited by 88 · see citing works
Multi-shot Temporal Event Localization: a Benchmark
Liu, X., Hu, Y. G., Bai, S., Ding, F., Bai, X., & Torr, P. H. S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01241 · Cited by 92 · see citing works
A Perceptual Prediction Framework for Self Supervised Event Segmentation
Aakur, S. N., & Sarkar, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00129 · Cited by 79 · see citing works
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Chen, B., Rouditchenko, A., Duarte, K., Kuehne, H., Thomas, S., Boggust, A., Panda, R., Kingsbury, B. E. D., Feris, R., Harwath, D. F., et al. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00791 · Cited by 77 · see citing works
Self-citationGlobal2Local: Efficient Structure Search for Video Action Segmentation
Gao, S., Han, Q., Li, Z., Peng, P., Wang, L., & Cheng, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01653 · Cited by 74 · see citing works
SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation
Fayyaz, M., & Gall, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00058 · Cited by 70 · see citing works
Self-citationIntra- and Inter-Action Understanding via Temporal Action Parsing
Shao, D., Zhao, Y., Dai, B., & Lin, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00081 · Cited by 66 · see citing works
D2-Net: Weakly-Supervised Action Localization via Discriminative Embeddings and Denoised Activations
Narayan, S., Cholakkal, H., Hayat, M., Khan, F. S., Yang, M., & Shao, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01335 · Cited by 70 · see citing works
Future Transformer for Long-term Action Anticipation
Gong, D., Lee, J., Kim, M., Ha, S. J., & Cho, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00306 · Cited by 67 · see citing works
Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering
Kumar, S., Haresh, S., Ahmed, A., Konin, A., Zia, M. Z., & Tran, Q.-H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01954 · Cited by 72 · see citing works
Fast Weakly Supervised Action Segmentation Using Mutual Consistency
Souri, Y., Fayyaz, M., Minciullo, L., Francesca, G., & Gall, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3089127 · Cited by 69 · see citing works
Self-citationJoint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences
VidalMata, R. G., Scheirer, W. J., Kukleva, A., Cox, D., & Kuehne, H. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00128 · Cited by 57 · see citing works
Self-citationZero-Shot Anticipation for Instructional Activities
Sener, F., & Yao, A. (2018) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2019.00095 · Cited by 75 · see citing works
RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks
Gao, S., Li, Z., Qi, H., Cheng, M., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183829 · Cited by 57 · see citing works
Continuous Human Action Recognition for Human-machine Interaction: A Review
Gammulle, H., Ahmedt‐Aristizabal, D., Denman, S., Tychsen-Smith, L., Petersson, L., & Fookes, C. (2023) · ACM Computing Surveys · doi:10.1145/3587931 · Cited by 59 · see citing works
Weakly Supervised Temporal Action Localization Using Deep Metric Learning
Islam, A., & Radke, R. J. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093620 · Cited by 52 · see citing works
Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks
Rashid, M., Kjellström, H., & Lee, Y. J. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093404 · Cited by 51 · see citing works
Set-Constrained Viterbi for Set-Supervised Action Segmentation
Li, J., & Todorović, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01083 · Cited by 50 · see citing works
Self-Regulated Learning for Egocentric Video Activity Anticipation
Qi, Z., Wang, S., Su, C., Su, L., Huang, Q., & Tian, Q. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059923 · Cited by 57 · see citing works
RefineLoc: Iterative Refinement for Weakly-Supervised Action Localization
Pardo, A., Alwassel, H., Heilbron, F. C., Thabet, A., & Ghanem, B. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00336 · Cited by 52 · see citing works
Uncertainty-Aware Anticipation of Activities
Farha, Y. A., & Gall, J. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00151 · Cited by 50 · see citing works
Self-citationFast and Unsupervised Action Boundary Detection for Action Segmentation
Du, Z., Wang, X., Zhou, G., & Wang, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00332 · Cited by 45 · see citing works
Unsupervised Procedure Learning via Joint Dynamic Summarization
Elhamifar, E., & Naing, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00644 · Cited by 45 · see citing works
A Context Knowledge Map Guided Coarse-to-Fine Action Recognition
Ji, Y., Zhan, Y., Yang, Y., Xu, X., Shen, F., & Shen, H. T. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2952088 · Cited by 42 · see citing works
Long-Term Anticipation of Activities with Cycle Consistency
Farha, Y. A., Ke, Q., Schiele, B., & Gall, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-71278-5_12 · Cited by 42 · see citing works
Self-citationEfficient Two-Step Networks for Temporal Action Segmentation
Li, Y., Dong, Z., Liu, K., Feng, L., Hu, L., Zhu, J., Li, X., Wang, Y., & Liu, S. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.121 · Cited by 43 · see citing works
Park, J., Kim, D., Huh, S., & Jo, S. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108764 · Cited by 43 · see citing works
Action Shuffle Alternating Learning for Unsupervised Action Segmentation
Li, J., & Todorović, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01244 · Cited by 43 · see citing works
A review of video action recognition based on 3D convolution
Huang, X., & Cai, Z.-B. (2023) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.108713 · Cited by 39 · see citing works
Self-supervised Multi-task Procedure Learning from Instructional Videos
Elhamifar, E., & Huynh, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_33 · Cited by 36 · see citing works
Vats, K., Fani, M., Walters, P., Clausi, D. A., & Zelek, J. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00449 · Cited by 35 · see citing works
Learning Discriminative Prototypes with Dynamic Time Warping
Chang, X., Tung, F., & Mori, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00829 · Cited by 35 · see citing works
Mavroudi, E., Bhaskara, D., Sefati, S., Ali, H., & Vidal, R. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00174 · Cited by 33 · see citing works
Modeling Sub-Actions for Weakly Supervised Temporal Action Localization
Huang, L., Huang, Y., Ouyang, W., & Wang, L. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3078324 · Cited by 34 · see citing works
Action Segmentation with Mixed Temporal Domain Adaptation
Chen, M.-H., Li, B., Bao, Y., & AlRegib, G. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093535 · Cited by 33 · see citing works
SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation
Wang, Z., Chen, H., Li, X., Liu, C., Xiong, Y., Tighe, J., & Fowlkes, C. C. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00025 · Cited by 33 · see citing works
Adaptive receptive field U-shaped temporal convolutional network for vulgar action segmentation
Cao, J., Xu, R., Lin, X., Qin, F., Peng, Y. K., & Shao, Y. (2023) · Neural Computing and Applications · doi:10.1007/s00521-022-08190-5 · Cited by 29 · see citing works
Unsupervised Video Representation Learning by Bidirectional Feature Prediction
Behrmann, N., Gall, J., & Noroozi, M. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00171 · Cited by 28 · see citing works
Self-citationMa, H., Yang, Z., & Liu, H. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3132016 · Cited by 27 · see citing works
SA-Net: Robust State-Action Recognition for Learning from Observations
Soans, N., Asali, E., Hong, Y., & Doshi, P. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197393 · Cited by 25 · see citing works
Weakly-Supervised Action Segmentation and Alignment via Transcript-Aware Union-of-Subspaces Learning
Lu, Z., & Elhamifar, E. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00798 · Cited by 25 · see citing works
P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision
Zhao, H., Hadji, I., Dvornik, N., Derpanis, K. G., Wildes, R. P., & Jepson, A. D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00295 · Cited by 25 · see citing works
Set-Supervised Action Learning in Procedural Task Videos via Pairwise Order Consistency
Lu, Z., & Elhamifar, E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01928 · Cited by 25 · see citing works
Self-Attention Temporal Convolutional Network for Long-Term Daily Living Activity Detection
Dai, R., Minciullo, L., Garattoni, L., Francesca, G., & Brémond, F. (2019) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2019.8909841 · Cited by 25 · see citing works
Learning to Segment Actions from Observation and Narration
Fried, D., Alayrac, J.-B., Blunsom, P., Dyer, C., Clark, S. C., & Nematzadeh, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.231 · Cited by 25 · see citing works
Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos
Ghoddoosian, R., Dwivedi, I., Agarwal, N., Choi, C., & Dariush, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01341 · Cited by 24 · see citing works
A flexible model for training action localization with varying levels of supervisionPreprint
Chéron, G., Alayrac, J.-B., Ivan Laptev, A., & Schmid, C. (2018) · arXiv (Cornell University) · Cited by 24 · see citing works
Anchor-Constrained Viterbi for Set-Supervised Action Segmentation
Li, J., & Todorović, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00968 · Cited by 23 · see citing works
TL;DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency
Narasimhan, M., Nagrani, A., Sun, C., Rubinstein, M., Darrell, T., Rohrbach, A., & Schmid, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_31 · Cited by 23 · see citing works
Timestamp-Supervised Action Segmentation with Graph Convolutional Networks
Khan, H., Haresh, S., Ahmed, A., Siddiqui, S., Konin, A., Zia, M. Z., & Tran, Q.-H. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981351 · Cited by 22 · see citing works
SVIP: Sequence VerIfication for Procedures in Videos
Qian, Y., Luo, W., Lian, D., Tang, X., Zhao, P., & Gao, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01927 · Cited by 20 · see citing works
Weakly-Supervised Action Segmentation and Unseen Error Detection in Anomalous Instructional Videos
Ghoddoosian, R., Dwivedi, I., Agarwal, N., & Dariush, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00929 · Cited by 20 · see citing works
Transferring Knowledge From Text to Video: Zero-Shot Anticipation for Procedural Actions
Sener, F., Saraf, R., & Yao, A. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3218596 · Cited by 20 · see citing works
Ghoddoosian, R., Sayed, S., & Athitsos, V. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00020 · Cited by 18 · see citing works
Temporal Action Segmentation With High-Level Complex Activity Labels
Ding, G., & Yao, A. (2021) · IEEE transactions on multimedia · doi:10.1109/tmm.2022.3231099 · Cited by 24 · see citing works
Gated forward refinement network for action segmentation
Wang, D., Yuan, Y., & Wang, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.066 · Cited by 18 · see citing works
Gao, S., Wu, R., Liu, S., Erkan, U., Toktaş, A., Liu, J., & Tang, X. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02251-y · Cited by 19 · see citing works
TSRN: two-stage refinement network for temporal action segmentation
Tian, X., Jin, Y., & Tang, X. (2023) · Pattern Analysis and Applications · doi:10.1007/s10044-023-01166-8 · Cited by 18 · see citing works
Permutation-Aware Activity Segmentation via Unsupervised Frame-to-Segment Alignment
Tran, Q.-H., Mehmood, A., Ahmed, M., Naufil, M., Zafar, A., Konin, A., & Zia, M. Z. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00630 · Cited by 18 · see citing works
FIFA: Fast Inference Approximation for Action Segmentation
Souri, Y., Farha, Y. A., Despinoy, F., Francesca, G., & Gall, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92659-5_18 · Cited by 16 · see citing works
Self-citationBottom-up improved multistage temporal convolutional network for action segmentation
Chen, W., Chai, Y., Qi, M., Sun, H., Pu, Q., Kong, J., & Zheng, C. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03382-x · Cited by 14 · see citing works
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2020) · arXiv · Cited by 350 · see citing works
“Other approaches apply a coarse temporal modeling using Markov models on top of frame-wise classifiers [8], [9], [10].”
MethodSelf-citationMulti-Hierarchical Category Supervision for Weakly-Supervised Temporal Action Localization
Li, G., Li, J., Wang, N., Ding, X., Li, Z., & Gao, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3124671 · Cited by 14 · see citing works
Leveraging triplet loss for unsupervised action segmentation
Bueno-Benito, E., Vecino, B. T., & Dimiccoli, M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00520 · Cited by 13 · see citing works
Kuehne, H., Iqbal, A., Richard, A., & Gall, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01012 · Cited by 13 · see citing works
Self-citationDilated Transformer with Feature Aggregation Module for Action Segmentation
Du, Z., & Wang, Q. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-11133-9 · Cited by 13 · see citing works
Xu, A., & Zheng, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01728 · Cited by 12 · see citing works
CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation
Wang, X., Liu, J., Mei, T., & Luo, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3263387 · Cited by 13 · see citing works
Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions
Li, Z., He, L., & Xu, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_33 · Cited by 12 · see citing works
- Authors
- Richard, A., Kuehne, H., & Gall, J.
- Affiliations
- University of Bonn (DE)
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 1273-1282
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2600081845
- Cited by
- 195
- References
- 35
- Citations by year
- 2026: 6 · 2025: 15 · 2024: 16 · 2023: 19 · 2022: 28 · 2021: 42 · 2020: 22 · 2019: 22 · 2018: 23 · 2017: 2
- FWCI
- 5.52
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- weakly supervised learning, human action recognition, temporal action segmentation, recurrent neural networks, temporal alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
