Weakly-supervised action segmentation with iterative soft boundary assignment
Open accessAbstract
In this work, we address the task of weakly-supervised human action segmentation in long, untrimmed videos. Recent methods have relied on expensive learning models, such as Recurrent Neural Networks (RNN) and Hidden Markov Models (HMM). However, these methods suffer from expensive computational cost, thus are unable to be deployed in large scale. To overcome the limitations, the keys to our design are efficiency and scalability. We propose a novel action modeling framework, which consists of a new temporal convolutional network, named Temporal Convolutional Feature Pyramid Network (TCFPN), for predicting frame-wise action labels, and a novel training strategy for weakly-supervised sequence modeling, named Iterative Soft Boundary Assignment (ISBA), to align action sequences and update the network in an iterative fashion. The proposed framework is evaluated on two benchmark datasets, Breakfast and Hollywood Extended, with four different evaluation metrics. Extensive experimental results show that our methods achieve competitive or superior performance to state-of-the-art methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous temporal models operate on a low temporal resolution of 1-3 frames per second.
“Previous temporal models operate on a low temporal resolution of 1-3 frames per second [15, 17, 5].”
From MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019) · cites this paper as [5]
Section 5.2 (Implementation Details; page unavailable) says the authors use features from previous work and down-sample frames to about 1 fps. This supports a low-resolution example, but the copy does not establish that previous temporal models generally operate across the stated 1–3 fps range. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPrevious temporal models operate on a low temporal resolution of 1-3 frames per second.
“Previous temporal models operate on a low temporal resolution of 1-3 frames per second [15, 17, 5].”
From MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (2023) · cites this paper as [5]
Section 5.2 (Implementation Details; page unavailable) says the authors use features from previous work and down-sample frames to about 1 fps. That supports an example of low temporal resolution, but the paper does not establish that previous temporal models generally operate at 1–3 fps. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPrevious temporal convolutional approaches [15,17] operate on downsampled videos with a temporal resolution of 1-3 frames per second.
“Both approaches in [15, 17] operate on downsampled videos with a temporal resolution of 1-3 frames per second.”
From MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019) · cites this paper as [17]
Section 5.2 (Implementation Details; page unavailable) says this paper uses frames downsampled to about 1 fps, within the claimed range. Section 3 identifies its method as a temporal convolutional network. Our copy does not establish that the other cited approach also uses 1–3 fps. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPrevious temporal models operate on a low temporal resolution of 1-3 frames per second.
“Previous temporal models operate on a low temporal resolution of 1-3 frames per second [15, 17, 5].”
From MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 21 · Cited by 124
Research works printed but not linked: 20
- J.-B. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev, and S. Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4575- 4583, 2016. 1
- P. Bojanowski, R. Lajugie, F. Bach, I. Laptev, J. Ponce, C. Schmid, and J. Sivic. Weakly supervised action labeling in videos under ordering constraints. In European Confer- ence on Computer Vision, pages 628-643. Springer, 2014. 1, 2, 6, 7
- A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Jour- nal of the royal statistical society. Series B (methodological), pages 1-38, 1977. 2
- L. Ding and C. Xu. Tricornet: A hybrid temporal convolu- tional and recurrent network for video action segmentation. arXiv preprint arXiv:1705.07818, 2017. 1, 2 · arXiv
- L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell. Localizing moments in video with natural language. In IEEE International Conference on Computer Vision, 2017. 1
- R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles. Dense-captioning events in videos. In IEEE International Conference on Computer Vision, 2017. 1
- H. Kuehne, J. Gall, and T. Serre. An end-to-end generative framework for video segmentation and recognition. In Ap- plications of Computer Vision (WACV), 2016 IEEE Winter Conference on, pages 1-8. IEEE, 2016. 1, 2, 6
- I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld. Learning realistic human actions from movies. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE, 2008. 1
- C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 1, 2, 3, 6
- T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 2, 3
- M. Marszalek, I. Laptev, and C. Schmid. Actions in context. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 2929-2936. IEEE, 2009. 1
- B. Ni, V. R. Paramathayalan, and P. Moulin. Multiple granu- larity analysis for fine-grained action detection. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 756-763, 2014. 6
- B. Singh, T. K. Marks, M. Jones, O. Tuzel, and M. Shao. A multi-stream bi-directional recurrent neural network for fine- grained action detection. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 1961-1970, 2016. 1, 2
- S. Stein and S. J. McKenna. Combining embedded ac- celerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM inter- national joint conference on Pervasive and ubiquitous com- puting, pages 729-738. ACM, 2013. 6
- M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urta- sun, and S. Fidler. Movieqa: Understanding stories in movies through question-answering. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2016. 1
- C. Xu, S.-H. Hsieh, C. Xiong, and J. J. Corso. Can humans fly? action understanding with multiple classes of actors. In IEEE Conference on Computer Vision and Pattern Recogni- tion, 2015. 1
- J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video de- scription dataset for bridging video and language. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. 1
- S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei. Every moment counts: Dense detailed label- ing of actions in complex videos. International Journal of Computer Vision, pages 1-15, 2015. 1, 2
- H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu. Video paragraph captioning using hierarchical recurrent neural net- works. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. 1
- L. Zhou, C. Xu, and J. J. Corso. Towards automatic learn- ing of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, 2018. 1
- Authors
- Ding, L., & Xu, C.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Rochester (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1803.10699
- OpenAlex
- W2794481829
- Cited by
- 124
- References
- 21
- Citations by year
- 2025: 3 · 2024: 7 · 2023: 16 · 2022: 24 · 2021: 35 · 2020: 25 · 2019: 13 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1741472)
- Nvidia
- Tencent
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- human action recognition, untrimmed video, temporal convolutional network, scalability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
