MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
The DOI matches its Crossref record, with a small difference in title or year; 56 of the 60 references it cites were found in a registry or an index; 4 of the 14 sources its claims cite were checked for support.
Abstract
With the success of deep learning in classifying short trimmed videos, more attention has been focused on temporally segmenting and classifying activities in long untrimmed videos. State-of-the-art approaches for action segmentation utilize several layers of temporal convolution and temporal pooling. Despite the capabilities of these approaches in capturing temporal dependencies, their predictions suffer from over-segmentation errors. In this paper, we propose a multi-stage architecture for the temporal action segmentation task that overcomes the limitations of the previous approaches. The first stage generates an initial prediction that is refined by the next ones. In each stage we stack several layers of dilated temporal convolutions covering a large receptive field with few parameters. While this architecture already performs well, lower layers still suffer from a small receptive field. To address this limitation, we propose a dual dilated layer that combines both large and small receptive fields. We further decouple the design of the first stage from the refining stages to address the different requirements of these stages. Extensive evaluation shows the effectiveness of the proposed model in capturing long-range dependencies and recognizing action segments. Our models achieve state-of-the-art results on three datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.
Overview
CoChat Summary
TL;DR
The paper introduces MS-TCN, a multi-stage temporal convolutional network for fully supervised temporal action segmentation that operates at full temporal resolution and refines predictions stage by stage, trained with a combined cross-entropy and truncated-MSE smoothing loss. It reports state-of-the-art results on 50Salads, GTEA, and Breakfast.
Key findings
- The multi-stage architecture substantially outperforms a single-stage TCN of equal parameter count (48 layers) on 50Salads: F1@{10,25,50} of 76.3/74.0/64.5 vs 49.0/46.4/40.2, Edit 67.9 vs 40.7, Acc 80.7 vs 78.0 — a margin of up to 27%.
- The proposed smoothing loss improves over cross-entropy alone on 50Salads: L_cls gives F1@{10,25,50} 71.3/69.7/60.7, Edit 64.2, Acc 79.9, while L_cls + λL_T-MSE gives 76.3/74.0/64.5, Edit 67.9, Acc 80.7 (an absolute F1/edit improvement of ~5%); it also beats a KL-divergence smoothing loss (71.9/69.3/60.1, Edit 64.6, Acc 80.2).
- Four stages is optimal; adding a fifth degrades performance (F1@10 76.4 but F1@25/50 drop to 73.4/63.6 and Acc to 79.5), attributed to over-fitting from more parameters.
- Concatenating features to the input probabilities of higher stages causes a large drop (F1@10 56.2 vs 76.3, Edit 47.6 vs 67.9, ~20% drop), so only probabilities are passed.
- The model outperforms state-of-the-art methods on all three datasets, with up to 12.6% improvement in frame-wise accuracy on 50Salads; on GTEA with fine-tuning it reaches F1@{10,25,50} 87.5/85.4/74.6, Edit 81.4, Acc 79.2.
Methods
- Architecture: a single-stage TCN of dilated 1D convolutions (kernel size 3, acausal, dilation doubled per layer 1–512, residual connections, no pooling or fully connected layers), stacked into a multi-stage model where each stage refines the previous stage's frame-wise probabilities; implementation uses 4 stages, 10 dilated layers per stage, 64 filters, dropout, Adam with learning rate 0.0005.
- Loss: sum over stages of a per-stage loss combining cross-entropy classification loss and a truncated mean squared error smoothing loss over frame-wise log-probabilities, with τ = 4 and λ = 0.15.
- Data and evaluation: I3D features at 15 fps on 50Salads (50 videos, 17 classes, 5-fold cross-validation), GTEA (28 videos, 11 classes, leave-one-subject-out), and Breakfast (1,712 videos, 48 actions, standard 4 splits); metrics are frame-wise accuracy, segmental edit distance, and segmental F1 at IoU thresholds 10%, 25%, 50%.
Limitations
- Stated: the weakly supervised setting is explicitly out of scope ("the weakly supervised case is beyond the scope of this paper"); performance is slightly worse on longer videos due to the limited receptive field (GTEA ≥1.5 min: F1@10 81.2, Edit 71.8, Acc 75.9 vs <1 min: 89.6, 82.5, 76.6); a fifth stage degrades results, possibly from over-fitting.
- Evident from the text: evaluation is confined to cooking/egocentric activity datasets (50Salads, GTEA, Breakfast), so generalization to other domains is untested; the model is fully supervised and requires frame-level annotations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
3 Valid · 1 Not valid · 5 Not checked · 71 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.”
- Also stated in: § 1. Introduction · plain
“Our approach achieves state-of-the-art results on three challenging benchmarks for action segmentation: 50Salads [25], Georgia Tech Egocentric Activities (GTEA) [8], and the Breakfast dataset [12].”
What the sources say
- Not checked — no full text of the sourcecites [25] [25] Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In ACM International Joint Conference on Pervasive and Ubiquitous Computing , pages 729-738, 2013. - source
- Supportedcites [8] [8] Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocentric activities. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3281-3288, 2011. - source
The citation identifies the GTEA dataset, not the citing authors’ own model or its results. Section 5.1 describes the paper’s egocentric daily-activity dataset, and the Conclusion identifies its release as GTEA. No page numbers are available in our copy. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [12] [12] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goaldirected human activities. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 780787, 2014. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § 1. Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in the table, all of these models achieve a comparable frame-wise accuracy.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Extensive evaluation on three datasets shows the effectiveness of our model in capturing long range dependencies between action classes and producing high quality predictions.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This architecture is trained using a combination of a classification loss and a proposed smoothing loss that penalizes over-segmentation errors.”
- Also stated in: § 1. Introduction · plain
“we further employ a smoothing loss during training which penalizes over-segmentation errors in the predictions.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1. Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4, the impact of λ is very small on the performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
45 more claims
Create a free account to see all 80 claims and every source verdict.
- Probably not a substantial claim. 8
- Borderline: one of the tests for a substantial claim is unclear. 1
- It may state two separate claims as one. 22
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 4×
“While the dual dilated layer combines local and global context from the input sequence, there are other techniques in the literature for fusing multi-scale features like feature pyramid networks (FPN) [47].”
BackgroundInfluentialQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1705.07750 · Cited by 10,485
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · cited 7×
“Most of the efforts, however, have been focused on classifying short trimmed videos [1], [2], [3], [4].”
BackgroundMethodInfluentialTwo-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · cited 1×
“Most of the efforts, however, have been focused on classifying short trimmed videos [1], [2], [3], [4].”
BackgroundStacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · cited 1×
“Stacking several predictors sequentially has shown significant improvements in many tasks like human pose estimation [44], [45], [46].”
BackgroundWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007 · cited 1×
“Most of the efforts, however, have been focused on classifying short trimmed videos [1], [2], [3], [4].”
BackgroundRes2Net: A New Multi-Scale Backbone Architecture
Gao, S.-H., Cheng, M.-M., Zhao, K., Zhang, X.-Y., Yang, M.-H., & Torr, P. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2938758 · Cited by 3,385 · cited 1×
“Multi-scale information is important for speech synthesis [29] and image recognition [30].”
BackgroundSelf-citationWei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966 · cited 1×
“Stacking several predictors sequentially has shown significant improvements in many tasks like human pose estimation [44], [45], [46].”
BackgroundTemporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · cited 7×
“For that reason, we use the segmental F1 score as a measure of the quality of the prediction as proposed by [11].”
BackgroundMethodInfluentialTemporal Action Localization in Untrimmed Videos via Multi-stage CNNs
Shou, Z., Wang, D., & Chang, S.-F. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.119 · Cited by 968 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
Xu, H., Das, A., & Saenko, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.617 · Cited by 783 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundRethinking the Faster R-CNN Architecture for Temporal Action Localization
Chao, Y.-W., Vijayanarasimhan, S., Seybold, B., Ross, D. A., Deng, J., & Sukthankar, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00124 · Cited by 725 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundBMN: Boundary-Matching Network for Temporal Action Proposal Generation
Lin, T., Liu, X., Li, X., Ding, E., & Wen, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00399 · Cited by 694 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundSpatiotemporal Multiplier Networks for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Wildes, R. P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.787 · Cited by 679
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673 · cited 3×
“To be consistent with [14], we set Lr 1⁄4 10 which achieves a reasonable tradeoff on performance with respect to all evaluation metrics as shown in Table 9.”
BackgroundMethodInfluentialSelf-citationSpatiotemporal Residual Networks for Video Action Recognition
Feichtenhofer, C., Pinz, A. J., & Wildes, R. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02155 · Cited by 597
End-to-End Learning of Action Detection from Frame Glimpses in Videos
Yeung, S., Russakovsky, O., Mori, G., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.293 · Cited by 585 · cited 1×
“Other approaches combine the proposal generation and classification in a single-stage architecture which enables end-to-end training [41], [42], [43].”
BackgroundThe Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · cited 3×
“Egocentric Activities (GTEA) [16], and the Breakfast dataset [17].”
BackgroundMethodLearning to recognize objects in egocentric activities
Fathi, A., Ren, X., & Rehg, J. M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995444 · Cited by 557 · cited 2×
“Egocentric Activities (GTEA) [16], and the Breakfast dataset [17].”
BackgroundMethodGraph Convolutional Networks for Temporal Action Localization
Zeng, R., Huang, W., Gan, C., Tan, M., Rong, Y., Zhao, P., & Huang, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00719 · Cited by 551 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundA database for fine grained activity detection of cooking activities
Rohrbach, M., Amin, S., Andriluka, M., & Schiele, B. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247801 · Cited by 521 · cited 2×
“In earlier approaches, a sliding window approach is applied with non-maximum suppression [5], [6].”
BackgroundMethodTURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
Gao, J., Yang, Z., Sun, C., Chen, K., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.392 · Cited by 498 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundA Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461
Combining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · cited 3×
“Our approach achieves state-of-the-art results on three challenging benchmarks for action segmentation: 50Salads [15], Georgia Tech”
MethodSST: Single-Stream Temporal Action Proposals
Buch, S., Escorcia, V., Shen, C., Ghanem, B., & Niebles, J. C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.675 · Cited by 433 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundUnderstanding egocentric activities
Fathi, A., Farhadi, A., & Rehg, J. M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126269 · Cited by 384 · cited 1×
“Other approaches model actions based on the change in the state of objects and materials [18] or based on the interactions between hands and objects [19].”
BackgroundGaussian Temporal Awareness Networks for Action Localization
Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., & Mei, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00043 · Cited by 380 · cited 1×
“Other approaches combine the proposal generation and classification in a single-stage architecture which enables end-to-end training [41], [42], [43].”
BackgroundLearning latent temporal structure for complex event detection
Tang, K., Fei-Fei, L., & Koller, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247808 · Cited by 355
Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation
Lea, C., Reiter, A., Vidal, R., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_3 · Cited by 246 · cited 1×
“Other approaches apply a coarse temporal modeling using Markov models on top of frame-wise classifiers [8], [9], [10].”
MethodEnd-to-End, Single-Stream Temporal Action Detection in Untrimmed Videos
Buch, S., Escorcia, V., Ghanem, B., & Niebles, J. C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.93 · Cited by 240 · cited 1×
“Other approaches combine the proposal generation and classification in a single-stage architecture which enables end-to-end training [41], [42], [43].”
BackgroundCascaded Boundary Regression for Temporal Action Detection
Gao, J., Yang, Z., & Nevatia, R. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.52 · Cited by 232 · cited 1×
“The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
BackgroundConnectionist Temporal Modeling for Weakly Supervised Action Labeling
Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_9 · Cited by 226
Temporal Action Detection Using a Statistical Language Model
Richard, A., & Gall, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.341 · Cited by 215
Self-citationTemporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202 · cited 4×
“approaches in [11], [12], [13] operate on downsampled videos with a temporal resolution of 1-3 frames per second.”
BackgroundMethodInfluentialWeakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · cited 1×
“Other approaches apply a coarse temporal modeling using Markov models on top of frame-wise classifiers [8], [9], [10].”
MethodSelf-citationFirst Person Action Recognition Using Deep Learned Descriptors
Singh, S., Arora, C., & Jawahar, C. V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.287 · Cited by 192 · cited 1×
“In [28], a three-stream architecture that operates on spatial, temporal and egocentric streams is introduced to learn egocentric-specific features.”
BackgroundWeakly Supervised Action Labeling in Videos under Ordering Constraints
Bojanowski, P., Lajugie, R., Bach, F., Laptev, I., Ponce, J., Schmid, C., & Sivic, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_41 · Cited by 191
An end-to-end generative framework for video segmentation and recognition
Kuehne, H., Gall, J., & Serre, T. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477701 · Cited by 188 · cited 2×
“[8] represent the frames of a video using Fisher vectors of improved dense trajectories, and then each action ismodeledwith a hiddenMarkov model (HMM).”
MethodSelf-citationModeling Actions through State Changes
Fathi, A., & Rehg, J. M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.333 · Cited by 145 · cited 1×
“Other approaches model actions based on the change in the state of objects and materials [18] or based on the interactions between hands and objects [19].”
BackgroundTemporal Convolutional Networks for Action Segmentation and DetectionPreprint
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05267 · Cited by 136
Weakly-supervised action segmentation with iterative soft boundary assignment
Ding, L., & Xu, C. (2018) · arXiv · doi:10.48550/arxiv.1803.10699 · Cited by 124 · cited 5×
“Ding and Xu [13] add lateral connections to the encoder-decoder TCN [11] and propose a temporal convolutional feature pyramid network for predicting frame-wise action labels.”
BackgroundMethodInfluentialWeakly supervised learning of actions from transcripts
Kuehne, H., Richard, A., & Gall, J. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.004 · Cited by 119 · cited 2×
“[22] combine HMMswith a Gaussianmixturemodel (GMM) as a frame-wise classifier.”
MethodSelf-citationWeakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
Xu, C., & Ding, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00681 · Cited by 104 · cited 5×
“Ding and Xu [13] add lateral connections to the encoder-decoder TCN [11] and propose a temporal convolutional feature pyramid network for predicting frame-wise action labels.”
BackgroundMethodInfluentialA Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Kuehne, H., Richard, A., & Gall, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884469 · Cited by 93
Self-citationTemporal Sequence Modeling for Video Event Detection
Cheng, Y., Fan, Q., Pankanti, S., & Choudhary, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.286 · Cited by 80 · cited 1×
“[21] represent videos as a sequence of visual words, and model the temporal dependency by employing a Bayesian non-parametric model of discrete sequences to jointly classify and segment video sequences.”
MethodFrom Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity
Vo, N. N., & Bobick, A. F. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.338 · Cited by 77 · cited 1×
“Vo and Bobick [25] use a Bayes network to segment activities.”
MethodBody Parts Dependent Joint Regressors for Human Pose Estimation in Still Images
Dantone, M., Gall, J., Leistner, C., & Van Gool, L. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2318702 · Cited by 73 · cited 1×
“Stacking several predictors sequentially has shown significant improvements in many tasks like human pose estimation [44], [45], [46].”
BackgroundSelf-citationRecognition of Complex Events: Exploiting Temporal Dynamics between Underlying Concepts
Bhattacharya, S., Kalayeh, M. M., Sukthankar, R., & Shah, M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.287 · Cited by 67 · cited 1×
“[20] use a vector time series representa-”
Mac, K.-N., Joshi, D., Yeh, R., Xiong, J., Feris, R., & Do, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00638 · Cited by 43 · cited 1×
“[31] propose to learn spatio-temporal features using deformable convolutions and local consistency constraints.”
BackgroundWeakly Supervised Action Learning with RNN based Fine-to-coarse ModelingPreprint
Richard, A., Kuehne, H., & Gall, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08132 · Cited by 15
Self-citationWeakly supervised learning of actions from transcriptsPreprint
Kuehne, H., Richard, A., & Gall, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02237 · Cited by 4
Self-citationSegmental Spatiotemporal CNNs for Fine-grained Action SegmentationPreprint
Lea, C., Reiter, A., Vidal, R., & Hager, G. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02995 · Cited by 4
Research works printed but not linked: 1
- karaman, 2014, Fast saliency based pooling of fisher encoded dense trajectories, Proc Eur Conf Comput Vis THUMOS Workshop
Cited by
Multi-Stage Progressive Image Restoration
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., & Shao, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01458 · Cited by 2,095
MAXIM: Multi-Axis MLP for Image Processing
Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00568 · Cited by 616
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities
Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., & Yao, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02042 · Cited by 213
Improving Action Segmentation via Graph-Based Temporal Reasoning
Huang, Y., Sugano, Y., & Sato, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01404 · Cited by 226
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
Liu, Y., Liu, Y., Jiang, C., Lyu, K., Wan, W., Shen, H., Liang, B., Fu, Z., Wang, H., & Yi, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02034 · Cited by 477
Self-citation
Metadata
- Authors
- Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J.
- Affiliations
- University of Bonn (DE)
- Nankai University (CN)
- University of Bonn, Bonn, Germany
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-06-01
- Volume
- 45
- Issue
- 6
- Pages
- 6647-6658
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3083550439
- Semantic Scholar
- 2db9d36748bf7058cbe72856021500794f711d37
- Cited by
- 349
- References
- 60
- Citations by year
- 2026: 45 · 2025: 69 · 2024: 74 · 2023: 49
- FWCI
- 15.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Deutsche Forschungsgemeinschaft (GA 1927/4-1)
- ERC Starting (677650)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- temporal action segmentation, long-range temporal dependencies, over-segmentation, untrimmed video, activity recognition, receptive fields
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.