Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Joint segmentation and classification of fine-grained actions is important for applications of human-robot interaction, video surveillance, and human skill evaluation. However, despite substantial recent progress in large-scale action classification, the performance of state-of-the-art fine-grained action recognition approaches remains low. We propose a model for action segmentation which combines low-level spatiotemporal features with a high-level segmental classifier. Our spatiotemporal CNN is comprised of a spatial component that represents relationships between objects and a temporal component that uses large 1D convolutional filters to capture how object relationships change across time. These features are used in tandem with a semi-Markov model that captures transitions from one action to another. We introduce an efficient constrained segmental inference algorithm for this model that is orders of magnitude faster than the current approach. We highlight the effectiveness of our Segmental Spatiotemporal CNN on cooking and surgical action datasets for which we observe substantially improved performance relative to recent baseline methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 4×
“In this section we introduce a CNN topology, inspired by VGG [16], that by construction captures object state and location in fine-grained actions.”
BackgroundMethodResultInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“We find that pre-trained networks like VGG and AlexNet are inadequate for differentiating the small motions performed in our applications.”
ResultInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“Large-scale Action Recognition: While recent work has extended CNN models to video [2,22,24,25,26,27,28], often results are only superior when concatenated with IDT features [22,26,27].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · cited 1×
“Large-scale Action Recognition: While recent work has extended CNN models to video [2,22,24,25,26,27,28], often results are only superior when concatenated with IDT features [22,26,27].”
BackgroundLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2×
“Large-scale Action Recognition: While recent work has extended CNN models to video [2,22,24,25,26,27,28], often results are only superior when concatenated with IDT features [22,26,27].”
BackgroundShow and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“Recently, there has been significant interest in Recurrent Neural Networks (RNNs), specifically those using Long Short Term Memory (LSTM) (e.g. [28,33,34]).”
BackgroundInternational Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 1×
“Recently, there has been significant interest in Recurrent Neural Networks (RNNs), specifically those using Long Short Term Memory (LSTM) (e.g. [28,33,34]).”
BackgroundAction Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Advances in Neural Information Processing Systems 27 (NIPS 2014)
Zhong, M., Goddard, N., & Sutton, C. A. (2014) · Cited by 2,636
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578 · cited 2×
“sets for which we observe substantially improved performance relative to recent baseline methods. 1 Introduction New spatiotemporal feature representations [1,2] and massive datasets like ActivityNet [3] have catalyzed progress towards large-scale action recognition in recent years. In large-scale action recognition,…”
BackgroundResultInfluentialA guided tour to approximate string matching
Navarro, G. (2001) · ACM Computing Surveys · doi:10.1145/375360.375365 · Cited by 2,561 · cited 1×
“This score is computed using the Levenshtein distance which is a function of segment insertions, deletions, and substitutions [42].”
MethodAction recognition by dense trajectories
Wang, H., Klaser, A., Schmid, C., & Liu, C.-L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995407 · Cited by 2,154 · cited 1×
“t activity analysis research has approached action classification where simple bag of words-based approaches have been relatively successful. Typically spatiotemporal features, like Dense Trajectories [30,31], are used with a histogram of dictionary elements or a Fisher Vector encoding [31]. Dense Trajectories concaten…”
MethodLong-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 968
Learning human activities and object affordances from RGB-D videos
Koppula, H. S., Gupta, R., & Saxena, A. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913478446 · Cited by 683
P-CNN: Pose-Based CNN Features for Action Recognition
Cheron, G., Laptev, I., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.368 · Cited by 633 · cited 1×
“[29] show that if temporal segmentation on MPII is known then CNN-based pose features achieve 71.4% mAP.”
BackgroundInfluential(2004) · Cited by 610
Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks
Sun, L., Jia, K., Yeung, D.-Y., & Shi, B. E. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.522 · Cited by 577 · cited 4×
“This is much larger than those used other related work (e.g. [2,28]).”
BackgroundInfluentialCombining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · cited 3×
“University of Dundee 50 Salads: Stein and McKenna introduced 50 Salads [19] for evaluating fine-grained action recognition in the cooking domain.”
BackgroundMethod(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9 · Cited by 394
Understanding egocentric activities
Fathi, A., Farhadi, A., & Rehg, J. M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126269 · Cited by 384 · cited 1×
“Georgia Tech Egocentric Activities [41] has 28 videos across seven tasks.”
BackgroundExploring convolutional neural network structures and optimization techniques for speech recognition
Abdel-Hamid, O., Deng, L., & Yu, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-744 · Cited by 384 · cited 1×
“It is more similar to those in speech recognition (e.g. [35,36]) which learn phonemes using 1D convolutional filters or in robotics which learn sensor-based action filters [37].”
BackgroundLearning latent temporal structure for complex event detection
Tang, K., Fei-Fei, L., & Koller, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247808 · Cited by 355 · cited 1×
“Temporal Models: Several papers have used Conditional Random Fields for action segmentation and classification (e.g. [11,21,31,32]).”
MethodDelving into egocentric actions
Li, Y., Ye, Z., & Rehg, J. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298625 · Cited by 284 · cited 5×
“In contrast, despite active development on fine-grained action recognition (e.g. [4,5,6]) progress has been comparatively modest.”
BackgroundMethodInfluentialBeyond Short Snippets: Deep Networks for Video Classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.08909 · Cited by 244 · cited 4×
“This is much larger than those used other related work (e.g. [2,28]).”
BackgroundInfluentialWhat do 15,000 object categories tell us about classifying and localizing actions?
Jain, M., van Gemert, J. C., & Snoek, C. G. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298599 · Cited by 185 · cited 3×
“Large-scale Action Classification: Recent efforts to extend CNN models to video [2,24,26,27,28,29,30] improve over holistic methods by encoding spatial and temporal information.”
BackgroundMethodSurgical gesture classification from video and kinematic data
Zappella, L., Béjar, B., Hager, G. D., & Vidal, R. (2013) · Medical Image Analysis · doi:10.1016/j.media.2013.04.007 · Cited by 170 · cited 2×
“This is notably higher than the state of the art [10] which achieves 81.17% using a video-based linear dynamical system model and also better than their hybrid approach using video and kinematics which achieves 86.56%.”
BackgroundMethodInfluentialSelf-citationParsing Videos of Actions with Segmental Grammars
Pirsiavash, H., & Ramanan, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.85 · Cited by 161 · cited 3×
“Holistic Features: Holistic methods using spatio-temporal features with a bag of words representation are standard for large-scale [1,20,21,22] and fine-grained [4,5,6,21,23] action analysis.”
MethodModeling Actions through State Changes
Fathi, A., & Rehg, J. M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.333 · Cited by 145 · cited 3×
“Object-centric methods (e.g. [5,6,30]), first detect the identity and location of objects in an image.”
BackgroundMethodRecognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data
Rohrbach, M., Rohrbach, A., Regneri, M., Amin, S., Andriluka, M., Pinkal, M., & Schiele, B. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-015-0851-8 · Cited by 143 · cited 7×
“[4] combine IDT with pose features to get a detection score of 34.5% compared to 29.5% without pose features.”
BackgroundMethodInfluentialHuman Action Segmentation and Recognition Using Discriminative Semi-Markov Models
Shi, Q., Cheng, L., Wang, L., & Smola, A. J. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-010-0384-0 · Cited by 138
Fine-grained kitchen activity recognition using RGB-D
Lei, J., Ren, X., & Fox, D. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370248 · Cited by 132 · cited 1×
“There are many applications for this task including in industrial manufacturing [7,8], surgical training [9,10,11], and general human activity analysis (e.g. cooking, sports) [4,6,12,13,14,15].”
BackgroundActivity recognition using semi-Markov models on real world smart home datasets
van Kasteren, T. L. M., Englebienne, G., & Kröse, B. J. A. (2010) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-2010-0070 · Cited by 131
Multi-agent event recognition in structured scenarios
Morariu, V. I., & Davis, L. S. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995386 · Cited by 130 · cited 1×
“There are many applications for this task including in industrial manufacturing [7,8], surgical training [9,10,11], and general human activity analysis (e.g. cooking, sports) [4,6,12,13,14,15].”
BackgroundSurgical Gesture Segmentation and Recognition
Tao, L., Zappella, L., Hager, G. D., & Vidal, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40760-4_43 · Cited by 121 · cited 4×
“However, for joint segmentation and clas-sification the improvement of the state of the art [11] is modest.”
BackgroundMethodResultInfluentialSelf-citation(2016) · MPG.PuRe (Max Planck Society) · Cited by 107
Learning convolutional action primitives for fine-grained action recognition
Lea, C., Vidal, R., & Hager, G. D. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487305 · Cited by 106 · cited 4×
“Their approach uses holistic features with a Markov Semi-Markov CRF. Metrics: We evaluate on segmental and frame-wise metrics as suggested by [37] for the 50 Salads dataset.”
BackgroundMethodInfluentialSelf-citationFrom Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity
Vo, N. N., & Bobick, A. F. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.338 · Cited by 77 · cited 1×
“There are many applications for this task including in industrial manufacturing [7,8], surgical training [9,10,11], and general human activity analysis (e.g. cooking, sports) [4,6,12,13,14,15].”
BackgroundAnticipating human actions for collaboration in the presence of task and sensor uncertainty
Hawkins, K. P., Bansal, S., Vo, N. N., & Bobick, A. F. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907165 · Cited by 77 · cited 1×
“There are many applications for this task including in industrial manufacturing [7,8], surgical training [9,10,11], and general human activity analysis (e.g. cooking, sports) [4,6,12,13,14,15].”
BackgroundMultiple Granularity Analysis for Fine-Grained Action Detection
Ni, B., Paramathayalan, V. R., & Moulin, P. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.102 · Cited by 58 · cited 2×
“Object-centric methods (e.g. [5,6,30]), first detect the identity and location of objects in an image.”
BackgroundMethodInfluentialAction Recognition Using Temporal Templates
Bobick, A., & Davis, J. W. (1997) · Computational imaging and vision · doi:10.1007/978-94-015-8935-2_6 · Cited by 55
Fine-Grained Activity Recognition with Holistic and Pose Based Features
Pishchulin, L., Andriluka, M., & Schiele, B. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11752-2_56 · Cited by 43 · cited 1×
“Holistic Features: Holistic methods using spatio-temporal features with a bag of words representation are standard for large-scale [1,20,21,22] and fine-grained [4,5,6,21,23] action analysis.”
MethodEncoding Feature Maps of CNNs for Action Recognition
Peng, X., & Schmid, C. (2015) · HAL · Cited by 15
Research works printed but not linked: 3
- Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., Béjar, B., Yuh, D.D., et al.: JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: MICCAI Workshop: Modeling and Monitoring of Computer Assisted Interventions (M2CAI) (2014)
- Hannun, A.Y., Case, C., Casper, J., Catanzaro, B.C., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., Ng, A.Y.: Deep speech: scaling up end-to-end speech recognition. CoRR abs/1412.5567 (2014) · arXiv
- Ian Goodfellow, Y.B., Courville, A.: Deep Learning. MIT Press (2016, in preparation). http://www.deeplearningbook.org/
Cited by
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
Self-citationTemporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_7 · Cited by 924
“We trained the model of [8], except instead of using Motion History Images (MHI) as input to the CNN, we concatenate the following for image It at frame t: [It, It−d − It, It+d − It, It−2d − It, It+2d − It] for d = 0.”
BackgroundMethodInfluentialSelf-citationMS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673
“Earlier approaches for action segmentation can be grouped into two categories: sliding window approaches [22, 11, 19], that use temporal windows of different scales to detect action segments, and hybrid approaches that apply a coarse temporal modeling using Markov models on top of frame-wise classifiers [13, 16, 21].”
MethodGoing deeper into action recognition: A survey
Herath, S., Harandi, M., & Porikli, F. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.010 · Cited by 634
Shou, Z., Chan, J., Zareian, A., Miyazawa, K., & Chang, S.-F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.155 · Cited by 574
“Various methods have been developed, from the perspective of supervoxel merging [20, 54, 55], tracking [69, 42, 62, 52], object detection and linking [28, 14, 75, 42, 62], spatio-temporal segmentation [31, 70], and leveraging still images [21, 58, 22].”
Method
Metadata
- Authors
- Lea, C., Reiter, A., Vidal, R., & Hager, G. D.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Lea, Colin
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2016
- Pages
- 36-52
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2491875666
- Cited by
- 246
- References
- 44
- Citations by year
- 2026: 9 · 2025: 21 · 2024: 19 · 2023: 27 · 2022: 37 · 2021: 37 · 2020: 29 · 2019: 29 · 2018: 21 · 2017: 14 · 2016: 3
- FWCI
- 7.59
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- spatio-temporal convolutional neural network, semi-Markov model, constrained inference, surgical action recognition, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.