A Perceptual Prediction Framework for Self Supervised Event Segmentation
Open accessAbstract
Temporal segmentation of long videos is an important problem, that has largely been tackled through supervised learning, often requiring large amounts of annotated training data. In this paper, we tackle the problem of self-supervised temporal segmentation that alleviates the need for any supervision in the form of labels (full supervision) or temporal ordering (weak supervision). We introduce a self-supervised, predictive learning framework that draws inspiration from cognitive psychology to segment long, visually complex videos into constituent events. Learning involves only a single pass through the training data. We also introduce a new adaptive learning paradigm that helps reduce the effect of catastrophic forgetting in recurrent neural networks. Extensive experiments on three publicly available datasets - Breakfast Actions, 50 Salads, and INRIA Instructional Videos datasets show the efficacy of the proposed approach. We show that the proposed approach outperforms weakly-supervised and unsupervised baselines by up to 24% and achieves competitive segmentation results compared to fully supervised baselines with only a single pass through the training data. Finally, we show that the proposed self-supervised learning paradigm learns highly discriminating features to improve action recognition.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In auto-regressive CPD methods, change points are detected at samples associated with a salient increase in the prediction error.
“In these approaches, change points are detected at samples associated with a salient increase in the prediction error.”
From Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding (2021) · cites this paper as [1]
CPD methods for video and time series use an auto-regressive model for prediction.
“There are change point and temporal anomaly detection methods for video [1] and time series [26] that use an auto-regressive model for prediction.”
From Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding (2021) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 1× · see citing works
“oded in terms of coherent, stable structures called “events”. Computer vision research has largely focused on the problem of recognizing and describing these events in terms of either labeled actions [18,17,18,2,1] or in terms of sentences (captioning)[1,33,32,12,5,36]. Such approaches assume that the video is already…”
BackgroundSpeed of processing in the human visual system
Thorpe, S., Fize, D., & Marlot, C. (1996) · Nature · doi:10.1038/381520a0 · Cited by 3,538 · see citing works
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407 · see citing works
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · cited 4× · see citing works
“G and IDT [20] to show the impact of temporal modeling. It should be noted Supervision Approach MoF Full VGG**[20] 7.6% IDT**[20] 54.3% S-CNN + LSTM[20] 66.6% TDRN[21] 68.1% ST-CNN + Seg[20] 72.0% TCN[25] 73.4% None LSTM + KNN[4] 54.0% Ours (LSTM + AL) 60.6% Table 2. Segmentation Results on the 50 Salads dataset, at g…”
BackgroundMethodInfluentialThe Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · see citing works
Combining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · see citing works
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_9 · Cited by 226 · see citing works
Temporal Action Detection Using a Statistical Language Model
Richard, A., & Gall, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.341 · Cited by 215 · cited 3× · see citing works
“Full SVM [19] 15.8 - HTK(64)[20] 56.3 - ED-TCN[27] 43.3 42.0 TCFPN[10] 52.0 54.9 GRU[29] 60.6 - Weak OCDC[6] 8.9 23.4 ECTC[16] 27.7 - Fine2Coarse[28] 33.3 47. proach [30], requires the number of clusters (from ground truth) to achieve the performance whereas our approach does not require such knowledge and is done in…”
MethodInfluentialWeakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · see citing works
Flagged first, then by citations
Research works printed but not linked: 30
- Sathyanarayanan Aakur, Fillipe DM de Souza, and Sudeep Sarkar. Going deeper with semantics: Exploiting seman- tic contextualization for interpretation of human activity in videos. In IEEE Winter Conference on Applications of Com- puter Vision (WACV). IEEE, 2019. 1, 6, 8
- Sathyanarayanan N. Aakur, Fillipe DM de Souza, and Sudeep Sarkar. Towards a knowledge-based approach for generating video descriptions. In Conference on Computer and Robot Vision (CRV). Springer, 2017. 1
- Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsu- pervised learning from narrated instruction videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4575-4583, 2016. 2, 5, 6, 7, 8
- Bharat Lal Bhatnagar, Suriya Singh, Chetan Arora, CV Jawahar, and KCIS CVIT. Unsupervised learning of deep feature representation for clustering egocentric actions. In International Joint Conference on Artificial Intelligence (IJ- CAI), pages 1447-1453. AAAI Press, 2017. 2, 7
- Yi Bin, Yang Yang, Fumin Shen, Xing Xu, and Heng Tao Shen. Bidirectional long-short term memory for video de- scription. In ACM Conference on Multimedia (ACM MM), pages 436-440. ACM, 2016. 1
- Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly su- pervised action labeling in videos under ordering constraints. In European Conference on Computer Vision (ECCV), pages 628-643. Springer, 2014. 2, 7, 8
- Gail A Carpenter and Stephen Grossberg. Adaptive reso- nance theory. Springer, 2016. 2
- Rizwan Chaudhry, Avinash Ravichandran, Gregory Hager, and René Vidal. Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1932- 1939. IEEE, 2009. 3
- Fillipe DM de Souza, Sudeep Sarkar, Anuj Srivastava, and Jingyong Su. Spatially coherent interpretations of videos us- ing pattern theory. International Journal on Computer Vision (IJCV), pages 1-21, 2016. 6, 8
- Li Ding and Chenliang Xu. Weakly-supervised action seg- mentation with iterative soft boundary assignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2, 7
- Joaquin M Fuster. The prefrontal cortex and its relation to behavior. In Progress in brain research, volume 87, pages 201-211. Elsevier, 1991. 2
- Ana Garcia del Molino, Joo-Hwee Lim, and Ah-Hwee Tan. Predicting visual context for unsupervised event segmenta- tion in continuous photo-streams. In ACM Conference on Multimedia (ACM MM), pages 10-17. ACM, 2018. 2
- Zhao Guo, Lianli Gao, Jingkuan Song, Xing Xu, Jie Shao, and Heng Tao Shen. Attention-based lstm with semantic consistency for videos captioning. In ACM Conference on Multimedia (ACM MM), pages 357-361. ACM, 2016. 1
- Catherine Hanson and Stephen José Hanson. Development of schemata during event parsing: Neisser's perceptual cycle as a recurrent connectionist network. Journal of Cognitive Neuroscience, 8(2):119-134, 1996. 2
- Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997. 3, 5
- Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In Neural Information Pro- cessing Systems, pages 667-675, 2016. 4
- Hilde Kuehne, Juergen Gall, and Thomas Serre. An end-to- end generative framework for video segmentation and recog- nition. In IEEE Winter Conference on Applications of Com- puter Vision (WACV), pages 1-8. IEEE, 2016. 7, 8
- Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager. Segmental spatiotemporal cnns for fine-grained action seg- mentation. In European Conference on Computer Vision (ECCV), pages 36-52. Springer, 2016. 2, 6, 7
- Peng Lei and Sinisa Todorovic. Temporal deformable resid- ual networks for action segmentation in videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6742-6751, 2018. 7
- Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008. 2
- Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy. What's cookin'? interpreting cooking videos using text, speech and vision. arXiv preprint arXiv:1503.01558, 2015. 2, 7, 8 · arXiv
- Fadime Sener and Angela Yao. Unsupervised learning and segmentation of complex activities from video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2, 6, 7, 8
- Teun Adrianus Van Dijk, Walter Kintsch, and Teun Adrianus Van Dijk. Strategies of discourse comprehension. Academic Press New York, 1933. 2
- Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Don- ahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In IEEE International Conference on Computer Vision (ICCV), pages 4534-4542, 2015. 1
- Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Mar- cus Rohrbach, Raymond Mooney, and Kate Saenko. Trans- lating videos to natural language using deep recurrent neural networks. arXiv preprint arXiv:1412.4729, 2014. 1 · arXiv
- Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. An- ticipating visual representations from unlabeled video. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 98-106, 2016. 4
- Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. De- scribing videos by exploiting temporal structure. In IEEE International Conference on Computer Vision (ICCV), pages 4507-4515, 2015. 1
- Jeffrey M Zacks and Khena M Swallow. Event segmentation. Current Directions in Psychological Science, 16(2):80-84, 2007. 2
- Jeffrey M Zacks and Barbara Tversky. Event structure in perception and conception. Psychological bulletin, 127(1):3, 2001. 2
- Jeffrey M Zacks, Barbara Tversky, and Gowri Iyer. Perceiv- ing, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130(1):29, 2001. 1, 2
Cited by
Sorted by citations
Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding
Deldari, S., Smith, D. V., Xue, H., & Salim, F. D. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449903 · Cited by 136 · see citing works
- Authors
- Aakur, S. N., & Sarkar, S.
- Affiliations
- University of South Florida (US)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 1197-1206
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2900157177
- Cited by
- 79
- References
- 10
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 14 · 2023: 15 · 2022: 16 · 2021: 14 · 2020: 7
- FWCI
- 3.02
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- event segmentation, temporal video segmentation, self-supervised learning, weakly supervised learning, catastrophic forgetting, recurrent neural networks, perceptual prediction, human action recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
