Modeling Actions through State Changes
Abstract
In this paper we present a model of action based on the change in the state of the environment. Many actions involve similar dynamics and hand-object relationships, but differ in their purpose and meaning. The key to differentiating these actions is the ability to identify how they change the state of objects and materials in the environment. We propose a weakly supervised method for learning the object and material states that are necessary for recognizing daily actions. Once these state detectors are learned, we can apply them to input videos and pool their outputs to detect actions. We further demonstrate that our method can be used to segment discrete actions from a continuous video of an activity. Our results outperform state-of-the-art action recognition and activity segmentation results.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,726 · see citing works
Learning realistic human actions from movies
Ivan Laptev, A., Marszałek, M., Schmid, C., & Rozenfeld, B. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587756 · Cited by 3,481 · see citing works
Recognizing human actions: a local SVM approach
Schüldt, C., Laptev, I., & Caputo, B. (2004) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2004.1334462 · Cited by 3,022 · see citing works
The recognition of human movement using temporal templates
Bobick, A., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,830 · see citing works
A survey of advances in vision-based human motion capture and analysis
Moeslund, T. B., Hilton, A., & Krüger, V. (2006) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.08.002 · Cited by 2,546 · see citing works
Machine Recognition of Human Activities: A Survey
Turaga, P. K., Chellappa, R., Subrahmanian, V. S., & Udrea, O. (2008) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2008.2005594 · Cited by 1,336 · see citing works
Recognizing action at a distance
Efros, A. A., Berg, Mori, & Malik (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238420 · Cited by 1,242 · see citing works
Marszałek, M., Laptev, I., & Schmid, C. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206557 · Cited by 1,197 · see citing works
A Statistical Approach to Texture Classification from Single Images
Varma, M., & Zisserman, A. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-4635-4 · Cited by 1,144 · see citing works
What, where and who? Classifying events by scene and object recognition
Li, L.-J., & Fei-Fei, L. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408872 · Cited by 790 · see citing works
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248010 · Cited by 666 · see citing works
Parametric hidden Markov models for gesture recognition
Wilson, A. D., & Bobick, A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.790429 · Cited by 574 · see citing works
Modeling mutual context of object and human pose in human-object interaction activities
Yao, B., & Fei-Fei, L. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540235 · Cited by 574 · see citing works
Learning to recognize objects in egocentric activities
Fathi, A., Ren, X., & Rehg, J. M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995444 · Cited by 557 · see citing works
Self-citationObserving Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition
Gupta, A. K., Kembhavi, A., & Davis, L. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.83 · Cited by 494 · see citing works
From contours to regions: An empirical evaluation
Arbeláez, P., Maire, M., Fowlkes, C. C., & Malik, J. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206707 · Cited by 447 · see citing works
Learning to Recognize Daily Actions Using Gaze
Fathi, A., Li, Y., & Rehg, J. M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_23 · Cited by 396 · see citing works
Self-citationUnderstanding egocentric activities
Fathi, A., Farhadi, A., & Rehg, J. M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126269 · Cited by 384 · see citing works
Self-citationA Scalable Approach to Activity Recognition based on Object Use
Wu, J., Osuntogun, A., Choudhury, T., Philipose, M., & Rehg, J. M. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408865 · Cited by 328 · see citing works
Self-citationLarge displacement optical flow
Brox, T., Bregler, C., & Malik, J. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206697 · Cited by 281 · see citing works
Joint segmentation and classification of human actions in video
Hoai, M., Lan, Z., & la Torre, F. D. (2011) · doi:10.1109/cvpr.2011.5995470 · Cited by 238 · see citing works
Temporal segmentation and activity classification from first-person sensing
Spriggs, E., la Torre, F. D., & Hebert, M. (2009) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2009.5204354 · Cited by 234 · see citing works
Tracklet Descriptors for Action Modeling and Video Analysis
Raptis, M., & Soatto, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15549-9_42 · Cited by 145 · see citing works
Object structure and action requirements: A compatibility model for functional recognition
Vaina, L. M., & Jaulent, M. (1991) · International Journal of Intelligent Systems · doi:10.1002/int.4550060306 · Cited by 48 · see citing works
Computational perception of scene dynamics
Mann, R., Jepson, A. S., & Siskind, J. M. (1996) · Lecture notes in computer science · doi:10.1007/3-540-61123-1_167 · Cited by 42 · see citing works
Taralova, E. H., la Torre, F. D., & Hebert, M. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126462 · Cited by 26 · see citing works
Sand, P., & Teller, S. J. (2004) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1186562.1015765 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · see citing works
“In the literature, our task goes by the name action segmentation [7, 8, 6, 15, 29, 16, 10] or action detection [28, 20, 21, 25].”
MethodMS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673 · see citing works
“While traditional approaches use a sliding window approach with non-maximum suppression [22, 11], Fathi and Rehg [7] model actions based on the change in the state of objects and materials.”
MethodEgo4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · see citing works
Self-citationA Survey on Activity Detection and Classification Using Wearable Sensors
Cornacchia, M., Ozcan, K., Zheng, Y., & Velipasalar, S. (2016) · IEEE Sensors Journal · doi:10.1109/jsen.2016.2628346 · Cited by 427 · see citing works
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349 · see citing works
“Other approaches model actions based on the change in the state of objects and materials [18] or based on the interactions between hands and objects [19].”
BackgroundDelving into egocentric actions
Li, Y., Ye, Z., & Rehg, J. M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298625 · Cited by 285 · see citing works
Self-citationGoing Deeper into First-Person Activity Recognition
Ma, M., Fan, H., & Kitani, K. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.209 · Cited by 281 · see citing works
Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation
Lea, C., Reiter, A., Vidal, R., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_3 · Cited by 246 · see citing works
Wang, X., Farhadi, A., & Gupta, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.291 · Cited by 228 · see citing works
Temporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202 · see citing works
An end-to-end generative framework for video segmentation and recognition
Kuehne, H., Gall, J., & Serre, T. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477701 · Cited by 188 · see citing works
Attributes as Operators: Factorizing Unseen Attribute-Object Compositions
Nagarajan, T., & Grauman, K. L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_11 · Cited by 178 · see citing works
The Evolution of First Person Vision Methods: A Survey
Betancourt, A., Morerio, P., Regazzoni, C. S., & Matthias Rauterberg, A. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2409731 · Cited by 176 · see citing works
Improving Action Segmentation via Graph-Based Temporal Reasoning
Huang, Y., Sugano, Y., & Sato, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01404 · Cited by 160 · see citing works
Asynchronous Temporal Fields for Action Recognition
Sigurdsson, G. A., Divvala, S., Farhadi, A., & Gupta, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.599 · Cited by 150 · see citing works
Boundary-Aware Cascade Networks for Temporal Action Segmentation
Wang, Z., Gao, Z., Wang, L., Li, Z., & Wu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_3 · Cited by 146 · see citing works
Visual Semantic Planning Using Deep Successor Representations
Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L., Gupta, A., Mottaghi, R., & Farhadi, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.60 · Cited by 129 · see citing works
Temporal Action Segmentation: An Analysis of Modern Techniques
Ding, G., Şener, F., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327284 · Cited by 124 · see citing works
Recognition of Activities of Daily Living with Egocentric Vision: A Review
Nguyen, T.-H.-C., Nebel, J., & Flórez‐Revuelta, F. (2016) · Sensors · doi:10.3390/s16010072 · Cited by 118 · see citing works
Gesture Recognition in Ego-centric Videos Using Dense Trajectories and Hand Segmentation
Baraldi, L., Paci, F., Serra, G., Luca Benini, C., & Cucchiara, R. (2014) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2014.107 · Cited by 94 · see citing works
Egocentric Future Localization
Park, H. S., Hwang, J.-J., Niu, Y., & Shi, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.508 · Cited by 94 · see citing works
Damen, D., Leelasawassuk, T., Haines, O., Calway, A. D., & Mayol-Cuevas, W. W. (2014) · Explore Bristol Research · Cited by 93 · see citing works
Disentangling Visual Embeddings for Attributes and Objects
Saini, N., Pham, K., & Shrivastava, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01329 · Cited by 86 · see citing works
Detecting Snap Points in Egocentric Video with a Web Photo Prior
Xiong, B., & Grauman, K. L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_19 · Cited by 82 · see citing works
Analysis of Hand Segmentation in the Wild
Khan, A. U., & Borji, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00495 · Cited by 79 · see citing works
Joint Discovery of Object States and Manipulation Actions
Alayrac, J.-B., Šivic, J., Laptev, I., & Lacoste-Julien, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.234 · Cited by 77 · see citing works
From Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity
Vo, N., & Bobick, A. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.338 · Cited by 76 · see citing works
CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications
Wang, T., Qian, X., He, F., Hu, X., Huo, K., Cao, Y., & Ramani, K. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415815 · Cited by 74 · see citing works
Global2Local: Efficient Structure Search for Video Action Segmentation
Gao, S., Han, Q., Li, Z., Peng, P., Wang, L., & Cheng, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01653 · Cited by 74 · see citing works
Temporal Alignment Networks for Long-term Video
Han, T., Xie, W., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00292 · Cited by 72 · see citing works
Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals
Nakamura, K., Yeung, S. Y., Alahi, A., & Fei-Fei, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.721 · Cited by 68 · see citing works
Temporal Convolutional Networks for Action Segmentation and DetectionPreprint
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05267 · Cited by 67 · see citing works
Egocentric Vision-based Action Recognition: A survey
Núñez-Marcos, A., Azkune, G., & Arganda‐Carreras, I. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.11.081 · Cited by 61 · see citing works
Deep Attention Network for Egocentric Action Recognition
Lu, M., Li, Z.-N., Wang, Y., & Pan, G. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2901707 · Cited by 60 · see citing works
Generating Notifications for Missing Actions: Don't Forget to Turn the Lights Off!
Soran, B., Farhadi, A., & Shapiro, L. G. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.530 · Cited by 59 · see citing works
Explainable Video Action Reasoning via Prior Knowledge and State Transitions
Zhuo, T., Cheng, Z., Zhang, P., Wong, Y., & Kankanhalli, M. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351040 · Cited by 59 · see citing works
RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks
Gao, S., Li, Z., Qi, H., Cheng, M., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183829 · Cited by 57 · see citing works
Damen, D., Leelasawassuk, T., & Mayol-Cuevas, W. W. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.02.016 · Cited by 56 · see citing works
Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home
Likitlersuang, J., Sumitro, E. R., Cao, T., Visée, R. J., Kalsi‐Ryan, S., & Zariffa, J. (2019) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-019-0557-1 · Cited by 56 · see citing works
Temporal Relational Modeling with Self-Supervision for Action Segmentation
Wang, D., Hu, D., Li, X., & Dou, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16377 · Cited by 54 · see citing works
Gesture Recognition using Wearable Vision Sensors to Enhance Visitors' Museum Experiences
Baraldi, L., Paci, F., Serra, G., Luca Benini, C., & Cucchiara, R. (2015) · IEEE Sensors Journal · doi:10.1109/jsen.2015.2411994 · Cited by 53 · see citing works
Learning Attention as Disentangler for Compositional Zero-Shot Learning
Hao, S., Han, K., & Wong, K.-Y. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01470 · Cited by 52 · see citing works
Towards Structured Analysis of Broadcast Badminton Videos
Ghosh, A., Singh, S., & Jawahar, C. V. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00039 · Cited by 51 · see citing works
A Decomposable Causal View of Compositional Zero-Shot Learning
Yang, M., Xu, C., Wu, A., & Deng, C. D. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3200578 · Cited by 46 · see citing works
KrishnaCam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks
Singh, K. K., Fatahalian, K., & Efros, A. A. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477717 · Cited by 46 · see citing works
Chand, S., Zheng, H., & Lu, Y. (2024) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2024.10.002 · Cited by 41 · see citing works
Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions
Sudhakaran, S., & Lanz, O. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.276 · Cited by 40 · see citing works
Physical Causality of Action Verbs in Grounded Language Understanding
Gao, Q., Doering, M., Yang, S., & Chai, J. Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1171 · Cited by 39 · see citing works
Long Activity Video Understanding Using Functional Object-Oriented Network
Jelodar, A. B., Paulius, D., & Sun, Y. M. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2885228 · Cited by 38 · see citing works
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented LearningPreprint
Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., & Zhu, S. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05757 · Cited by 37 · see citing works
Action Recognition in the Presence of One Egocentric and Multiple Static Cameras
Soran, B., Farhadi, A., & Shapiro, L. G. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16814-2_12 · Cited by 35 · see citing works
Parsa, B., Samani, E. U., Hendrix, R., Devine, C., Singh, S. M., Devasia, S., & Banerjee, A. G. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2925305 · Cited by 35 · see citing works
Domain-Specific Priors and Meta Learning for Few-Shot First-Person Action Recognition
Coskun, H., Zia, M. Z., Tekin, B., Bogo, F., Navab, N., Tombari, F., & Sawhney, H. S. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3058606 · Cited by 35 · see citing works
Jointly Recognizing Object Fluents and Tasks in Egocentric Videos
Liu, Y., Wei, P., & Zhu, S. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.318 · Cited by 33 · see citing works
Sensor substitution for video-based action recognition
Rupprecht, C., Lea, C., Tombari, F., Navab, N., & Hager, G. D. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759769 · Cited by 26 · see citing works
Local Temporal Bilinear Pooling for Fine-Grained Action Parsing
Zhang, Y., Tang, S., Muandet, K., Jarvers, C., & Heiko Neumann, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01228 · Cited by 26 · see citing works
Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos
Souček, T., Alayrac, J.-B., Miech, A., Laptev, I., & Šivic, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01357 · Cited by 25 · see citing works
Fathaliyan, A. H., Wang, X., & Santos, V. J. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00025 · Cited by 22 · see citing works
What Action Causes This? Towards Naive Physical Action-Effect Prediction
Gao, Q., Yang, S., Chai, J. Y., & Vanderwende, L. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1086 · Cited by 20 · see citing works
Xing, H., & Burschka, D. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981771 · Cited by 19 · see citing works
Likitlersuang, J., & Zariffa, J. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636748 · Cited by 19 · see citing works
Market basket analysis from egocentric videos
Santarcangelo, V., Farinella, G. M., Furnari, A., & Battiato, S. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.06.010 · Cited by 19 · see citing works
Transformation Driven Visual Reasoning
Hong, X., Lan, Y., Pang, L., Guo, J., & Cheng, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00683 · Cited by 19 · see citing works
A Survey on Recent Advances of Computer Vision Algorithms for Egocentric VideoPreprint
Bambach, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.02825 · Cited by 19 · see citing works
Egocentric Action Recognition by Capturing Hand-Object Contact and Object State
Shiota, T., Takagi, M., Kumagai, K., Seshimo, H., & Aono, Y. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00641 · Cited by 18 · see citing works
Gated forward refinement network for action segmentation
Wang, D., Yuan, Y., & Wang, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.066 · Cited by 18 · see citing works
Multi-Task Learning of Object States and State-Modifying Actions From Web Videos
Souček, T., Alayrac, J.-B., Miech, A., Laptev, I., & Šivic, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3362288 · Cited by 17 · see citing works
Learning Object State Changes in Videos: An Open-World Perspective
Xue, Z., Kumar, A., & Grauman, K. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01750 · Cited by 17 · see citing works
SEMBED: Semantic Embedding of Egocentric Action Videos
Wray, M., Moltisanti, D., Mayol-Cuevas, W. W., & Damen, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46604-0_38 · Cited by 17 · see citing works
Activities of Daily Living Monitoring via a Wearable Camera: Toward Real-World Applications
Cartas, A., Radeva, P. I., & Dimiccoli, M. (2020) · IEEE Access · doi:10.1109/access.2020.2990333 · Cited by 17 · see citing works
Ego4D: Around the World in 3,600 Hours of Egocentric Video
Grauman, K. L., Westbury, A., Byrne, E., Cartillier, V., Chavis, Z. Q., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Kukreja, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3381075 · Cited by 17 · see citing works
Self-citationOn-the-fly hand detection training with application in egocentric action recognition
Kumar, J., Li, Q., Kyal, S., Bernal, E. A., & Bala, R. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301344 · Cited by 16 · see citing works
Wang, X., Fathaliyan, A. H., & Santos, V. J. (2020) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2020.567571 · Cited by 17 · see citing works
Temporal Action Detection with Multi-level Supervision
Shi, B., Dai, Q., Hoffman, J., Saenko, K., Darrell, T., & Xu, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00792 · Cited by 16 · see citing works
Multi-User Egocentric Online System for Unsupervised Assistance on Object Usage
Damen, D., Haines, O., Leelasawassuk, T., Calway, A. D., & Mayol-Cuevas, W. W. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16199-0_34 · Cited by 16 · see citing works
A KinFu based approach for robot spatial attention and view planning
Monica, R., Aleotti, J., & Caselli, S. (2015) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2015.09.010 · Cited by 16 · see citing works
Nagarajan, T., & Grauman, K. L. (2018) · Cited by 16 · see citing works
About Time: Advances, Challenges, and Outlooks of Action Understanding
Stergiou, A., & Poppe, R. W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02478-4 · Cited by 15 · see citing works
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2020) · arXiv · Cited by 350 · see citing works
“Other approaches model actions based on the change in the state of objects and materials [18] or based on the interactions between hands and objects [19].”
BackgroundChop & Learn: Recognizing and Generating Object-State Compositions
Saini, N., Wang, H., Swaminathan, A., Jayasundara, V., He, B., Gupta, K., & Shrivastava, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01852 · Cited by 14 · see citing works
Egocentric articulated pose tracking for action recognition
Yonemoto, H., Murasaki, K., Osawa, T., Sudo, K., Shimamura, J., & Taniguchi, Y. (2015) · Machine Vision Applications · doi:10.1109/mva.2015.7153142 · Cited by 14 · see citing works
Dilated Transformer with Feature Aggregation Module for Action Segmentation
Du, Z., & Wang, Q. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-11133-9 · Cited by 13 · see citing works
Head Pose Estimation in First-Person Camera Views
Alletto, S., Serra, G., Calderara, S., & Cucchiara, R. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.718 · Cited by 13 · see citing works
A smart cooking device for assisting cognitively impaired users
Bouchard, B., Bouchard, K., & Bouzouane, A. (2020) · Journal of Reliable Intelligent Environments · doi:10.1007/s40860-020-00104-3 · Cited by 13 · see citing works
Likitlersuang, J., Visée, R. J., Kalsi‐Ryan, S., & Zariffa, J. (2021) · Spinal Cord Series and Cases · doi:10.1038/s41394-021-00382-w · Cited by 13 · see citing works
COBE: Contextualized Object Embeddings from Narrated Instructional VideoPreprint
Bertasius, G., & Torresani, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07306 · Cited by 13 · see citing works
One-shot Video Graph Generation for Explainable Action Reasoning
Han, Y., Zhuo, T., Zhang, P., Huang, W., Zha, Y., Zhang, Y., & Kankanhalli, M. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.069 · Cited by 12 · see citing works
First-Person Action Decomposition and Zero-Shot Learning
Zhang, Y. C., Li, Y., & Rehg, J. M. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.21 · Cited by 12 · see citing works
Self-citationYang, J., Qiu, L., Corona, E. A., Shi, L., Bui, H., Lam, M. S., & Landay, J. A. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00108 · Cited by 11 · see citing works
Recognition from Hand Cameras: A Revisit with Deep Learning
Chan, C.-S., Chen, S.-Z., Xie, P.-X., Chang, C., & Sun, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_31 · Cited by 11 · see citing works
Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
Nakamura, K., Ohashi, H., & Okada, M. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475557 · Cited by 10 · see citing works
Zhang, Y. C., & Rehg, J. M. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3214291 · Cited by 10 · see citing works
Self-citationAsynchronous Temporal Fields for Action RecognitionPreprint
Sigurdsson, G. A., Divvala, S., Farhadi, A., & Gupta, A. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06371 · Cited by 10 · see citing works
Visual Reasoning: From State to Transformation
Hong, X., Lan, Y., Pang, L., Guo, J., & Cheng, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3268093 · Cited by 8 · see citing works
Wang, X., & Santos, V. J. (2023) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2023.3328888 · Cited by 9 · see citing works
LogiCzsl: Exploring Logic-induced Representation for Compositional Zero-shot Learning
Wu, P., Lu, X., Hu, H. P., Xian, Y., Shen, J., & Wang, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02821 · Cited by 8 · see citing works
Liquid Pouring Monitoring via Rich Sensory Inputs
Wu, T.-Y., Lin, J.-T., Wang, T.-H., Hu, C.-W., Niebles, J. C., & Sun, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_21 · Cited by 8 · see citing works
Video-Based Fatigue Estimation for Human-Robot Task Allocation Optimisation
Zheng, H. R., Chand, S., Keshvarparast, A., Battini, D., & Lu, Y. (2023) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case56687.2023.10260460 · Cited by 8 · see citing works
EgoTV: Egocentric Task Verification from Natural Language Task Descriptions
Hazra, R., Chen, B., Rai, A., Kamra, N., & Desai, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01414 · Cited by 8 · see citing works
Temporally coherent interpretations for long videos using pattern theory
de Souza, F. D. M., Sarkar, S., Srivastava, A., & Su, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298727 · Cited by 8 · see citing works
- Authors
- Fathi, A., & Rehg, J. M.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- 2013 IEEE Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2013-06
- Pages
- 2579-2586
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1967686239
- Cited by
- 145
- References
- 29
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 11 · 2023: 9 · 2022: 10 · 2021: 12 · 2020: 11 · 2019: 12 · 2018: 11 · 2017: 12 · 2016: 23 · 2015: 18 · 2014: 5
- FWCI
- 9.78
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- SMARTech Repository (Georgia Institute of Technology) (submittedVersion)
- Funders
- National Science Foundation
- Multidisciplinary University Research Initiative
- Army Research Office
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- human action recognition, weakly supervised learning, activity segmentation, hand-object interaction, video action detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
