Understanding egocentric activities
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
We present a method to analyze daily activities, such as meal preparation, using video from an egocentric camera. Our method performs inference about activities, actions, hands, and objects. Daily activities are a challenging domain for activity recognition which are well-suited to an egocentric approach. In contrast to previous activity recognition methods, our approach does not require pre-trained detectors for objects and hands. Instead we demonstrate the ability to learn a hierarchical model of an activity by exploiting the consistent appearance of objects, hands, and actions that results from the egocentric context. We show that joint modeling of activities, actions, and objects leads to superior performance in comparison to the case where they are considered independently. We introduce a novel representation of actions based on object-hand interactions and experimentally demonstrate the superior performance of our representation in comparison to standard activity representations such as bag of words.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Clustering by Passing Messages Between Data Points
Frey, B. J., & Dueck, D. (2007) · Science · doi:10.1126/science.1136800 · Cited by 6,758
Learning realistic human actions from movies
Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587756 · Cited by 3,444
Recognizing human actions: a local SVM approach
Schuldt, C., Laptev, I., & Caputo, B. (2004) · Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. · doi:10.1109/icpr.2004.1334462 · Cited by 3,010
Recognizing human actions: A local SVM approach
Schüldt, C., Laptev, I., & Barbara Caputo, B. (2004) · Cited by 2,906
The recognition of human movement using temporal templates
Bobick, A. F., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,811
Transductive Inference for Text Classification using Support Vector Machines
Joachims, T. (1999) · Cited by 2,723
Laptev, I. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-1838-7 · Cited by 2,629
Towards a general theory of action and time
Allen, J. F. (1984) · Artificial Intelligence · doi:10.1016/0004-3702(84)90008-0 · Cited by 2,559
Research works printed but not linked: 1
- spriggs, 2009, Temporal segmentation and activity classification from firstperson sensing, Egovision Workshop
Cited by
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
“In the literature, our task goes by the name action segmentation [7, 8, 6, 15, 29, 16, 10] or action detection [28, 20, 21, 25].”
MethodTemporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_7 · Cited by 924
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673
“In [6], actions are represented based on the interactions between hands and objects.”
BackgroundDiscovering important people and objects for egocentric video summarization
Lee, Y. J., Ghosh, J., & Grauman, K. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247820 · Cited by 658
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248010 · Cited by 649
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Garcia-Hernando, G., Yuan, S., Baek, S., & Kim, T.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00050 · Cited by 519
Story-Driven Summarization for Egocentric Video
Lu, Z., & Grauman, K. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.350 · Cited by 491
A Survey on Activity Detection and Classification Using Wearable Sensors
Cornacchia, M., Ozcan, K., Zheng, Y., & Velipasalar, S. (2017) · IEEE Sensors Journal · doi:10.1109/jsen.2016.2628346 · Cited by 424
Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions
Bambach, S., Lee, S., Crandall, D. J., & Yu, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.226 · Cited by 408
Learning to Recognize Daily Actions Using Gaze
Fathi, A., Li, Y., & Rehg, J. M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33718-5_23 · Cited by 385
Self-citationMS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349
“Other approaches model actions based on the change in the state of objects and materials [18] or based on the interactions between hands and objects [19].”
BackgroundYadav, S. K., Tiwari, K., Pandey, H. M., & Akbar, S. A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106970 · Cited by 311
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
Li, Y., Liu, M., & Rehg, J. M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_38 · Cited by 307
Self-citationSocial interactions: A first-person perspective
Fathi, A., Hodgins, J. K., & Rehg, J. M. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247805 · Cited by 288
Self-citationDelving into egocentric actions
Li, Y., Ye, Z., & Rehg, J. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298625 · Cited by 284
Self-citationFirst-Person Activity Recognition: What Are They Doing to Me?
Ryoo, M. S., & Matthies, L. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.352 · Cited by 279
Going Deeper into First-Person Activity Recognition
Ma, M., Fan, H., & Kitani, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.209 · Cited by 274
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
Tekin, B., Bogo, F., & Pollefeys, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00464 · Cited by 269
Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation
Lea, C., Reiter, A., Vidal, R., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_3 · Cited by 246
“Georgia Tech Egocentric Activities [41] has 28 videos across seven tasks.”
BackgroundTemporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202
First Person Action Recognition Using Deep Learned Descriptors
Singh, S., Arora, C., & Jawahar, C. V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.287 · Cited by 192
Furnari, A., & Farinella, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00635 · Cited by 182
“Specifically, we build on previous ideas investigated in the context of action recognition such as the use of multiple modalities for video analysis [49], the use of Temporal Segment Networks [61] as a principled way to train CNNs for action recognition, as well as the explicit encoding of object-based features [11, 3…”
BackgroundMethodModeling Actions through State Changes
Fathi, A., & Rehg, J. M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.333 · Cited by 145
Self-citation(2016) · MPG.PuRe (Max Planck Society) · Cited by 107
Video based activity recognition in trauma resuscitation
Chakraborty, I., Elgammal, A., & Burd, R. S. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553758 · Cited by 29
MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2020) · arXiv · Cited by 350
“Other approaches model actions based on the change in the state of objects and materials [18] or based on the interactions between hands and objects [19].”
BackgroundRecognising complex activities with histograms of relative tracklets
Stein, S., & McKenna, S. J. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.08.012 · Cited by 11
Metadata
- Authors
- Fathi, A., Farhadi, A., & Rehg, J. M.
- Affiliations
- Georgia Institute of Technology (US)
- University of Illinois Urbana-Champaign (US)
- Venue
- 2011 International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2011-11
- Pages
- 407-414
- OpenAlex
- W2149276562
- Cited by
- 384
- References
- 36
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 14 · 2023: 15 · 2022: 18 · 2021: 35 · 2020: 31 · 2019: 32 · 2018: 36 · 2017: 41 · 2016: 60 · 2015: 49 · 2014: 25 · 2013: 11 · 2012: 8
- FWCI
- 10.50
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- SMARTech Repository (Georgia Institute of Technology) (submittedVersion)
- Funders
- National Science Foundation
- Multidisciplinary University Research Initiative
- Army Research Office
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- egocentric video, activity recognition, daily activities, meal preparation, hand detection, object detection, action representation, bag-of-words
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.