The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Going Deeper into First-Person Activity Recognition
Open accessAbstract
We bring together ideas from recent work on feature design for egocentric action recognition under one framework by exploring the use of deep convolutional neural networks (CNN). Recent work has shown that features such as hand appearance, object attributes, local hand motion and camera ego-motion are important for characterizing first-person actions. To integrate these ideas under one framework, we propose a twin stream network architecture, where one stream analyzes appearance information and the other stream analyzes motion information. Our appearance stream encodes prior knowledge of the egocentric paradigm by explicitly training the network to segment hands and localize objects. By visualizing certain neuron activation of our network, we show that our proposed architecture naturally learns features that capture object attributes and hand-object configurations. Our extensive experiments on benchmark egocentric action datasets show that our deep architecture enables recognition rates that significantly outperform state-of-the-art techniques - an average 6:6% increase in accuracy over all datasets. Furthermore, by learning to recognize objects, actions and activities jointly, the performance of individual recognition tasks also increase by 30% (actions) and 14% (objects). We also include the results of extensive ablative analysis to highlight the importance of network design decisions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Learning realistic human actions from movies
Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587756 · Cited by 3,444
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.222 · Cited by 3,150 · cited 1×
“Due to the limited sizes of the three public datasets, we adopt the finetuning [23] approach to initialize our networks.”
MethodLaptev, I. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-1838-7 · Cited by 2,629
Improving the Fisher Kernel for Large-Scale Image Classification
Perronnin, F., Sánchez, J., & Mensink, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_11 · Cited by 2,588
Action recognition by dense trajectories
Wang, H., Klaser, A., Schmid, C., & Liu, C.-L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995407 · Cited by 2,154
Human activity analysis: A review
Aggarwal, J. K., & Ryoo, M. S. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922653 · Cited by 2,056
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · cited 1×
“In our experiments, we crop and resize input images to 256 × 256 and calculate optical flow using OpenCV GPU implementation of [37].”
MethodThe Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438
Action recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299059 · Cited by 1,194
Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice
Peng, X., Wang, L., Wang, X., & Qiao, Y. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.03.013 · Cited by 677
Discovering important people and objects for egocentric video summarization
Lee, Y. J., Ghosh, J., & Grauman, K. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247820 · Cited by 658
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248010 · Cited by 649
Learning to recognize objects in egocentric activities
Fathi, A., Ren, X., & Rehg, J. M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995444 · Cited by 557
Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition
Gupta, A., Kembhavi, A., & Davis, L. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.83 · Cited by 502
Flowing ConvNets for Human Pose Estimation in Videos
Pfister, T., Charles, J., & Zisserman, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.222 · Cited by 500
Action recognition by learning mid-level motion features
Fathi, A., & Mori, G. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587735 · Cited by 458
Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions
Bambach, S., Lee, S., Crandall, D. J., & Yu, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.226 · Cited by 408
Learning to Recognize Daily Actions Using Gaze
Fathi, A., Li, Y., & Rehg, J. M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33718-5_23 · Cited by 385
Understanding egocentric activities
Fathi, A., Farhadi, A., & Rehg, J. M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126269 · Cited by 384
Delving into egocentric actions
Li, Y., Ye, Z., & Rehg, J. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298625 · Cited by 284
Pixel-Level Hand Detection in Ego-centric Videos
Li, C., & Kitani, K. M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.458 · Cited by 238
Self-citationLearning to Predict Gaze in Egocentric Video
Li, Y., Fathi, A., & Rehg, J. M. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.399 · Cited by 230
Fast unsupervised ego-action learning for first-person sports videos
Kitani, K. M., Okabe, T., Sato, Y., & Sugimoto, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995406 · Cited by 220
Self-citationTemporal segmentation and activity classification from first-person sensing
Spriggs, E. H., De La Torre, F., & Hebert, M. (2009) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2009.5204354 · Cited by 219
Pooled motion features for first-person videos
Ryoo, M. S., Rothrock, B., & Matthies, L. H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298691 · Cited by 161
Modeling Actions through State Changes
Fathi, A., & Rehg, J. M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.333 · Cited by 145 · cited 1×
“We report comparative results on two subsets used in previous work [8, 20, 9]: 71 classes and 61 classes.”
MethodCompact CNN for indexing egocentric videos
Poleg, Y., Ephrat, A., Peleg, S., & Arora, C. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477708 · Cited by 105
Cited by
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Garcia-Hernando, G., Yuan, S., Baek, S., & Kim, T.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00050 · Cited by 519
Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior
Rasouli, A., Kotseruba, I., & Tsotsos, J. K. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.33 · Cited by 425
Recognizing Human Actions as the Evolution of Pose Estimation Maps
Liu, M., & Yuan, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00127 · Cited by 336
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
Li, Y., Liu, M., & Rehg, J. M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_38 · Cited by 307
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
Kazakos, E., Nagrani, A., Zisserman, A., & Damen, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00559 · Cited by 302
Metadata
- Authors
- Ma, M., Fan, H., & Kitani, K. M.
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 1894-1903
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2964222622
- Cited by
- 274
- References
- 37
- FWCI
- 15.74
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.