The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
Open accessAbstract
We present a unified framework for understanding 3D hand and object interactions in raw image sequences from egocentric RGB cameras. Given a single RGB image, our model jointly estimates the 3D hand and object poses, models their interactions, and recognizes the object and action classes with a single feed-forward pass through a neural network. We propose a single architecture that does not rely on external detection algorithms but rather is trained end-to-end on single images. We further merge and propagate information in the temporal domain to infer interactions between hand and object trajectories and recognize actions. The complete model takes as input a sequence of frames and outputs per-frame 3D hand and object pose predictions along with the estimates of object and action categories for the entire sequence. We demonstrate state-of-the-art performance of our algorithm even in comparison to the approaches that work on depth data and ground-truth annotations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
EPnP: An Accurate O(n) Solution to the PnP Problem
Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0152-6 · Cited by 3,298
Generalized procrustes analysis
Gower, J. C. (1975) · Psychometrika · doi:10.1007/bf02291478 · Cited by 3,267
The recognition of human movement using temporal templates
Bobick, A. F., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,811
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.213 · Cited by 2,684
Behavior Recognition via Sparse Spatio-Temporal Features
Dollar, P., Rabaud, V., Cottrell, G., & Belongie, S. (2006) · 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance · doi:10.1109/vspets.2005.1570899 · Cited by 2,460
Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words
Niebles, J. C., Wang, H., & Fei-Fei, L. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0122-4 · Cited by 1,475
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
Laptev, & Lindeberg (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238378 · Cited by 1,300
LIFT: Learned Invariant Feature Transform
Yi, K. M., Trulls, E., Lepetit, V., & Fua, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_28 · Cited by 1,202
Romero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again
Kehl, W., Manhardt, F., Tombari, F., Ilic, S., & Navab, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.169 · Cited by 1,047
HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences
Oreifej, O., & Liu, Z. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.98 · Cited by 956
Real-Time Seamless Single Shot 6D Object Pose Prediction
Tekin, B., Sinha, S. N., & Fua, P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00038 · Cited by 900
Self-citationRad, M., & Lepetit, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.413 · Cited by 843
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 761
A Biologically Inspired System for Action Recognition
Jhuang, H., Serre, T., Wolf, L., & Poggio, T. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408988 · Cited by 723
Learning human activities and object affordances from RGB-D videos
Koppula, H. S., Gupta, R., & Saxena, A. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913478446 · Cited by 683
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248010 · Cited by 649
Detecting and Recognizing Human-Object Interactions
Gkioxari, G., Girshick, R., Dollar, P., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00872 · Cited by 602
Learning to recognize objects in egocentric activities
Fathi, A., Ren, X., & Rehg, J. M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995444 · Cited by 557
Uncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image
Brachmann, E., Michel, F., Krull, A., Yang, M. Y., Gumhold, S., & Rother, C. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.366 · Cited by 530
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00013 · Cited by 521
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Garcia-Hernando, G., Yuan, S., Baek, S., & Kim, T.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00050 · Cited by 519
Learning Joint Reconstruction of Hands and Manipulated Objects
Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01208 · Cited by 513
Scaling Egocentric Vision: The "Equation missing" Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_44 · Cited by 504
Learning to Recognize Daily Actions Using Gaze
Fathi, A., Li, Y., & Rehg, J. M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33718-5_23 · Cited by 385
Understanding egocentric activities
Fathi, A., Farhadi, A., & Rehg, J. M. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126269 · Cited by 384
Jointly Learning Heterogeneous Features for RGB-D Activity Recognition
Hu, J.-F., Zheng, W.-S., Lai, J., & Zhang, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2640292 · Cited by 383
Jointly learning heterogeneous features for RGB-D activity recognition
Hu, J.-F., Zheng, W.-S., Lai, J., & Zhang, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299172 · Cited by 358
PoTion: Pose MoTion Representation for Action Recognition
Choutas, V., Weinzaepfel, P., Revaud, J., & Schmid, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00734 · Cited by 332
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P., Breuel, T., Gall, J., & Kautz, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316
Delving into egocentric actions
Li, Y., Ye, Z., & Rehg, J. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298625 · Cited by 284
Cross-Modal Deep Variational Hand Pose Estimation
Spurr, A., Song, J., Park, S., & Hilliges, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00017 · Cited by 281
Latent-Class Hough Forests for 3D Object Detection and Pose Estimation
Tejani, A., Tang, D., Kouskouridas, R., & Kim, T.-K. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_30 · Cited by 280
First-Person Activity Recognition: What Are They Doing to Me?
Ryoo, M. S., & Matthies, L. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.352 · Cited by 279
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0895-4 · Cited by 277
Self-citationScaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.02748 · Cited by 274
Oikonomidis, I., Kyriazis, N., & Argyros, A. A. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126483 · Cited by 274
Going Deeper into First-Person Activity Recognition
Ma, M., Fan, H., & Kitani, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.209 · Cited by 274
Training a Feedback Loop for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.379 · Cited by 272
BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis
Yuan, S., Ye, Q., Stenger, B., Jain, S., & Kim, T.-K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.279 · Cited by 243
Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.131 · Cited by 242
Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation
Tekin, B., Marquez-Neila, P., Salzmann, M., & Fua, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.425 · Cited by 242
Self-citationReal-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
Sridhar, S., Mueller, F., Zollhöfer, M., Casas, D., Oulasvirta, A., & Theobalt, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_19 · Cited by 240
Fast and robust hand tracking using detection-guided optimization
Sridhar, S., Mueller, F., Oulasvirta, A., & Theobalt, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298941 · Cited by 228
Fast unsupervised ego-action learning for first-person sports videos
Kitani, K. M., Okabe, T., Sato, Y., & Sugimoto, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995406 · Cited by 220
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals
Yuan, S., Garcia-Hernando, G., Stenger, B., Moon, G., Chang, J. Y., Lee, K. M., Molchanov, P., Kautz, J., Honari, S., Ge, L., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00279 · Cited by 217
Gradient descent optimization of smoothed information retrieval metrics
Chapelle, O., & Wu, M. (2010) · Information Retrieval · doi:10.1007/s10791-009-9110-3 · Cited by 198
Learning Motion Categories using both Semantic and Structural Information
Wong, S.-F., Kim, T.-K., & Cipolla, R. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383332 · Cited by 194
3D Action Recognition from Novel Viewpoints
Rahmani, H., & Mian, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.167 · Cited by 187
Figure-ground segmentation improves handled object recognition in egocentric video
Ren, X., & Gu, C. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540074 · Cited by 163
Object Level Visual Reasoning in Videos
Baradel, F., Neverova, N., Wolf, C., Mille, J., & Mori, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_7 · Cited by 163
Pooled motion features for first-person videos
Ryoo, M. S., Rothrock, B., & Matthies, L. H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298691 · Cited by 161
Actor and Observer: Joint Modeling of First and Third-Person Videos
Sigurdsson, G. A., Gupta, A., Schmid, C., Farhadi, A., & Alahari, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00772 · Cited by 159
Tracking a hand manipulating an object
Hamer, H., Schindler, K., Koller-Meier, E., & Gool, L. V. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459282 · Cited by 149
Understanding Everyday Hands in Action from RGB-D Images
Rogez, G., Supančič, J. S., & Ramanan, D. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.443 · Cited by 147
Hands in action: real-time 3D reconstruction of hands in interaction with objects
Romero, J., Kjellström, H., & Kragić, D. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509753 · Cited by 124
From Lifestyle Vlogs to Everyday Interactions
Fouhey, D. F., Kuo, W., Efros, A. A., & Malik, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00524 · Cited by 117
Compact CNN for indexing egocentric videos
Poleg, Y., Ephrat, A., Peleg, S., & Arora, C. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477708 · Cited by 105
First-person pose recognition using egocentric workspaces
Rogez, G., Supančič, J. S., & Ramanan, D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299061 · Cited by 94
Efficiently Creating 3D Training Data for Fine Hand Pose Estimation
Oberweger, M., Riegler, G., Wohlhart, P., & Lepetit, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.536 · Cited by 88
Rad, M., & Lepetit, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10896 · Cited by 82
Understanding Hand-Object Manipulation with Grasp Types and Object Attributes
Cai, M., Kitani, K., & Sato, Y. (2016) · Robotics: Science and Systems · doi:10.15607/rss.2016.xii.034 · Cited by 74
Robust Hand Pose Estimation during the Interaction with an Unknown Object
Choi, C., Yoon, S. H., Chen, C.-N., & Karthik Ramani, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.339 · Cited by 63
Activity Recognition Using an Egocentric Perspective of Everyday Objects
Surie, D., Pederson, T., Lagriffoul, F., Janlert, L.-E., & Sjölie, D. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73549-6_25 · Cited by 62
Joint Learning of Object and Action Detectors
Kalogeiton, V. S., Weinzaepfel, P., Ferrari, V., & Schmid, C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.219 · Cited by 62
Occlusion-Aware Hand Pose Estimation Using Hierarchical Mixture Density Network
Ye, Q., & Kim, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_49 · Cited by 57
First-Person Action-Object Detection with EgoNet
Bertasius, G., Park, H. S., Yu, S. X., & Shi, J. (2017) · Robotics: Science and Systems · doi:10.15607/rss.2017.xiii.012 · Cited by 57
Joint 3D Tracking of a Deformable Object in Interaction with a Hand
Tsoli, A., & Argyros, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_30 · Cited by 49
Action recognition in RGB-D egocentric videos
Tang, Y., Tian, Y., Lu, J., Feng, J., & Zhou, J. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296915 · Cited by 47
Cited by
Learning Joint Reconstruction of Hands and Manipulated Objects
Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01208 · Cited by 513
HOnnotate: A Method for 3D Annotation of Hand and Object Poses
Hampali, S., Rad, M., Oberweger, M., & Lepetit, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00326 · Cited by 422
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
Chao, Y.-W., Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y. S., Van Wyk, K., Iqbal, U., Birchfield, S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00893 · Cited by 285
Understanding Human Hands in Contact at Internet Scale
Shan, D., Geng, J., Shu, M., & Fouhey, D. F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00989 · Cited by 256
Metadata
- Authors
- Tekin, B., Bogo, F., & Pollefeys, M.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 4506-4515
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2940216740
- Cited by
- 269
- References
- 75
- FWCI
- 12.82
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.