The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Learning Joint Reconstruction of Hands and Manipulated Objects
Open accessAbstract
Estimating hand-object manipulations is essential for in- terpreting and imitating human actions. Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation. Yet, reconstructing hands and objects during manipulation is a more challeng- ing task due to significant occlusions of both the hand and object. While presenting challenges, manipulations may also simplify the problem since the physics of contact re- stricts the space of valid hand-object configurations. For example, during manipulation, the hand and object should be in contact but not interpenetrate. In this work, we regu- larize the joint reconstruction of hands and objects with ma- nipulation constraints. We present an end-to-end learnable model that exploits a novel contact loss that favors phys- ically plausible hand-object constellations. Our approach improves grasp quality metrics over baselines, using RGB images as input. To train and evaluate the model, we also propose a new large-scale synthetic dataset, ObMan, with hand-object manipulations. We demonstrate the transfer- ability of ObMan-trained models to real data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Self-citationReal-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
VoxNet: A 3D Convolutional Neural Network for real-time object recognition
Maturana, D., & Scherer, S. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353481 · Cited by 3,458
A Point Set Generation Network for 3D Object Reconstruction from a Single Image
Fan, H., Su, H., & Guibas, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.264 · Cited by 2,402
End-to-End Recovery of Human Shape and Pose
Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00744 · Cited by 1,870
Self-citationDeep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
Choy, C. B., Xu, D., Gwak, J., Chen, K., & Savarese, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_38 · Cited by 1,632
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2023) · doi:10.5281/zenodo.10089705 · Cited by 1,623
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., & Jiang, Y.-G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_4 · Cited by 1,306
Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.058 · Cited by 1,268
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., & Xiao, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03365 · Cited by 1,245
Hand Keypoint Detection in Single Images Using Multiview Bootstrapping
Simon, T., Joo, H., Matthews, I., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.494 · Cited by 1,119
Romero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056
Fast, Minimum Storage Ray-Triangle Intersection
Möller, T., & Trumbore, B. (1997) · Journal of Graphics Tools · doi:10.1080/10867651.1997.10487468 · Cited by 1,038
The GRASP Taxonomy of Human Grasp Types
Feix, T., Romero, J., Schmiedmayer, H.-B., Dollar, A. M., & Kragic, D. (2016) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2015.2470657 · Cited by 1,024
Miller, A. T., & Allen, P. K. (2004) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2004.1371616 · Cited by 1,023
Kato, H., Ushiku, Y., & Harada, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00411 · Cited by 954
Learning from Synthetic Humans
Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M. J., Laptev, I., & Schmid, C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.492 · Cited by 908
Self-citationFerrari, C., & Canny, J. (2003) · Proceedings 1992 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1992.219918 · Cited by 893
Efficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 761
Learning to Estimate 3D Human Pose and Shape from a Single Color Image
Pavlakos, G., Zhu, L., Zhou, X., & Daniilidis, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00055 · Cited by 607
Real-time 3D model acquisition
Rusinkiewicz, S., Hall-Holt, O., & Levoy, M. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566600 · Cited by 576
Learning Category-Specific Mesh Reconstruction from Image Collections
Kanazawa, A., Tulsiani, S., Efros, A. A., & Malik, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_23 · Cited by 530
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00013 · Cited by 521
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Garcia-Hernando, G., Yuan, S., Baek, S., & Kim, T.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00050 · Cited by 519
An overview of 3D object grasp synthesis algorithms
Sahbani, A., El-Khoury, S., & Bidaud, P. (2012) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2011.07.016 · Cited by 478
Partitioned Sampling, Articulated Objects, and Interface-Quality Hand Tracking
MacCormick, J., & Isard, M. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45053-x_1 · Cited by 469
Visual tracking of high DOF articulated structures: An application to human hand tracking
Rehg, J. M., & Kanade, T. (1994) · Lecture Notes in Computer Science · doi:10.1007/bfb0028333 · Cited by 453
Sun, X., Wei, Y., Liang, S., Tang, X., & Sun, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298683 · Cited by 414
Real-time 3D model acquisition
Rusinkiewicz, S., Hall-Holt, O., & Levoy, M. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566600 · Cited by 319
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P., Breuel, T., Gall, J., & Kautz, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316
Goldfeder, C., Ciocarlie, M., Dang, H., & Allen, P. K. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152709 · Cited by 309
Hands Deep in Deep Learning for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06807 · Cited by 298
3D-CODED: 3D Correspondences by Deep Deformation
Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., & Aubry, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_15 · Cited by 293
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images
Cai, Y., Ge, L., Cai, J., & Yuan, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_41 · Cited by 291
Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_61 · Cited by 284
Cross-Modal Deep Variational Hand Pose Estimation
Spurr, A., Song, J., Park, S., & Hilliges, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00017 · Cited by 281
Motion Capture of Hands in Action Using Discriminative Salient Points
Ballan, L., Taneja, A., Gall, J., Van Gool, L., & Pollefeys, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_46 · Cited by 280
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0895-4 · Cited by 277
Self-citationOikonomidis, I., Kyriazis, N., & Argyros, A. A. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126483 · Cited by 274
Training a Feedback Loop for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.379 · Cited by 272
Modeling the constraints of human hand motion
Lin, J., Wu, Y., & Huang, T. S. (2002) · Proceedings Workshop on Human Motion · doi:10.1109/humo.2000.897381 · Cited by 270
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
Tekin, B., Bogo, F., & Pollefeys, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00464 · Cited by 269
Model-Based 3D Hand Pose Estimation from Monocular Video
de La Gorce, M., Fleet, D. J., & Paragios, N. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.33 · Cited by 268
Model-based 3D tracking of an articulated hand
Stenger, B., Mendonca, P. R. S., & Cipolla, R. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990976 · Cited by 263
Robinette, K. M., Blackwell, S., Daanen, H., Boehmer, M., & Fleming, S. (2002) · doi:10.21236/ada406704 · Cited by 260
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
Wu, J., Wang, Y., Xue, T., Sun, X., Freeman, W. T., & Tenenbaum, J. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03129 · Cited by 244
Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.131 · Cited by 242
Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
Sridhar, S., Mueller, F., Zollhöfer, M., Casas, D., Oulasvirta, A., & Theobalt, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_19 · Cited by 240
Capturing natural hand articulation
Wu, Y., Lin, J. Y., & Huang, T. S. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937656 · Cited by 229
Tracking the articulated motion of two strongly interacting hands
Oikonomidis, I., Kyriazis, N., & Argyros, A. A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247885 · Cited by 225
Towards 3D hand tracking using a deformable model
Heap, T., & Hogg, D. (2002) · Proceedings of the Second International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1996.557255 · Cited by 220
Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests
Tang, D., Yu, T.-H., & Kim, T.-K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.400 · Cited by 219
Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild
Panteleris, P., Oikonomidis, I., & Argyros, A. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00054 · Cited by 201
Tracking a hand manipulating an object
Hamer, H., Schindler, K., Koller-Meier, E., & Gool, L. V. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459282 · Cited by 149
Understanding Everyday Hands in Action from RGB-D Images
Rogez, G., Supančič, J. S., & Ramanan, D. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.443 · Cited by 147
Hands in action: real-time 3D reconstruction of hands in interaction with objects
Romero, J., Kjellström, H., & Kragić, D. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509753 · Cited by 124
3D Hand Pose Tracking and Estimation Using Stereo MatchingPreprint
Zhang, J., Jiao, J., Chen, M., Qu, L., Xu, X., & Yang, Q. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07214 · Cited by 108
First-person pose recognition using egocentric workspaces
Rogez, G., Supančič, J. S., & Ramanan, D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299061 · Cited by 94
Hand-Object Contact Force Estimation from Markerless Visual Tracking
Pham, T.-H., Kyriazis, N., Argyros, A., & Kheddar, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2759736 · Cited by 90
DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth
Malik, J. N., Elhayek, A., Nunnari, F., Varanasi, K., Tamaddon, K., Héloir, A., & Stricker, D. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00023 · Cited by 85
Video-based hand manipulation capture through composite motion control
Wang, Y., Min, J., Zhang, J., Liu, Y., Xu, F., Dai, Q., & Chai, J. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462000 · Cited by 83
3D Hand Pose Detection in Egocentric RGB-D Images
Rogez, G., Khademi, M., Supančič, J. S., Montiel, J. M. M., & Ramanan, D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_25 · Cited by 81
3D Object Reconstruction from Hand-Object Interactions
Tzionas, D., & Gall, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.90 · Cited by 64
Self-citationAn object-dependent hand pose prior from sparse training data
Hamer, H., Gall, J., Weise, T., & Gool, L. V. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540150 · Cited by 62
Online loop closure for real-time interactive 3D scanning
Weise, T., Wismer, T., Leibe, B., & Gool, L. V. (2011) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2010.11.023 · Cited by 51
Joint 3D Tracking of a Deformable Object in Interaction with a Hand
Tsoli, A., & Argyros, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_30 · Cited by 49
3D Hand Pose Detection in Egocentric RGB-D ImagesPreprint
Rogez, G., Supančič, J. S., Khademi, M., Montiel, J. M. M., & Ramanan, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.0065 · Cited by 48
Monocular RGB Hand Pose Inference from Unsupervised Refinable Nets
Dibra, E., Melchior, S., Balkis, A., Wolf, T., Öztireli, C., & Groß, M. H. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00155 · Cited by 25
Learning to Estimate 3D Human Pose and Shape from a Single Color ImagePreprint
Pavlakos, G., Zhu, L., Zhou, X., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04092 · Cited by 22
3D-CODED : 3D Correspondences by Deep Deformation
Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., & Aubry, M. (2018) · HAL · doi:10.48550/arxiv.1806.05228 · Cited by 6
Cited by
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images
Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M. J., & Brox, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00090 · Cited by 445
HOnnotate: A Method for 3D Annotation of Hand and Object Poses
Hampali, S., Rad, M., Oberweger, M., & Lepetit, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00326 · Cited by 422
Moon, G., & Lee, K. M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_44 · Cited by 377
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
Chao, Y.-W., Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y. S., Van Wyk, K., Iqbal, U., Birchfield, S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00893 · Cited by 285
Understanding Human Hands in Contact at Internet Scale
Shan, D., Geng, J., Shu, M., & Fouhey, D. F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00989 · Cited by 256
Metadata
- Authors
- Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 11799-11808
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2935836316
- Cited by
- 513
- References
- 76
- FWCI
- 78.27
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.