Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation
Open accessThe DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
We propose a robust and accurate method for estimating the 3D poses of two hands in close interaction from a single color image. This is a very challenging problem, as large occlusions and many confusions between the joints may happen. State-of-the-art methods solve this problem by regressing a heatmap for each joint, which requires solving two problems simultaneously: localizing the joints and recognizing them. In this work, we propose to separate these tasks by relying on a CNN to first localize joints as 2D keypoints, and on self-attention between the CNN features at these keypoints to associate them with the corresponding hand joint. The resulting architecture, which we call “Keypoint Transformer”, is highly efficient as it achieves state-of-the-art performance with roughly half the number of model parameters on the InterHand2.6M dataset. We also show it can be easily extended to estimate the 3D pose of an object manipulated by one or two hands with high performance. Moreover, we created a new dataset of more than 75,000 images of two hands manipulating an object fully annotated in 3D and will make it publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027 · cited 3×
“[17] estimated a 3D pose from hand pointcloud data using a Transformer encoder-decoder architecture and proposed to generate query embeddings from input point-cloud instead of learning them as in [8, 60].”
BackgroundMethodTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · cited 2×
“We refer the reader to [21] for a detailed survey.”
BackgroundTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
End-to-End Recovery of Human Shape and Pose
Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00744 · Cited by 2,407
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.019 · Cited by 2,139
On the Continuity of Rotation Representations in Neural Networks
Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00589 · Cited by 1,255 · cited 1×
“The object rotation is parameterized using the method proposed in [59].”
MethodRomero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056
Learning Texture Transformer Network for Image Super-Resolution
Yang, F., Yang, H., Fu, J., Lu, H., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00583 · Cited by 966
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 761 · cited 1×
“Many approaches have been proposed, mostly based on direct prediction with different convolutional network architectures [15, 18, 29, 36, 44, 49, 61] of the 3D joint locations or angles, or relying on rendering for fine pose estimation and tracking [2, 12, 32, 40, 50].”
BackgroundEnd-to-End Human Pose and Mesh Reconstruction with Transformers
Lin, K., Wang, L., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00199 · Cited by 740 · cited 3×
“Features are often extracted from a CNN backbone and different architectures have been proposed to solve object detection [8, 60], image classification [9], pose estimation [6, 17, 27, 28] and low-level image tasks [24, 57].”
BackgroundMethodLearning to Estimate 3D Human Pose and Shape from a Single Color Image
Pavlakos, G., Zhu, L., Zhou, X., & Daniilidis, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00055 · Cited by 607
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Garcia-Hernando, G., Yuan, S., Baek, S., & Kim, T.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00050 · Cited by 519
Learning Joint Reconstruction of Hands and Manipulated Objects
Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01208 · Cited by 513 · cited 10×
“However, regressing MANO joint angles provides access to the complete hand mesh required for modeling contacts and interpenetration during interactions [5, 15, 46] or for learning in a weakly supervised setup [3,14,23], which could be interesting for future extension of our method.”
BackgroundMethodResultInfluentialPix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation
Park, K., Patten, T., & Vincze, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00776 · Cited by 455 · cited 1×
“We use a symmetry-aware object corner loss similar to [37] to train the network:”
MethodFreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images
Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M. J., & Brox, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00090 · Cited by 445 · cited 2×
“Discriminative methods [5, 14, 15, 20, 30, 36, 49, 62] mostly directly predict the hand pose from a single frame.”
BackgroundHOnnotate: A Method for 3D Annotation of Hand and Object Poses
Hampali, S., Rad, M., Oberweger, M., & Lepetit, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00326 · Cited by 422 · cited 15×
“This dataset is based on the work of [12], and we call it H2O-3D.”
BackgroundMethodResultInfluentialSelf-citationMoon, G., & Lee, K. M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_44 · Cited by 377
6-DoF object pose from semantic keypoints
Pavlakos, G., Zhou, X., Chan, A., Derpanis, K. G., & Daniilidis, K. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989233 · Cited by 366 · cited 1×
“As we rely on keypoints, we believe that our approach is more general and could be applied to other problems, such as human and other articulated objects pose prediction and object category pose prediction [39].”
MethodLin, K., Wang, L., & Liu, Z. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01270 · Cited by 337 · cited 2×
“Features are often extracted from a CNN backbone and different architectures have been proposed to solve object detection [8, 60], image classification [9], pose estimation [6, 17, 27, 28] and low-level image tasks [24, 57].”
BackgroundMethodMonocular Total Capture: Posing Face, Body, and Hands in the Wild
Xiang, D., Joo, H., & Sheikh, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01122 · Cited by 335 · cited 1×
“Hybrid approaches [4, 7, 13, 31, 43, 45, 47, 48, 51, 54,55] try to combine the best of these two worlds by using discriminative methods to detect visual cues in the image followed by model fitting.”
MethodHand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P., Breuel, T., Gall, J., & Kautz, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316
Taylor, J., Bordeaux, L., Cashman, T., Corish, B., Keskin, C., Sharp, T., Soto, E., Sweeney, D., Valentin, J., Luff, B., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925965 · Cited by 294 · cited 1×
“Hybrid approaches [4, 7, 13, 31, 43, 45, 47, 48, 51, 54,55] try to combine the best of these two worlds by using discriminative methods to detect visual cues in the image followed by model fitting.”
MethodMoon, G., Yu, S.-I., Wen, H., Shiratori, T., & Lee, K. M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58565-5_33 · Cited by 293
Motion Capture of Hands in Action Using Discriminative Salient Points
Ballan, L., Taneja, A., Gall, J., Van Gool, L., & Pollefeys, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_46 · Cited by 280 · cited 2×
“[4, 51] detect the fingertips as discriminative points and used them in the optimization along with a collision term and physical modelling.”
MethodCapturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0895-4 · Cited by 277 · cited 2×
“[4, 51] detect the fingertips as discriminative points and used them in the optimization along with a collision term and physical modelling.”
MethodTraining a Feedback Loop for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.379 · Cited by 272 · cited 1×
“Many approaches have been proposed, mostly based on direct prediction with different convolutional network architectures [15, 18, 29, 36, 44, 49, 61] of the 3D joint locations or angles, or relying on rendering for fine pose estimation and tracking [2, 12, 32, 40, 50].”
BackgroundSelf-citationH+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
Tekin, B., Bogo, F., & Pollefeys, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00464 · Cited by 269 · cited 3×
“Many approaches have been proposed, mostly based on direct prediction with different convolutional network architectures [15, 18, 29, 36, 44, 49, 61] of the 3D joint locations or angles, or relying on rendering for fine pose estimation and tracking [2, 12, 32, 40, 50].”
BackgroundMethodBOP Challenge 2020 on 6D Object Localization
Hodaň, T., Sundermeyer, M., Drost, B., Labbé, Y., Brachmann, E., Michel, F., Rother, C., & Matas, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-66096-3_39 · Cited by 268
Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
Sridhar, S., Mueller, F., Zollhöfer, M., Casas, D., Oulasvirta, A., & Theobalt, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_19 · Cited by 240 · cited 1×
“Hybrid approaches [4, 7, 13, 31, 43, 45, 47, 48, 51, 54,55] try to combine the best of these two worlds by using discriminative methods to detect visual cues in the image followed by model fitting.”
MethodReconstructing Hand-Object Interactions in the Wild
Cao, Z., Radosavovic, I., Kanazawa, A., & Malik, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01219 · Cited by 227 · cited 2×
“[7,13,31,36,54] compute dense features or keypoints from a single RGB or”
BackgroundMethodTracking the articulated motion of two strongly interacting hands
Oikonomidis, I., Kyriazis, N., & Argyros, A. A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247885 · Cited by 225
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild
Kulon, D., Guler, R. A., Kokkinos, I., Bronstein, M. M., & Zafeiriou, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00504 · Cited by 212
MEgATrack: monochrome egocentric articulated hand-tracking for virtual reality
Han, S., Liu, B., Cabezas, R., Twigg, C. D., Zhang, P., Petkau, J., Yu, T.-H., Tai, C.-J., Akbay, M., Wang, Z., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392452 · Cited by 205 · cited 3×
“[7,13,31,36,54] compute dense features or keypoints from a single RGB or”
BackgroundMethodPushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering
Baek, S., Kim, K. I., & Kim, T.-K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00116 · Cited by 201 · cited 1×
“Many approaches have been proposed, mostly based on direct prediction with different convolutional network architectures [15, 18, 29, 36, 44, 49, 61] of the 3D joint locations or angles, or relying on rendering for fine pose estimation and tracking [2, 12, 32, 40, 50].”
BackgroundWeakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Spurr, A., Iqbal, U., Molchanov, P. A., Hilliges, O., & Kautz, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_13 · Cited by 201
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose
Brahmbhatt, S., Tang, C., Twigg, C. D., Kemp, C. C., & Hays, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58601-0_22 · Cited by 200
Grasping Field: Learning Implicit Representations for Human Grasps
Karunratanakul, K., Yang, J., Zhang, Y., Black, M. J., Muandet, K., & Tang, S. (2020) · 2020 International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00043 · Cited by 198
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction
Hasson, Y., Tekin, B., Bogo, F., Laptev, I., Pollefeys, M., & Schmid, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00065 · Cited by 185
Real-time pose and shape reconstruction of two interacting hands with a single depth camera
Mueller, F., Davis, M., Bernard, F., Sotnychenko, O., Verschoor, M., Otaduy, M. A., Casas, D., & Theobalt, C. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322958 · Cited by 169
RGB2Hands: real-time tracking of 3D hand interactions from monocular RGB video
Wang, J., Mueller, F., Bernard, F., Sorli, S., Sotnychenko, O., Qian, N., Otaduy, M. Á., Casas, D., & Theobalt, C. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417852 · Cited by 155 · cited 3×
“Bottom-up approaches [30,54] directly estimate the 2D joint locations and their depths using one heatmap per joint.”
BackgroundMethodHand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation
Huang, L., Tan, J., Liu, J., & Yuan, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_2 · Cited by 140
TexturePose: Supervising Human Mesh Estimation With Texture Consistency
Pavlakos, G., Kolotouros, N., & Daniilidis, K. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00089 · Cited by 124
Interacting Two-Hand 3D Pose and Shape Reconstruction from Single Color Image
Zhang, B., Wang, Y., Deng, X., Zhang, Y., Tan, P., Ma, C., & Wang, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01116 · Cited by 123 · cited 2×
“[10, 22] improve the accuracy of 3D pose estimation during hand-hand interaction scenarios by incorporating joint-visibility and partsegmentation cues, whereas [58] utilize refinement layers to iteratively refine the estimated poses.”
BackgroundBaek, S., Kim, K. I., & Kim, T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00616 · Cited by 109 · cited 1×
“However, regressing MANO joint angles provides access to the complete hand mesh required for modeling contacts and interpenetration during interactions [5, 15, 46] or for learning in a weakly supervised setup [3,14,23], which could be interesting for future extension of our method.”
MethodArticulated distance fields for ultra-fast tracking of hands interacting
Taylor, J., Tankovich, V., Tang, D., Keskin, C., Kim, D., Davidson, P., Kowdle, A., & Izadi, S. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130853 · Cited by 91 · cited 1×
“Hybrid approaches [4, 7, 13, 31, 43, 45, 47, 48, 51, 54,55] try to combine the best of these two worlds by using discriminative methods to detect visual cues in the image followed by model fitting.”
MethodConstraining dense hand surface tracking with elasticity
Smith, B., Wu, C., Wen, H., Peluse, P., Sheikh, Y. A., Hodgins, J. K., & Shiratori, T. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417768 · Cited by 85
3D Object Reconstruction from Hand-Object Interactions
Tzionas, D., & Gall, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.90 · Cited by 78
The Center of Attention: Center-Keypoint Grouping via Attention for Multi-Person Pose Estimation
Brasó, G., Kister, N., & Leal-Taixé, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01164 · Cited by 68 · cited 4×
“Our method is also robust to undetected and falsely detected keypoints as we show in our discussions, while [6] cannot handle undetected keypoints.”
BackgroundMethodInfluentialLearning To Disambiguate Strongly Interacting Hands via Probabilistic Per-Pixel Part Segmentation
Fan, Z., Spurr, A., Kocabas, M., Tang, S., Black, M. J., & Hilliges, O. (2021) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00011 · Cited by 68 · cited 5×
“[10,22] use per-joint heatmaps coupled with joint visibility and segmentation guided features to improve the accuracy of the pose estimation, thus resulting in the model complexity that is higher than InterNet [30].”
BackgroundMethodInfluentialEnd-to-End Detection and Pose Estimation of Two Interacting Hands
Kim, D. U., Kim, K. I., & Baek, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01100 · Cited by 64 · cited 5×
“[10,22] use per-joint heatmaps coupled with joint visibility and segmentation guided features to improve the accuracy of the pose estimation, thus resulting in the model complexity that is higher than InterNet [30].”
BackgroundMethodInfluentialArmagan, A., Garcia-Hernando, G., Baek, S., Hampali, S., Rad, M., Zhang, Z., Xie, S., Chen, M., Zhang, B., Xiong, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58592-1_6 · Cited by 63
Self-citation
Cited by
A survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 451
Best practices for sign language technology research
Fox, N., Woll, B., & Cormier, K. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-023-01039-1 · Cited by 27
Metadata
- Authors
- Hampali, S., Sarkar, S. D., Rad, M., & Lepetit, V.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 11080-11090
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4298014233
- Cited by
- 162
- References
- 55
- FWCI
- 14.39
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.