Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 1×
“To evaluate the performance of the detector, we follow the PASCAL-VOC protocol (Everingham et al, 2010).”
MethodA Mathematical Introduction to Robotic Manipulation
Murray, R. M., Li, Z., & Sastry, S. S. (2017) · CRC Press eBooks · doi:10.1201/9781315136370 · Cited by 6,694
Shape matching and object recognition using shape contexts
Belongie, S., Malik, J., & Puzicha, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.993558 · Cited by 6,306 · cited 2×
“We use γa = 0.0015 Call, where Call is the total number of correspondences.”
MethodInfluentialEfficient variants of the ICP algorithm
Rusinkiewicz, S., & Levoy, M. (2002) · Proceedings Third International Conference on 3-D Digital Imaging and Modeling · doi:10.1109/im.2001.924423 · Cited by 3,945
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
Statistical Color Models with Application to Skin Detection
Jones, M. J., & Rehg, J. M. (2002) · International Journal of Computer Vision · doi:10.1023/a:1013200319198 · Cited by 1,542
Object modeling by registration of multiple range images
Chen, Y., & Medioni, G. (2002) · Proceedings. 1991 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1991.132043 · Cited by 1,409 · cited 2×
“…as a least squared error of point-to-point distances: Emodel→data(θ,D) = ∑ i ‖vi(θ)−Xi‖2 (6) An alternative to the point-to-point distance is the pointto-plane distance, which is commonly used for 3D reconstruction (Chen and Medioni, 1991; Rusinkiewicz and Levoy, 2001; Rusinkiewicz et al, 2002).”
BackgroundMethodEfficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
Vision-based hand pose estimation: A review
Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., & Twombly, X. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.012 · Cited by 866
Cited by
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01123 · Cited by 1,781
“is redirected to [4, 28], but for the sake of completion we include the mathematical formulation also here.”
BackgroundMethodSelf-citationHand Keypoint Detection in Single Images Using Multiview Bootstrapping
Simon, T., Joo, H., Matthews, I., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.494 · Cited by 1,119
Romero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056
“2e) is more realistic and better fits image data [Ballan et al. 2012; Tzionas et al. 2016].”
BackgroundMethodResultInfluentialGANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00013 · Cited by 521
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Garcia-Hernando, G., Yuan, S., Baek, S., & Kim, T.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00050 · Cited by 519
Learning Joint Reconstruction of Hands and Manipulated Objects
Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01208 · Cited by 513
Self-citationTotal Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies
Joo, H., Simon, T., & Sheikh, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00868 · Cited by 509
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images
Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M. J., & Brox, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00090 · Cited by 445
HOnnotate: A Method for 3D Annotation of Hand and Object Poses
Hampali, S., Rad, M., Oberweger, M., & Lepetit, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00326 · Cited by 422
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models
Xu, H., Bazavan, E. G., Zanfir, A., Freeman, W. T., Sukthankar, R., & Sminchisescu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00622 · Cited by 347
Monocular Total Capture: Posing Face, Body, and Hands in the Wild
Xiang, D., Joo, H., & Sheikh, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01122 · Cited by 335
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P., Breuel, T., Gall, J., & Kautz, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316
Self-citationHand PointNet: 3D Hand Pose Estimation Using Point Sets
Ge, L., Cai, Y., Weng, J., & Yuan, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00878 · Cited by 311
Taylor, J., Bordeaux, L., Cashman, T., Corish, B., Keskin, C., Sharp, T., Soto, E., Sweeney, D., Valentin, J., Luff, B., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925965 · Cited by 294
Moon, G., Yu, S.-I., Wen, H., Shiratori, T., & Lee, K. M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58565-5_33 · Cited by 293
Ge, L., Liang, H., Yuan, J., & Thalmann, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.602 · Cited by 291
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images
Cai, Y., Ge, L., Cai, J., & Yuan, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_41 · Cited by 291
Cross-Modal Deep Variational Hand Pose Estimation
Spurr, A., Song, J., Park, S., & Hilliges, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00017 · Cited by 281
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints
Hassan, M., Choutas, V., Tzionas, D., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00237 · Cited by 274
Self-citationGRAB: A Dataset of Whole-Body Human Grasping of Objects
Taheri, O., Ghorbani, N., Black, M. J., & Tzionas, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_34 · Cited by 274
Self-citationH+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
Tekin, B., Bogo, F., & Pollefeys, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00464 · Cited by 269
Self-citationReal-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.131 · Cited by 242
“Meth-ods for capturing complex hand-object interactions and object scanning were proposed [20, 1, 13, 41, 40, 21].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction
Hasson, Y., Tekin, B., Bogo, F., Laptev, I., Pollefeys, M., & Schmid, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00065 · Cited by 185
Self-citationHand-Object Contact Consistency Reasoning for Human Grasps Generation
Jiang, H., Liu, S., Wang, J., & Wang, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01092 · Cited by 165
Hampali, S., Sarkar, S. D., Rad, M., & Lepetit, V. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01081 · Cited by 162
“[4, 51] detect the fingertips as discriminative points and used them in the optimization along with a collision term and physical modelling.”
Method
Metadata
- Authors
- Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J.
- Venue
- International Journal of Computer Vision
- Published
- 2016-03-18
- Volume
- 118
- Issue
- 2
- Pages
- 172-193
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2227547437
- Cited by
- 277
- References
- 71
- FWCI
- 36.25
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.