The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences
Abstract
Fully articulated hand tracking promises to enable fundamentally new interactions with virtual and augmented worlds, but the limited accuracy and efficiency of current systems has prevented widespread adoption. Today's dominant paradigm uses machine learning for initialization and recovery followed by iterative model-fitting optimization to achieve a detailed pose fit. We follow this paradigm, but make several changes to the model-fitting, namely using: (1) a more discriminative objective function; (2) a smooth-surface model that provides gradients for non-linear optimization; and (3) joint optimization over both the model pose and the correspondences between observed data points and the model surface. While each of these changes may actually increase the cost per fitting iteration, we find a compensating decrease in the number of iterations. Further, the wide basin of convergence means that fewer starting points are needed for successful model fitting. Our system runs in real-time on CPU only, which frees up the commonly over-burdened GPU for experience designers. The hand tracker is efficient enough to run on low-power devices such as tablets. We can track up to several meters from the camera to provide a large working volume for interaction, even using the noisy data from current-generation depth cameras. Quantitative assessments on standard datasets show that the new approach exceeds the state of the art in accuracy. Qualitative results take the form of live recordings of a range of interactive experiences enabled by this new approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bundle Adjustment — A Modern Synthesis
Triggs, B., McLauchlan, P. F., Hartley, R. I., & Fitzgibbon, A. W. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-44480-7_21 · Cited by 3,988
Self-citationLoper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
Self-citationSmooth Subdivision Surfaces Based on Triangles
Loop, C. (1987) · Cited by 1,319
Efficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
Vision-based hand pose estimation: A review
Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., & Twombly, X. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.012 · Cited by 866
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
A Survey of Glove-Based Systems and Their Applications
Dipietro, L., Sabatini, A. M., & Dario, P. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2008.923862 · Cited by 691
Real-time hand-tracking with a color glove
Wang, R. Y., & Popović, J. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531369 · Cited by 609
Statistical methods for tomographic image reconstruction
Geman, S., & McClure, D. E. (1987) · Medical Entomology and Zoology · Cited by 579
Realtime and Robust Hand Tracking from Depth
Qian, C., Sun, X., Wei, Y., Tang, X., & Sun, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.145 · Cited by 470
Visual tracking of high DOF articulated structures: An application to human hand tracking
Rehg, J. M., & Kanade, T. (1994) · Lecture Notes in Computer Science · doi:10.1007/bfb0028333 · Cited by 453
Accurate, Robust, and Flexible Real-time Hand Tracking
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., et al. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702179 · Cited by 431
Self-citationDigits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor
Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., & Olivier, P. (2012) · Proceedings of the 25th annual ACM symposium on User interface software and technology · doi:10.1145/2380116.2380139 · Cited by 428
Self-citationSun, X., Wei, Y., Liang, S., Tang, X., & Sun, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298683 · Cited by 414
Estimating 3D hand pose from a cluttered image
Athitsos, V., & Sclaroff, S. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211500 · Cited by 379
Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_61 · Cited by 284
Self-citationLatent-Class Hough Forests for 3D Object Detection and Pose Estimation
Tejani, A., Tang, D., Kouskouridas, R., & Kim, T.-K. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_30 · Cited by 280
Motion Capture of Hands in Action Using Discriminative Salient Points
Ballan, L., Taneja, A., Gall, J., Van Gool, L., & Pollefeys, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_46 · Cited by 280
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0895-4 · Cited by 277
Training a Feedback Loop for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.379 · Cited by 272
The Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation
Taylor, J., Shotton, J., Sharp, T., & Fitzgibbon, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247664 · Cited by 270
Self-citationModel-Based 3D Hand Pose Estimation from Monocular Video
de La Gorce, M., Fleet, D. J., & Paragios, N. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.33 · Cited by 268
Model-based 3D tracking of an articulated hand
Stenger, B., Mendonca, P. R. S., & Cipolla, R. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990976 · Cited by 263
Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data
Sridhar, S., Oulasvirta, A., & Theobalt, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.305 · Cited by 247
Capturing natural hand articulation
Wu, Y., Lin, J. Y., & Huang, T. S. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937656 · Cited by 229
Fast and robust hand tracking using detection-guided optimization
Sridhar, S., Mueller, F., Oulasvirta, A., & Theobalt, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298941 · Cited by 228
6D hands: markerless hand-tracking for computer aided design
Wang, R., Paris, S., & Popović, J. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047269 · Cited by 226
Towards 3D hand tracking using a deformable model
Heap, T., & Hogg, D. (2002) · Proceedings of the Second International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1996.557255 · Cited by 220
Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests
Tang, D., Yu, T.-H., & Kim, T.-K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.400 · Cited by 219
Robust Articulated‐ICP for Real‐Time Hand Tracking
Tagliasacchi, A., Schröder, M., Tkach, A., Bouaziz, S., Botsch, M., & Pauly, M. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12700 · Cited by 216
Efficient Hand Pose Estimation from a Single Depth Image
Xu, C., & Cheng, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.429 · Cited by 187
View-independent recognition of hand postures
Wu, Y., & Huang, T. S. (2002) · Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662) · doi:10.1109/cvpr.2000.854749 · Cited by 185
Dynamics based 3D skeletal hand tracking
Melax, S., Keselman, L., & Orsten, S. (2013) · Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games · doi:10.1145/2448196.2448232 · Cited by 172
Spectrally optimal sampling for distribution ray tracing
Mitchell, D. P. (1991) · Proceedings of the 18th annual conference on Computer graphics and interactive techniques · doi:10.1145/122718.122736 · Cited by 167
Monnai, Y., Hasegawa, K., Fujiwara, M., Yoshino, K., Inoue, S., & Shinoda, H. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647407 · Cited by 152
Decision Jungles: Compact and Rich Models for Classification
Shotton, J., Sharp, T., Kohli, P., Nowozin, S., Winn, J. M., & Criminisi, A. (2013) · neural information processing systems · Cited by 151
Self-citationProceedings of the Graphics Interface 2001 Conference
Komura, T., & Shinagawa, Y. (2001) · Cited by 144
Opening the Black Box: Hierarchical Sampling Optimization for Estimating Human Hand Pose
Tang, D., Taylor, J., Kohli, P., Keskin, C., Kim, T., & Shotton, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.380 · Cited by 142
Self-citationMulti-output Learning for Camera Relocalization
Guzman-Rivera, A., Kohli, P., Glocker, B., Shotton, J., Sharp, T., Fitzgibbon, A. W., & Izadi, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.146 · Cited by 117
Self-citationSmart particle filtering for 3D hand tracking
Bray, M., Koller-Meier, E., & Gool, L. V. (2004) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2004.1301612 · Cited by 116
Learning an efficient model of hand shape variation from depth images
Khamis, S., Taylor, J., Shotton, J., Keskin, C., Izadi, S., & Fitzgibbon, A. W. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298869 · Cited by 115
Self-citationRobust Bundle Adjustment Revisited
Zach, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_50 · Cited by 109
Skinning: real-time shape deformation
Jacobson, A., & Gingold, Y. I. (2014) · SIGGRAPH Asia Courses · doi:10.1145/2659467.2675048 · Cited by 107
User-Specific Hand Modeling from Monocular Depth Sequences
Taylor, J., Stebbing, R. V., Ramakrishna, V., Keskin, C., Shotton, J., Izadi, S., Hertzmann, A., & Fitzgibbon, A. W. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.88 · Cited by 92
Self-citationVideo-based hand manipulation capture through composite motion control
Wang, Y., Min, J., Zhang, J., Liu, Y., Xu, F., Dai, Q., & Chai, J. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462000 · Cited by 83
Fits Like a Glove: Rapid and Reliable Hand Shape Personalization
Tan, D. J., Cashman, T. J., Taylor, J., Fitzgibbon, A. W., Tarlow, D., Khamis, S., Izadi, S., & Shotton, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.605 · Cited by 82
Self-citation3D Hand Pose Estimation Using Randomized Decision Forest with Segmentation Index Points
Li, P., Ling, H., Li, X., & Liao, C. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.100 · Cited by 62
Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model
Sridhar, S., Rhodin, H., Seidel, H., Oulasvirta, A., & Theobalt, C. (2014) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2014.37 · Cited by 58
Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model
Sridhar, S., Rhodin, H., Seidel, H., Oulasvirta, A., & Theobalt, C. (2014) · MPG.PuRe (Max Planck Society) · Cited by 47
Balloon Shapes: Reconstructing and Deforming Objects with Volume from Images
Vicente, S., & Agapito, L. (2013) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2013.37 · Cited by 46
Hierarchical particle filtering for 3D hand tracking
Makris, A., Kyriazis, N., & Argyros, A. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301343 · Cited by 30
Discriminative Ferns Ensemble for Hand Pose Recognition
Krupka, E., Vinnikov, A., Klein, B., Bar-Hillel, A., Freedman, D., & Stachniak, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.469 · Cited by 29
ICPIK: Inverse Kinematics based articulated-ICP
Fleishman, S., Kliger, M., Lerner, A., & Kutliroff, G. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301345 · Cited by 25
Cited by
AMASS: Archive of Motion Capture As Surface Shapes
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00554 · Cited by 1,323
Romero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056
Hand Gesture Recognition Based on Computer Vision: A Review of Techniques
Oudah, M., Al-Naji, A., & Chahl, J. (2020) · Journal of Imaging · doi:10.3390/jimaging6080073 · Cited by 635
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00013 · Cited by 521
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models
Xu, H., Bazavan, E. G., Zanfir, A., Freeman, W. T., Sukthankar, R., & Sminchisescu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00622 · Cited by 347
Metadata
- Authors
- Taylor, J., Bordeaux, L., Cashman, T., Corish, B., Keskin, C., Sharp, T., Soto, E., Sweeney, D., Valentin, J., Luff, B., Topalian, A., Wood, E., Khamis, S., Kohli, P., Izadi, S., Banks, R., Fitzgibbon, A., & Shotton, J.
- Venue
- ACM Transactions on Graphics
- Published
- 2016-07-11
- Volume
- 35
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2469784314
- Cited by
- 294
- References
- 54
- FWCI
- 72.49
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.