Accurate, Robust, and Flexible Real-time Hand Tracking
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
Self-citationA Hybrid of Genetic Algorithm and Particle Swarm Optimization for Recurrent Network Design
Juang, C.-F. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2003.818557 · Cited by 964 · cited 1×
“MODEL FITTING In order to optimize the golden energy and achieve an accurate hand pose estimate, we employ a model fitting algorithm that combines features of particle swarm optimization (PSO) and genetic algorithms (GA) in the spirit of [11, 17, 20].”
MethodEfficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874 · cited 6×
“MODEL FITTING In order to optimize the golden energy and achieve an accurate hand pose estimate, we employ a model fitting algorithm that combines features of particle swarm optimization (PSO) and genetic algorithms (GA) in the spirit of [11, 17, 20].”
BackgroundMethodInfluentialVision-based hand pose estimation: A review
Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., & Twombly, X. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.012 · Cited by 866 · cited 1×
“RGB input Early work relied on monocular RGB cameras, making the problem extremely challenging (see [8] for a survey).”
BackgroundVision as Bayesian inference: analysis by synthesis?
Yuille, A., & Kersten, D. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.002 · Cited by 849
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801 · cited 3×
“[30] demonstrate impressive hand tracking results using deep neural networks to predict feature locations and IK to infer a skeleton.”
BackgroundMethodA Survey of Glove-Based Systems and Their Applications
Dipietro, L., Sabatini, A. M., & Dario, P. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2008.923862 · Cited by 691 · cited 1×
“RELATED WORK Given our aims described above, we avoid encumbering the user’s hand with data gloves [7], colored gloves [32], wearable cameras [13], or markers [38], all of which can be barriers for natural interaction.”
BackgroundReal-time hand-tracking with a color glove
Wang, R. Y., & Popović, J. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531369 · Cited by 609
Decision Forests for Computer Vision and Medical Image Analysis
(2013) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-1-4471-4929-3 · Cited by 595
Self-citationRealtime and Robust Hand Tracking from Depth
Qian, C., Sun, X., Wei, Y., Tang, X., & Sun, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.145 · Cited by 470 · cited 4×
“Particle ‘Aging’ A per-generation clustering step was recently proposed in [20] to reduce an effect called ‘particle collapse’ whereby all particles are pulled too close together to a bad solution.”
BackgroundMethodInfluentialSelf-citationDigits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor
Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., & Olivier, P. (2012) · Proceedings of the 25th annual ACM symposium on User interface software and technology · doi:10.1145/2380116.2380139 · Cited by 428 · cited 1×
“RELATED WORK Given our aims described above, we avoid encumbering the user’s hand with data gloves [7], colored gloves [32], wearable cameras [13], or markers [38], all of which can be barriers for natural interaction.”
BackgroundSelf-citationLatent Regression Forest: Structured Estimation of 3D Articulated Hand Posture
Tang, D., Chang, H. J., Tejani, A., & Kim, T.-K. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.490 · Cited by 398 · cited 1×
“[27, 26] extend this work demonstrating more complex poses at 25Hz.”
BackgroundEstimating 3D hand pose from a cluttered image
Athitsos, V., & Sclaroff, S. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211500 · Cited by 379 · cited 1×
“Discriminative methods [2, 34] used small databases of restricted hand poses, limiting accuracy.”
MethodEfficient regression of general-activity human poses from depth images
Girshick, R., Shotton, J., Kohli, P., Criminisi, A., & Fitzgibbon, A. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126270 · Cited by 344 · cited 1×
“The offset translation regressor [9] is trained to predict the 3D offset from any RoI pixel’s 3D camera space location to the wrist joint position (i.”
MethodSelf-citationHand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_61 · Cited by 284 · cited 6×
“Existing approaches include multi-layer random forests [12], nearest-neighbor look-up [31], fingertip detection [20], or convolutional neural networks [30].”
BackgroundMethodInfluentialSelf-citationMotion Capture of Hands in Action Using Discriminative Salient Points
Ballan, L., Taneja, A., Gall, J., Van Gool, L., & Pollefeys, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_46 · Cited by 280 · cited 3×
“[3] demonstrate high-quality results closely fitting a detailed scanned mesh model to complex two-handed and hand-object interactions.”
BackgroundMethodOikonomidis, I., Kyriazis, N., & Argyros, A. A. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126483 · Cited by 274 · cited 1×
“Proofs of concept [19, 18] suggest that extending our approach to deal with these cases should not be too challenging, at least for the model fitter.”
BackgroundThe Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation
Taylor, J., Shotton, J., Sharp, T., & Fitzgibbon, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247664 · Cited by 270 · cited 3×
“We dub the above function the ‘golden energy’ to distinguish it from energies that do not directly aim to explain the image data such as [16, 23, 28].”
BackgroundMethodSelf-citationModel-Based 3D Hand Pose Estimation from Monocular Video
de La Gorce, M., Fleet, D. J., & Paragios, N. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.33 · Cited by 268 · cited 1×
“[6] automatically apply a scaling to the bones in the hand model during tracking.”
MethodModel-based 3D tracking of an articulated hand
Stenger, B., Mendonca, P. R. S., & Cipolla, R. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990976 · Cited by 263 · cited 1×
“[24, 35]), again resulting in limited accuracy.”
Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data
Sridhar, S., Oulasvirta, A., & Theobalt, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.305 · Cited by 247 · cited 9×
“Table 1 compares mean joint position errors obtained by our tracker and that of [23] (numbers are estimated from a graph in their paper).”
BackgroundMethodResultInfluentialCapturing natural hand articulation
Wu, Y., Lin, J. Y., & Huang, T. S. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937656 · Cited by 229 · cited 1×
“[24, 35]), again resulting in limited accuracy.”
6D hands: markerless hand-tracking for computer aided design
Wang, R., Paris, S., & Popović, J. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047269 · Cited by 226
Tracking the articulated motion of two strongly interacting hands
Oikonomidis, I., Kyriazis, N., & Argyros, A. A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247885 · Cited by 225 · cited 1×
“Proofs of concept [19, 18] suggest that extending our approach to deal with these cases should not be too challenging, at least for the model fitter.”
BackgroundTowards 3D hand tracking using a deformable model
Heap, T., & Hogg, D. (2002) · Proceedings of the Second International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1996.557255 · Cited by 220 · cited 1×
“An early example of online (10Hz) tracking with a simplified deformable hand model is [10].”
BackgroundReal-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests
Tang, D., Yu, T.-H., & Kim, T.-K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.400 · Cited by 219 · cited 1×
“[27, 26] extend this work demonstrating more complex poses at 25Hz.”
BackgroundEfficient Hand Pose Estimation from a Single Depth Image
Xu, C., & Cheng, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.429 · Cited by 187 · cited 1×
“[36] estimate the global orientation and location of the hand, regress candidate 21-DoF hand poses, and select the correct pose by minimizing reconstruction error.”
BackgroundView-independent recognition of hand postures
Wu, Y., & Huang, T. S. (2002) · Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662) · doi:10.1109/cvpr.2000.854749 · Cited by 185 · cited 1×
“Discriminative methods [2, 34] used small databases of restricted hand poses, limiting accuracy.”
MethodDynamics based 3D skeletal hand tracking
Melax, S., Keselman, L., & Orsten, S. (2013) · Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games · doi:10.1145/2448196.2448232 · Cited by 172 · cited 2×
“We dub the above function the ‘golden energy’ to distinguish it from energies that do not directly aim to explain the image data such as [16, 23, 28].”
BackgroundMethodConditional regression forests for human pose estimation
Sun, M., Kohli, P., & Shotton, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248079 · Cited by 167 · cited 3×
“Second, while following existing practice [25, 12] in breaking the regression into two ‘layers’ (or stages), we focus the first layer exclusively in predicting quantized global hand rotation.”
BackgroundMethodSelf-citationDecision Jungles: Compact and Rich Models for Classification
Shotton, J., Sharp, T., Kohli, P., Nowozin, S., Winn, J. M., & Criminisi, A. (2013) · neural information processing systems · Cited by 151
Self-citationSmart particle filtering for 3D hand tracking
Bray, M., Koller-Meier, E., & Gool, L. V. (2004) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2004.1301612 · Cited by 116 · cited 1×
“[4] used a more detailed 3D hand model.”
MethodUser-Specific Hand Modeling from Monocular Depth Sequences
Taylor, J., Stebbing, R. V., Ramakrishna, V., Keskin, C., Shotton, J., Izadi, S., Hertzmann, A., & Fitzgibbon, A. W. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.88 · Cited by 92 · cited 1×
“We believe that building an accurate personalized model of each user’s hand [29] can only improve the quality of tracking.”
BackgroundSelf-citationVideo-based hand manipulation capture through composite motion control
Wang, Y., Min, J., Zhang, J., Liu, Y., Xu, F., Dai, Q., & Chai, J. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462000 · Cited by 83 · cited 1×
“[33] show complex hand-object interactions by minimizing a silhouette, color, and edge-based objective using a physics engine, though take minutes per frame.”
MethodDiscriminative Ferns Ensemble for Hand Pose Recognition
Krupka, E., Vinnikov, A., Klein, B., Bar-Hillel, A., Freedman, D., & Stachniak, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.469 · Cited by 30 · cited 3×
“Several classifiers were investigated: decision forests [5], decision jungles [22], and discriminative ferns ensembles (DFE) [14].”
Self-citation
Research works printed but not linked: 5
- 10.5555/1949767.1949885 · doi:10.5555/1949767.1949885
- 10.5555/524467.796047 · doi:10.5555/524467.796047
- 10.5555/2532129.2532141 · doi:10.5555/2532129.2532141
- 10.5555/2354409.2354668 · doi:10.5555/2354409.2354668
- 10.5555/2421731.2421738 · doi:10.5555/2421731.2421738
Cited by
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,815
“Rendered depth maps of parametric models have been used prominently in training leading systems for human pose estimation and hand tracking [42, 43].”
MethodHand Keypoint Detection in Single Images Using Multiview Bootstrapping
Simon, T., Joo, H., Matthews, I., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.494 · Cited by 1,119
“[16] demonstrated practical performance across a large range, but there are still difficult cases such as hand-hand interactions and handobject interactions.”
BackgroundObservation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking
Cao, J., Pang, J., Weng, X., Khirodkar, R., & Kitani, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00934 · Cited by 1,115
Romero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056
Soli: ubiquitous gesture sensing with millimeter wave radar
Lien, J., Gillian, N., Karagozler, M. E., Amihood, P., Schwesig, C., Olson, E., Raja, H., & Poupyrev, I. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925953 · Cited by 957
“Using depth information by employing either multiple cameras or a single depth camera can significantly improve precision [Sharp et al. 2015], but does not scale well to the small form factor required by wearable and mobile devices.”
Background
Metadata
- Authors
- Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., Freedman, D., Kohli, P., Krupka, E., Fitzgibbon, A., & Izadi, S.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Microsoft (United States) (US)
- Microsoft Research Asia (China) (CN)
- Venue
- Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2015-04-18
- Pages
- 3633-3642
- OpenAlex
- W2023633446
- Cited by
- 431
- References
- 35
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 18 · 2023: 25 · 2022: 27 · 2021: 48 · 2020: 52 · 2019: 55 · 2018: 61 · 2017: 52 · 2016: 57 · 2015: 17
- FWCI
- 47.15
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- hand tracking, real-time tracking, depth camera, hand pose estimation, reinitialization, human-computer interaction, mobile device tracking, virtual reality headsets, robust tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.