The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB
Abstract
We address the highly challenging problem of real-time 3D hand tracking based on a monocular RGB-only sequence. Our tracking method combines a convolutional neural network with a kinematic 3D hand model, such that it generalizes well to unseen data, is robust to occlusions and varying camera viewpoints, and leads to anatomically plausible as well as temporally smooth hand motions. For training our CNN we propose a novel approach for the synthetic generation of training data that is based on a geometrically consistent image-to-image translation network. To be more specific, we use a neural network that translates synthetic images to "real" images, such that the so-generated images follow the same statistical distribution as real-world hand images. For training this translation network we combine an adversarial loss and a cycle-consistency loss with a geometric consistency loss in order to preserve geometric properties (such as hand pose) during translation. We demonstrate that our hand tracking system outperforms the current state-of-the-art on challenging RGB-only footage.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Adversarial Discriminative Domain Adaptation
Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.316 · Cited by 5,154
Unsupervised Domain Adaptation by Backpropagation
Ganin, Y., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.7495 · Cited by 2,521
A Generalized Solution of the Orthogonal Procrustes Problem
Schönemann, P. H. (1966) · Psychometrika · doi:10.1007/bf02289451 · Cited by 2,040
Learning from Simulated and Unsupervised Images through Adversarial Training
Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.241 · Cited by 1,814
Hand Keypoint Detection in Single Images Using Multiview Bootstrapping
Simon, T., Joo, H., Matthews, I., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.494 · Cited by 1,119
VNect: real-time 3D human pose estimation with a single RGB camera
Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., Xu, W., Casas, D., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073596 · Cited by 1,113
Self-citationLearning from Synthetic Humans
Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M. J., Laptev, I., & Schmid, C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.492 · Cited by 908
Photographic Image Synthesis with Cascaded Refinement Networks
Chen, Q., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.168 · Cited by 887
Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation
Johnson, S., & Everingham, M. (2010) · Procedings of the British Machine Vision Conference 2010 · doi:10.5244/c.24.12 · Cited by 819
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 761
Real-time hand-tracking with a color glove
Wang, R. Y., & Popović, J. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531369 · Cited by 609
Scribbler: Controlling Deep Image Synthesis with Sketch and Color
Sangkloy, P., Lu, J., Fang, C., Yu, F., & Hays, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.723 · Cited by 497
Realtime and Robust Hand Tracking from Depth
Qian, C., Sun, X., Wei, Y., Tang, X., & Sun, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.145 · Cited by 470
Accurate, Robust, and Flexible Real-time Hand Tracking
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., et al. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702179 · Cited by 431
1 € filter: a simple speed-based low-pass filter for noisy input in interactive systems
Casiez, G., Roussel, N., & Vogel, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208639 · Cited by 422
Model-based hand tracking using a hierarchical Bayesian filter
Stenger, B., Thayananthan, A., Torr, P. H. S., & Cipolla, R. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.189 · Cited by 415
Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture
Tang, D., Chang, H. J., Tejani, A., & Kim, T.-K. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.490 · Cited by 398
Taylor, J., Bordeaux, L., Cashman, T., Corish, B., Keskin, C., Sharp, T., Soto, E., Sweeney, D., Valentin, J., Luff, B., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925965 · Cited by 294
Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs
Ge, L., Liang, H., Yuan, J., & Thalmann, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.391 · Cited by 289
Real Time Hand Pose Estimation Using Depth Sensors
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2013) · Consumer Depth Cameras for Computer Vision · doi:10.1007/978-1-4471-4640-7_7 · Cited by 288
Motion Capture of Hands in Action Using Discriminative Salient Points
Ballan, L., Taneja, A., Gall, J., Van Gool, L., & Pollefeys, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_46 · Cited by 280
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0895-4 · Cited by 277
Oikonomidis, I., Kyriazis, N., & Argyros, A. A. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126483 · Cited by 274
Training a Feedback Loop for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.379 · Cited by 272
Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data
Sridhar, S., Oulasvirta, A., & Theobalt, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.305 · Cited by 247
Self-citationReal-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.131 · Cited by 242
Self-citationReal-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
Sridhar, S., Mueller, F., Zollhöfer, M., Casas, D., Oulasvirta, A., & Theobalt, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_19 · Cited by 240
Self-citationUser-defined gestures for augmented reality
Piumsomboon, T., Clark, A., Billinghurst, M., & Cockburn, A. (2013) · CHI '13 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/2468356.2468527 · Cited by 239
Fast and robust hand tracking using detection-guided optimization
Sridhar, S., Mueller, F., Oulasvirta, A., & Theobalt, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298941 · Cited by 228
Self-citation6D hands: markerless hand-tracking for computer aided design
Wang, R., Paris, S., & Popović, J. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047269 · Cited by 226
Towards 3D hand tracking using a deformable model
Heap, T., & Hogg, D. (2002) · Proceedings of the Second International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1996.557255 · Cited by 220
Robust Articulated‐ICP for Real‐Time Hand Tracking
Tagliasacchi, A., Schröder, M., Tkach, A., Bouaziz, S., Botsch, M., & Pauly, M. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12700 · Cited by 216
Real time hand pose estimation using depth sensors
Keskin, C., Kirac, F., Kara, Y. E., & Akarun, L. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130391 · Cited by 190
Efficient Hand Pose Estimation from a Single Depth Image
Xu, C., & Cheng, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.429 · Cited by 187
Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation
Wan, C., Probst, T., Gool, L. V., & Yao, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.132 · Cited by 167
Markussen, A., Jakobsen, M. R., & Hornbæk, K. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556964 · Cited by 163
Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation
Ye, Q., Yuan, S., & Kim, T.-K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_21 · Cited by 163
DeepHand: Robust Hand Pose Estimation by Completing a Matrix Imputed with Deep Features
Sinha, A., Choi, C., & Ramani, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.450 · Cited by 159
Rhodin, H., Richardt, C., Casas, D., Insafutdinov, E., Shafiei, M., Seidel, H., Schiele, B., & Theobalt, C. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980235 · Cited by 156
Self-citationTracking a hand manipulating an object
Hamer, H., Schindler, K., Koller-Meier, E., & Gool, L. V. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459282 · Cited by 149
Understanding Everyday Hands in Action from RGB-D Images
Rogez, G., Supančič, J. S., & Ramanan, D. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.443 · Cited by 147
Opening the Black Box: Hierarchical Sampling Optimization for Estimating Human Hand Pose
Tang, D., Taylor, J., Kohli, P., Keskin, C., Kim, T., & Shotton, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.380 · Cited by 142
Hands in action: real-time 3D reconstruction of hands in interaction with objects
Romero, J., Kjellström, H., & Kragić, D. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509753 · Cited by 124
Synthetic to Real Adaptation with Generative Correlation Alignment Networks
Peng, X., & Saenko, K. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00219 · Cited by 116
Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry
Sridhar, S., Feit, A. M., Theobalt, C., & Oulasvirta, A. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702136 · Cited by 113
Self-citation3D Hand Pose Tracking and Estimation Using Stereo MatchingPreprint
Zhang, J., Jiao, J., Chen, M., Qu, L., Xu, X., & Yang, Q. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07214 · Cited by 108
Multithreaded Hybrid Feature Tracking for Markerless Augmented Reality
Lee, T., & Höllerer, T. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2008.190 · Cited by 93
Efficiently Creating 3D Training Data for Fine Hand Pose Estimation
Oberweger, M., Riegler, G., Wohlhart, P., & Lepetit, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.536 · Cited by 88
Large-scale multiview 3D hand pose dataset
Gomez‐Donoso, F., Orts‐Escolano, S., & Cazorla, M. (2018) · Image and Vision Computing · doi:10.1016/j.imavis.2018.12.001 · Cited by 76
Hand Pose Estimation from Local Surface Normals
Wan, C., Yao, A., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_34 · Cited by 72
3D Object Reconstruction from Hand-Object Interactions
Tzionas, D., & Gall, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.90 · Cited by 64
Robust Hand Pose Estimation during the Interaction with an Unknown Object
Choi, C., Yoon, S. H., Chen, C.-N., & Karthik Ramani, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.339 · Cited by 63
Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model
Sridhar, S., Rhodin, H., Seidel, H., Oulasvirta, A., & Theobalt, C. (2014) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2014.37 · Cited by 58
Self-citationBack to RGB: 3D Tracking of Hands and Hand-Object Interactions Based on Short-Baseline Stereo
Panteleris, P., & Argyros, A. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.74 · Cited by 57
Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model
Sridhar, S., Rhodin, H., Seidel, H., Oulasvirta, A., & Theobalt, C. (2014) · MPG.PuRe (Max Planck Society) · Cited by 47
Self-citationLearning Hand Articulations by Hallucinating Heat Distribution
Choi, C., Kim, S., & Ramani, K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.337 · Cited by 39
Multi-touchless: Real-time fingertip detection and tracking using geodesic maxima
Krejov, P., & Bowden, R. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553778 · Cited by 38
Cited by
Learning Joint Reconstruction of Hands and Manipulated Objects
Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01208 · Cited by 513
3D Hand Shape and Pose Estimation From a Single RGB Image
Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., & Yuan, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01109 · Cited by 458
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images
Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M. J., & Brox, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00090 · Cited by 445
HOnnotate: A Method for 3D Annotation of Hand and Object Poses
Hampali, S., Rad, M., Oberweger, M., & Lepetit, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00326 · Cited by 422
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P., Breuel, T., Gall, J., & Kautz, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316
Metadata
- Authors
- Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., & Theobalt, C.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 49-59
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2897765997
- Cited by
- 521
- References
- 64
- FWCI
- 61.62
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.