The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Cross-Modal Deep Variational Hand Pose Estimation
Open accessAbstract
The human hand moves in complex and high-dimensional ways, making estimation of 3D hand pose configurations from images alone a challenging task. In this work we propose a method to learn a statistical hand model represented by a cross-modal trained latent space via a generative deep neural network. We derive an objective function from the variational lower bound of the VAE framework and jointly optimize the resulting cross-modal KL-divergence and the posterior reconstruction objective, naturally admitting a training regime that leads to a coherent latent space across multiple modalities such as RGB images, 2D keypoint detections or 3D hand configurations. Additionally, it grants a straightforward way of using semi-supervision. This latent space can be directly used to estimate 3D hand poses from RGB images, outperforming the state-of-the art in different settings. Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods. Finally, the model is fully generative and can synthesize consistent pairs of hand configurations across modalities. We evaluate our method on both RGB and depth datasets and analyze the latent space qualitatively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
(2011) · Cited by 2,752
A Simple Yet Effective Baseline for 3d Human Pose Estimation
Martinez, J., Hossain, R., Romero, J., & Little, J. J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.288 · Cited by 1,462
Postural Hand Synergies for Tool Use
Santello, M., Flanders, M., & Soechting, J. F. (1998) · The Journal of Neuroscience · doi:10.1523/jneurosci.18-23-10105.1998 · Cited by 1,215
Efficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.101 · Cited by 874
Vision-based hand pose estimation: A review
Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., & Twombly, X. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.012 · Cited by 866
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 761
Realtime and Robust Hand Tracking from Depth
Qian, C., Sun, X., Wei, Y., Tang, X., & Sun, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.145 · Cited by 470
Accurate, Robust, and Flexible Real-time Hand Tracking
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., et al. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702179 · Cited by 431
Sun, X., Wei, Y., Liang, S., Tang, X., & Sun, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298683 · Cited by 414
Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture
Tang, D., Chang, H. J., Tejani, A., & Kim, T.-K. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.490 · Cited by 398
Hands Deep in Deep Learning for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06807 · Cited by 298
Taylor, J., Bordeaux, L., Cashman, T., Corish, B., Keskin, C., Sharp, T., Soto, E., Sweeney, D., Valentin, J., Luff, B., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925965 · Cited by 294
Ge, L., Liang, H., Yuan, J., & Thalmann, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.602 · Cited by 291
Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs
Ge, L., Liang, H., Yuan, J., & Thalmann, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.391 · Cited by 289
Real Time Hand Pose Estimation Using Depth Sensors
Keskin, C., Kıraç, F., Kara, Y. E., & Akarun, L. (2013) · Consumer Depth Cameras for Computer Vision · doi:10.1007/978-1-4471-4640-7_7 · Cited by 288
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., & Gall, J. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0895-4 · Cited by 277
DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation
Oberweger, M., & Lepetit, V. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.75 · Cited by 270
Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data
Sridhar, S., Oulasvirta, A., & Theobalt, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.305 · Cited by 247
Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.131 · Cited by 242
Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests
Tang, D., Yu, T.-H., & Kim, T.-K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.400 · Cited by 219
Robust Articulated‐ICP for Real‐Time Hand Tracking
Tagliasacchi, A., Schröder, M., Tkach, A., Bouaziz, S., Botsch, M., & Pauly, M. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12700 · Cited by 216
Analysis of the synergies underlying complex hand manipulation
Todorov, E., & Ghahramani, Z. (2005) · The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2004.1404285 · Cited by 178
Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation
Wan, C., Probst, T., Gool, L. V., & Yao, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.132 · Cited by 167
Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation
Ye, Q., Yuan, S., & Kim, T.-K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_21 · Cited by 163
DeepHand: Robust Hand Pose Estimation by Completing a Matrix Imputed with Deep Features
Sinha, A., Choi, C., & Ramani, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.450 · Cited by 159
Opening the Black Box: Hierarchical Sampling Optimization for Estimating Human Hand Pose
Tang, D., Taylor, J., Kohli, P., Keskin, C., Kim, T., & Shotton, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.380 · Cited by 142
Aytar, Y., Castrejón, L., Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2753232 · Cited by 126
A simple yet effective baseline for 3d human pose estimationPreprint
Martínez, J., Hossain, R., Romero, J., & Little, J. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.03098 · Cited by 121
3D Hand Pose Tracking and Estimation Using Stereo MatchingPreprint
Zhang, J., Jiao, J., Chen, M., Qu, L., Xu, X., & Yang, Q. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07214 · Cited by 108
Cross-Modal Deep Variational Hashing
Liong, V. E., Lu, J., Tan, Y., & Zhou, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.439 · Cited by 100
Hand Pose Estimation from Local Surface Normals
Wan, C., Yao, A., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_34 · Cited by 72
A Collaborative Filtering Approach to Real-Time Hand Pose Estimation
Choi, C., Sinha, A., Choi, J. H., Jang, S., & Ramani, K. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.269 · Cited by 49
Real-time hand tracking using synergistic inverse kinematics
Schröder, M., Maycock, J., Ritter, H. J., & Botsch, M. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907660 · Cited by 36
Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose EstimationPreprint
Wan, C., Probst, T., Gool, L. V., & Yao, A. (2017) · arXiv (Cornell University) · doi:10.3929/ethz-b-000211573 · Cited by 23
Cited by
Sign Language Recognition: A Deep Survey
Rastgoo, R., Kiani, K., & Escalera, S. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113794 · Cited by 575
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks
Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T.-J., Yuan, J., & Thalmann, N. M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00236 · Cited by 519
Learning Joint Reconstruction of Hands and Manipulated Objects
Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01208 · Cited by 513
3D Hand Shape and Pose Estimation From a Single RGB Image
Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., & Yuan, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01109 · Cited by 458
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Iqbal, U., Molchanov, P., Breuel, T., Gall, J., & Kautz, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_8 · Cited by 316
Metadata
- Authors
- Spurr, A., Song, J., Park, S., & Hilliges, O.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 89-98
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2795065101
- Cited by
- 281
- References
- 39
- FWCI
- 33.45
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.