CoChat
Sign Up
Cited by 1,781Human Pose and Action Recognition3D Shape Modeling and AnalysisFace recognition and analysis

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

Open access
Pavlakos, Georgios, Choutas, Vasileios, Ghorbani, Nima, Bolkart, Timo, Osman, Ahmed A., Tzionas, Dimitrios, Black, Michael J.·2019·2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2019.01123
Open this paper in CoChat — free accountOpen PDF

Abstract

To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2019.01123

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 19

  1. [1][1] Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In OSDI, vol- ume 16, pages 265–283, 2016. 6, 7
  2. [2][2] Ijaz Akhter and Michael J. Black. Pose-conditioned joint angle limits for 3D human pose reconstruction. In CVPR, 2015. 6
  3. [3][3] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2D human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014. 4, 7, 9
  4. [4][4] Luca Ballan, Aparna Taneja, Juergen Gall, Luc Van Gool, and Marc Pollefeys. Motion capture of hands in action using discriminative salient points. In ECCV, 2012. 1, 2
  5. [5][5] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In ECCV, 2016. 4, 10
  6. [11][11] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010. 7
  7. [12][12] Sam Johnson and Mark Everingham. Learning effective hu- man pose estimation from inaccurate annotation. In CVPR, 2011. 7
  8. [13][13] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total cap- ture: A 3D deformation model for tracking faces, hands, and bodies. In CVPR, 2018. 3
  9. [14][14] Tero Karras. Maximizing parallelism in the construction of BVHs, Octrees, and K-d trees. In Proceedings of the Fourth ACM SIGGRAPH / Eurographics Conference on High-Performance Graphics, pages 33–37, 2012. 3
  10. [15][15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 7
  11. [16][16] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics (TOG), 36(6):194, 2017. 1
  12. [18][18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 7
  13. [19][19] Matthew Loper, Naureen Mahmood, and Michael J Black. MoSh: Motion and shape capture from sparse markers. ACM Transactions on Graphics (TOG), 33(6):220, 2014. 6
  14. [20][20] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Recti- fier nonlinearities improve neural network acoustic models. In ICML Workshops, 2013. 6
  15. [21][21] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Ger- ard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. arXiv:1904.03278, 2019. 6 · arXiv
  16. [22][22] Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer, New York, 2nd edition, 2006. 3
  17. [24][24] Paschalis Panteleris, Iason Oikonomidis, and Antonis Argy- ros. Using a single RGB frame for real time 3D hand pose estimation in the wild. In WACV, 2018. 1, 2
  18. [25][25] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pyTorch. In NIPS-W, 2017. 6
  19. [28][28] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics sim- ulation. IJCV, 118(2):172–193, 2016. 1, 2

Cited by

Sorted by citations

Authors
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., & Black, M. J.
Affiliations
  • University of Pennsylvania (US)
  • Max Planck Institute for Intelligent Systems (DE)
  • Max Planck Society (DE)
ORCID
Venue
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2019-06
Pages
10967-10977
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2019.01123
OpenAlex
W2939665722
Cited by
1,781
References
6
Citations by year
2026: 254 · 2025: 357 · 2024: 369 · 2023: 308 · 2022: 189 · 2021: 182 · 2020: 103 · 2019: 19
FWCI
40.18
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
  • arXiv (Cornell University)
Funders
Nvidia
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
hand pose estimation, facial expression estimation, single-image 3D reconstruction, SMPL-X, monocular RGB, gender detection, human action analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review