End-to-End Human Pose and Mesh Reconstruction with Transformers
Open accessAbstract
We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image. Our method uses a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, and outputs 3D joint coordinates and mesh vertices simultaneously. Compared to existing techniques that regress pose and shape parameters, METRO does not rely on any parametric mesh models like SMPL, thus it can be easily extended to other objects such as hands. We further relax the mesh topology and allow the transformer self-attention mechanism to freely attend between any two vertices, making it possible to learn non-local relationships among mesh vertices and joints. With the proposed masked vertex modeling, our method is more robust and effective in handling challenging situations like partial occlusions. METRO generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets. Moreover, we demonstrate the generalizability of METRO to 3D hand reconstruction in the wild, outperforming existing state-of-the-art methods on FreiHAND dataset.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ViT achieves better performance compared to CNN architecture, i.e., ResNet.
“We choose to ViT in this case since it is shown to achieve better performance as compared to CNN architecture, i.e., ResNet [24].”
From Deepfake Detection: Analyzing Model Generalization Across Architectures, Datasets, and Pre-Training Paradigms (2024) · cites this paper as [24]
Transformers can encode implicit knowledge.
“(1) Transformers can encode implicit knowledge [32].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [32]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Deep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894 · see citing works
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · see citing works
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734 · cited 8× · see citing works
“Among the new references, HKMR [11] regresses SMPL parameters by leveraging a pre-specified hierarchical kinematic structure that consists of a root chain and five child chains corresponding to 5 end effectors (head, left/right arms, left/right legs).”
MethodInfluentialHuman3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 7× · see citing works
“Mean-Per-Joint-Position-Error (MPJPE) [21] is a metric for evaluating human 3D pose [22, 24, 8].”
BackgroundInfluentialGeneralized procrustes analysis
Gower, J. C. (1975) · Psychometrika · doi:10.1007/bf02291478 · Cited by 3,267 · see citing works
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.471 · Cited by 2,699 · see citing works
“Cloze Procedure”: A New Tool for Measuring Readability
Taylor, W. L. (1953) · Journalism Quarterly · doi:10.1177/107769905303000401 · Cited by 2,332 · see citing works
End-to-End Recovery of Human Shape and Pose
Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00744 · Cited by 1,899 · see citing works
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01123 · Cited by 1,781 · see citing works
Mining actionlet ensemble for action recognition with depth cameras
Wang, J., Liu, Z., Wu, Y. N., & Yuan, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247813 · Cited by 1,556 · see citing works
Self-citationA Decomposable Attention Model for Natural Language Inference
Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,453 · see citing works
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
Saito, S., Huang, Z., Natsume, R., Morishima, S., Li, H., & Kanazawa, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00239 · Cited by 1,228 · see citing works
Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision
Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., & Theobalt, C. (2017) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00064 · Cited by 1,149 · see citing works
VIBE: Video Inference for Human Body Pose and Shape Estimation
Kocabas, M., Athanasiou, N., & Black, M. J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00530 · Cited by 1,065 · see citing works
Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera
von Marcard, T., Henschel, R. D., Black, M. J., Rosenhahn, B., & Pons‐Moll, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_37 · Cited by 1,062 · see citing works
Romero (2017) · doi:10.1145/3130800.3130883 · Cited by 1,056 · see citing works
Neural Relation Extraction with Selective Attention over Instances
Lin, Y., Shen, S., Liu, Z., Luan, H., & Sun, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1200 · Cited by 1,052 · see citing works
Self-citationLearning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop
Kolotouros, N., Pavlakos, G., Black, M. J., & Daniilidis, K. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00234 · Cited by 1,046 · see citing works
(2020) · International Conference on Machine Learning · Cited by 1,012 · see citing works
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization
Saito, S., Simon, T., Saragih, J., & Joo, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00016 · Cited by 787 · see citing works
Learning to Estimate 3D Human Pose and Shape from a Single Color Image
Pavlakos, G., Zhu, L., Zhou, X., & Daniilidis, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00055 · Cited by 602 · see citing works
Generating 3D Faces Using Convolutional Mesh Autoencoders
Ranjan, A., Bolkart, T., Sanyal, S., & Black, M. J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_43 · Cited by 570 · see citing works
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction
Kolotouros, N., Pavlakos, G., & Daniilidis, K. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00463 · Cited by 556 · see citing works
Unite the People: Closing the Loop Between 3D and 2D Human Representations
Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M. J., & Gehler, P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.500 · Cited by 533 · see citing works
Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation
Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., & Schiele, B. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00062 · Cited by 498 · see citing works
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images
Zimmermann, C., Ceylan, D., Yang, J., Russell, B. C., Argus, M. J., & Brox, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00090 · Cited by 462 · see citing works
BodyNet: Volumetric Inference of 3D Human Body Shapes
Varol, G., Ceylan, D., Russell, B. C., Yang, J., Yumer, E., Laptev, I., & Schmid, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_2 · Cited by 420 · see citing works
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB
Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Sridhar, S., Pons‐Moll, G., & Theobalt, C. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00024 · Cited by 395 · see citing works
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose
Choi, H., Moon, G., & Lee, K. M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58571-6_45 · Cited by 394 · see citing works
Moon, G., & Lee, K. M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58571-6_44 · Cited by 385 · see citing works
Estimating human shape and pose from a single image
Guan, P., Weiss, A., Bălan, A. O., & Black, M. J. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459300 · Cited by 377 · see citing works
Deep High-Resolution Representation Learning for Visual RecognitionPreprint
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y. S., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07919 · Cited by 350 · see citing works
Huang, Y., Kaufmann, M., Aksan, E., Black, M. J., Hilliges, O., & Pons-Moll, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275108 · Cited by 342 · see citing works
3D-CODED: 3D Correspondences by Deep Deformation
Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., & Aubry, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_15 · Cited by 305 · see citing works
Inverse Kinematics Techniques in Computer Graphics: A Survey
Aristidou, A., Lasenby, J., Chrysanthou, Y., & Shamir, A. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13310 · Cited by 266 · see citing works
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior
Zhou, X., Zhu, M., Pavlakos, G., Leonardos, S., Derpanis, K. G., & Daniilidis, K. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2816031 · Cited by 223 · see citing works
Direct Prediction of 3D Body Poses from Motion Compensated Sequences
Tekin, B., Rozantsev, A., Lepetit, V., & Fua, P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.113 · Cited by 216 · see citing works
Self-supervised Learning of Motion CapturePreprint
Tung, H.-Y. F., Tung, H.-W., Yumer, E., & Fragkiadaki, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01337 · Cited by 198 · see citing works
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human DigitizationPreprint
Saito, S., Huang, Z., Natsume, R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05172 · Cited by 144 · see citing works
Self-supervised learning of motion capture
Tung, H.-Y. F., Tung, H.-W., Yumer, E., & Fragkiadaki, K. (2017) · neural information processing systems · Cited by 123 · see citing works
BodyNet: Volumetric Inference of 3D Human Body ShapesPreprint
Varol, G., Ceylan, D., Russell, B. C., Yang, J., Yumer, E., Ivan Laptev, A., & Schmid, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.04875 · Cited by 63 · see citing works
Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild
Rong, Y., Liu, Z., Li, C., Cao, K., & Loy, C. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00544 · Cited by 61 · see citing works
Self-citation3D Scene Reconstruction With Multi-Layer Depth and Epipolar Transformers
Shin, D., Ren, Z., Sudderth, E., & Fowlkes, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00226 · Cited by 57 · see citing works
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human DigitizationPreprint
Saito, S., Simon, T., Saragih, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00452 · Cited by 42 · see citing works
Delving Deep Into Hybrid Annotations for 3D Human Recovery in the WildPreprint
Rong, Y., Liu, Z., Li, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06442 · Cited by 5 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion
Black, M. J., Patel, P., Tesch, J., & Yang, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00843 · Cited by 148 · see citing works
Khan, S. A., & Dang-Nguyen, D.-T. (2024) · IEEE Access · doi:10.1109/access.2023.3348450 · Cited by 53 · see citing works
Wang, P., Zheng, W., Chen, T., & Wang, Z. (2022) · arXiv · doi:10.48550/arxiv.2203.05962 · Cited by 80 · see citing works
“…al., 2021; Zheng et al., 2021a; Sun et al., 2020), segmentation (Chen et al., 2021a; Wang et al., 2021), image generation (Parmar et al., 2018; Jiang et al., 2021a), video processing (Zhou et al., 2018; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021; Lin et al., 2021). relate to.”
BackgroundEgo-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02557-6 · Cited by 3 · see citing works
- Authors
- Lin, K., Wang, L., & Liu, Z.
- Affiliations
- Microsoft (United States) (US)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 1954-1963
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3175199633
- Cited by
- 740
- References
- 58
- Citations by year
- 2026: 56 · 2025: 123 · 2024: 192 · 2023: 210 · 2022: 117 · 2021: 42
- FWCI
- 103.07
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- human pose estimation, 3D mesh reconstruction, Transformer encoder, metro, single-image 3D reconstruction, self-attention, partial occlusion, 3D hand reconstruction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
