The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
VIBE: Video Inference for Human Body Pose and Shape Estimation
Open accessAbstract
Human motion is fundamental to understanding behavior. Despite progress on single-image 3D pose and shape estimation, existing video-based state-of-the-art methods fail to produce accurate and natural motion sequences due to a lack of ground-truth 3D motion data for training. To address this problem, we propose "Video Inference for Body Pose and Shape Estimation'' (VIBE), which makes use of an existing large-scale motion capture dataset (AMASS) together with unpaired, in-the-wild, 2D keypoint annotations. Our key novelty is an adversarial learning framework that leverages AMASS to discriminate between real human motions and those produced by our temporal pose and shape regression networks. We define a novel temporal network architecture with a self-attention mechanism and show that adversarial training, at the sequence level, produces kinematically plausible motion sequences without in-the-wild ground-truth 3D labels. We perform extensive experimentation to analyze the importance of motion and demonstrate the effectiveness of VIBE on challenging 3D pose estimation datasets, achieving state-of-the-art performance. Code and pretrained models are available at https://github.com/mkocabas/VIBE
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 1×
“We use a ResNet-50 network [22] as an image encoder pretrained on single frame pose and shape estimation task [29, 36] that outputs f i ∈ R 2048 .”
MethodQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · cited 1×
“PennAction [64] and PoseTrack [3] are the only ground-truth 2D video datasets we use, while InstaVariety [30] and Kinetics-400 [13] are pseudo ground-truth datasets annotated using a 2D keypoint detector [12, 34].”
MethodImproved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
(2017) · International Conference on Machine Learning · Cited by 6,382
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536
Visual perception of biological motion and a model for its analysis
Johansson, G. (1973) · Perception & Psychophysics · doi:10.3758/bf03212378 · Cited by 4,462
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Self-citationHuman3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653
The Kinetics Human Action Video Dataset
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 2,593
End-to-End Recovery of Human Shape and Pose
Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00744 · Cited by 1,870
Self-citationExpressive Body Capture: 3D Hands, Face, and Body From a Single Image
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01123 · Cited by 1,781
Self-citationKeep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., & Black, M. J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_34 · Cited by 1,498
Self-citationSCAPE: shape completion and animation of people
Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., & Davis, J. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073207 · Cited by 1,489
AMASS: Archive of Motion Capture As Surface Shapes
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00554 · Cited by 1,323
Self-citation3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training
Pavllo, D., Feichtenhofer, C., Grangier, D., & Auli, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00794 · Cited by 1,297
On the Continuity of Rotation Representations in Neural Networks
Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00589 · Cited by 1,255
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
Saito, S., Huang, Z., Natsume, R., Morishima, S., Li, H., & Kanazawa, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00239 · Cited by 1,228
Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision
Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., & Theobalt, C. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00064 · Cited by 1,142
VNect: real-time 3D human pose estimation with a single RGB camera
Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., Xu, W., Casas, D., & Theobalt, C. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073596 · Cited by 1,113
Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera
von Marcard, T., Henschel, R., Black, M. J., Rosenhahn, B., & Pons-Moll, G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_37 · Cited by 1,058
Self-citationDeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.533 · Cited by 1,051
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop
Kolotouros, N., Pavlakos, G., Black, M., & Daniilidis, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00234 · Cited by 1,039
Self-citationRecovering 3D human pose from monocular images
Agarwal, A., & Triggs, B. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.21 · Cited by 727
Learning to Estimate 3D Human Pose and Shape from a Single Color Image
Pavlakos, G., Zhu, L., Zhou, X., & Daniilidis, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00055 · Cited by 607
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction
Kolotouros, N., Pavlakos, G., & Daniilidis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00463 · Cited by 548
Unite the People: Closing the Loop Between 3D and 2D Human Representations
Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M. J., & Gehler, P. V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.500 · Cited by 541
Self-citationLearning 3D Human Dynamics From Video
Kanazawa, A., Zhang, J. Y., Felsen, P., & Malik, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00576 · Cited by 531
Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation
Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., & Schiele, B. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00062 · Cited by 498
PoseTrack: A Benchmark for Human Pose Estimation and Tracking
Andriluka, M., Iqbal, U., Insafutdinov, E., Pishchulin, L., Milan, A., Gall, J., & Schiele, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00542 · Cited by 486
Unsupervised Image-to-Image Translation Networks
Liu, M.-Y., Breuel, T. M., & Kautz, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00848 · Cited by 477
Model-based vision: a program to see a walking person
Hogg, D. (1983) · Image and Vision Computing · doi:10.1016/0262-8856(83)90003-3 · Cited by 467
3D People Tracking with Gaussian Process Dynamical Models
Urtasun, R., Fleet, D. J., & Fua, P. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.15 · Cited by 439
MODEC: Multimodal Decomposable Models for Human Pose Estimation
Sapp, B., & Taskar, B. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.471 · Cited by 423
SCAPE: shape completion and animation of people
Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., & Davis, J. (2005) · ACM SIGGRAPH 2005 Papers · doi:10.1145/1186822.1073207 · Cited by 410
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB
Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Sridhar, S., Pons-Moll, G., & Theobalt, C. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00024 · Cited by 391
BodyNet: Volumetric Inference of 3D Human Body Shapes
Varol, G., Ceylan, D., Russell, B., Yang, J., Yumer, E., Laptev, I., & Schmid, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_2 · Cited by 385
From Actemes to Action: A Strongly-Supervised Representation for Detailed Action Understanding
Zhang, W., Zhu, M., & Derpanis, K. G. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.280 · Cited by 361
MoSh: motion and shape capture from sparse markers
Loper, M., Mahmood, N., & Black, M. J. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661273 · Cited by 344
Self-citationSelf-supervised Learning of Motion CapturePreprint
Tung, H.-Y. F., Tung, H.-W., Yumer, E., & Fragkiadaki, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01337 · Cited by 337
HP-GAN: Probabilistic 3D Human Motion Prediction via GAN
Barsoum, E., Kender, J., & Liu, Z. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00191 · Cited by 322
MultiPoseNet: Fast Multi-Person Pose Estimation Using Pose Residual Network
Kocabas, M., Karagoz, S., & Akbas, E. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_26 · Cited by 314
Self-citationSelf-Supervised Learning of 3D Human Pose Using Multi-View Geometry
Kocabas, M., Karagoz, S., & Akbas, E. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00117 · Cited by 293
Self-citationAdversarial Geometry-Aware Human Motion Prediction
Gui, L.-Y., Wang, Y.-X., Liang, X., & Moura, J. M. F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_48 · Cited by 276
Detect-and-Track: Efficient Pose Estimation in Videos
Girdhar, R., Gkioxari, G., Torresani, L., Paluri, M., & Tran, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00044 · Cited by 254
Learning 3D Human Pose from Structure and Motion
Dabral, R., Mundhada, A., Kusupati, U., Afaque, S., Sharma, A., & Jain, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01240-3_41 · Cited by 227
HoloPose: Holistic 3D Human Reconstruction In-The-Wild
Guler, R. A., & Kokkinos, I. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01114 · Cited by 221
Towards Accurate Marker-Less Human Shape and Pose Estimation over Time
Huang, Y., Bogo, F., Lassner, C., Kanazawa, A., Gehler, P. V., Romero, J., Akhter, I., & Black, M. J. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00055 · Cited by 207
Self-citationStructured Prediction Helps 3D Human Motion Modelling
Aksan, E., Kaufmann, M., & Hilliges, O. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00724 · Cited by 193
Inferring 3D structure with a statistical image-based shape model
Grauman, Shakhnarovich, & Darrell (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238408 · Cited by 192
Human Mesh Recovery From Monocular Images via a Skeleton-Disentangled Representation
Sun, Y., Ye, Y., Liu, W., Gao, W., Fu, Y., & Mei, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00545 · Cited by 189
Combined discriminative and generative articulated pose and non-rigid shape estimation
Sigal, L., Bălan, A. O., & Black, M. J. (2007) · Cited by 186
Self-citationThe Naked Truth: Estimating Body Shape Under Clothing
Bălan, A. O., & Black, M. J. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88688-4_2 · Cited by 176
Self-citationTolstikhin, I., Bousquet, O., Gelly, S., & Schölkopf, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01558 · Cited by 176
Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets
Yang, Z. C., Chen, W., Wang, F., & Xu, B. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1122 · Cited by 165
Baziotis, C., Nikolaos, A., Chronopoulou, A., Kolovou, A., Paraskevopoulos, G., Ellinas, N., Narayanan, S., & Potamianos, A. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1037 · Cited by 142
Adversarial Neural Machine TranslationPreprint
Wu, L., Xia, Y., Li, Z., Tian, F., Qin, T., Lai, J., & Liu, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06933 · Cited by 129
Sim2real transfer learning for 3D human pose estimation: motion to the rescue
Doersch, C., & Zisserman, A. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 106
From Variational to Deterministic Autoencoders
Ghosh, P. S., Sajjadi, M. S. M., Vergari, A., Black, M. J., & Schölkopf, B. (2020) · International Conference on Learning Representations · Cited by 72
Self-citationAdversarial Neural Machine Translation
Wu, L., Xia, Y., Tian, F., Li, Z., Qin, T., Lai, J., & Liu, T. (2018) · Asian Conference on Machine Learning · Cited by 57
Sim2real transfer learning for 3D pose estimation: motion to the rescuePreprint
Doersch, C., & Zisserman, A. (2019) · arXiv (Cornell University) · Cited by 15
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Cited by
End-to-End Human Pose and Mesh Reconstruction with Transformers
Lin, K., Wang, L., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00199 · Cited by 740
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
Yuan, Y., Weng, X., Ou, Y., & Kitani, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00967 · Cited by 631
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Li, R., Yang, S., Ross, D. A., & Kanazawa, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01315 · Cited by 553
“We use the 2D mean per joint position error MPJPE2D, commonly used for 3D reconstruction quality measurement [46, 39, 65]) to evaluate the consistency between the predicted 2D keypoints and the reconstructed 3D keypoints along with the estimated camera parameters.”
MethodAction-Conditioned 3D Human Motion Synthesis with Transformer VAE
Petrovich, M., Black, M. J., & Varol, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01080 · Cited by 493
Self-citationMotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
Metadata
- Authors
- Kocabas, M., Athanasiou, N., & Black, M. J.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 5252-5262
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2995321487
- Cited by
- 1,031
- References
- 71
- FWCI
- 47.61
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.