CoChat
Sign Up
Cited by 38Human Motion and AnimationHuman Pose and Action RecognitionGenerative Adversarial Networks and Image Synthesis

Bidirectional Transformer GAN for Long-term Human Motion Prediction

Open access
Zhao, Mengyi, Tang, Hao, Xie, Pan, Dai, Shuling, Sebe, Nicu, Wang, Wei·2023·ACM Transactions on Multimedia Computing, Communications, and Applications
DOI 10.1145/3579359
Open this paper in CoChat — free accountOpen PDF

Abstract

The mainstream motion prediction methods usually focus on short-term prediction, and their predicted long-term motions often fall into an average pose, i.e., the freezing forecasting problem [ 27 ]. To mitigate this problem, we propose a novel Bidirectional Transformer-based Generative Adversarial Network (BiTGAN) for long-term human motion prediction. The bidirectional setup leads to consistent and smooth generation in both forward and backward directions. Besides, to make full use of the history motions, we split them into two parts. The first part is fed to the Transformer encoder in our BiTGAN while the second part is used as the decoder input. This strategy can alleviate the exposure problem [ 37 ]. Additionally, to better maintain both the local (i.e., frame-level pose) and global (i.e., video-level semantic) similarities between the predicted motion sequence and the real one, the soft dynamic time warping (Soft-DTW) loss is introduced into the generator. Finally, we utilize a dual-discriminator to distinguish the predicted sequence at both frame and sequence levels. Extensive experiments on the public Human3.6M dataset demonstrate that our proposed BiTGAN achieves state-of-the-art performance on long-term (4 s ) human motion prediction, and reduces the average error of all actions by 4%.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3579359

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. Martin Arjovsky and Léon Bottou. 2017. Towards principled methods for training generative adversarial networks. In Proceeding of the 5th International Conference on Learning Representations.
  2. Yujun Cai, Lin Huang, Yiwei Wang, Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Xu Yang, Yiheng Zhu, Xiaohui Shen, Ding Liu, Jing Liu, and Nadia Magnenat-Thalmann. 2020. Learning progressive joint propagation for human motion prediction. In Proceedings of the European Conference on Computer Vision. Springer, 226-242.
  3. Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. 2020. Long-term human motion prediction with scene context. In Proceedings of the European Conference on Computer Vision. Springer, 387-404.
  4. Marco Cuturi and Mathieu Blondel. 2017. Soft-dtw: A differentiable loss function for time-series. In Proceedings of the International Conference on Machine Learning. PMLR, 894-903.
  5. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR abs/1901.02860 (2019). · arXiv
  6. Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. 2021. MSR-GCN: Multi-scale residual graph convolution networks for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11467-11476.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171-4186.
  8. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceeding of the 9th International Con- ference on Learning Representations.
  9. Yinglin Duan, Tianyang Shi, Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan. 2021. Single-Shot Motion Completion with Transformer. CoRR abs/2103.00776 (2021). · arXiv
  10. Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision. 4346-4354.
  11. Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G. Ororbia. 2019. A neural temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12116-12125.
  12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https://arxiv.org/abs/1412.6980. · arXiv
  13. Jogendra Nath Kundu, Maharshi Gor, and R. Venkatesh Babu. 2019. Bihmp-gan: Bidirectional 3D human motion pre- diction gan. In Proceedings of the AAAI Conference on Artificial Intelligence. 8553-8560.
  14. Jiachen Li, Fan Yang, Hengbo Ma, Srikanth Malla, Masayoshi Tomizuka, and Chiho Choi. 2021. Rain: Reinforced hybrid attention inference network for motion forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16096-16106.
  15. Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li. 2020. Learning to generate diverse dance motions with transformer. CoRR abs/2008.08171 (2020). · arXiv
  16. Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. 2021. Learn to dance with AIST++: Music conditioned 3D dance generation. arXiv:2101.08779. Retrieved from https://arxiv.org/abs/2101.08779. · arXiv
  17. Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. 2021. Multimodal motion prediction with stacked transformers. CoRR abs/2103.11624 (2021). · arXiv
  18. Zhenguang Liu, Shuang Wu, Shuyuan Jin, Qi Liu, Shijian Lu, Roger Zimmermann, and Li Cheng. 2019. Towards natural and accurate future motion prediction of humans and animals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10004-10012.
  19. Wei Mao, Miaomiao Liu, and Mathieu Salzmann. 2020. History repeats itself: Human motion prediction via motion attention. In Proceedings of the European Conference on Computer Vision. Springer, 474-489.
  20. Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. 2019. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9489-9497.
  21. Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recur- rent neural networks. In Proceeding of the 4th International Conference on Learning Representations.
  22. Theodoros Sofianos, Alessio Sampieri, Luca Franco, and Fabio Galasso. 2021. Space-time-separable graph convolu- tional network for pose forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11209- 11218.
  23. Hao Tang, Song Bai, Li Zhang, Philip H. S. Torr, and Nicu Sebe. 2020. Xinggan for person image generation. In Pro- ceedings of the European Conference on Computer Vision. Springer, 717-734.
  24. Hao Tang, Dan Xu, Wei Wang, Yan Yan, and Nicu Sebe. 2018. Dual generator generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the Asian Conference on Computer Vision. Springer, 3-21.
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceeding of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. 5998-6008.
  26. Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, and Gerard Pons-Moll. 2018. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European Conference on Computer Vision. 601-617.
  27. Borui Wang, Ehsan Adeli, Hsu-kuang Chiu, De-An Huang, and Juan Carlos Niebles. 2019. Imitation learning for human pose prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7124-7133.

Cited by

Sorted by citations

Related

Authors
Zhao, M., Tang, H., Xie, P., Dai, S., Sebe, N., & Wang, W.
Affiliations
  • Beihang University (CN)
  • ETH Zurich (CH)
  • University of Trento (IT)
ORCID
Venue
ACM Transactions on Multimedia Computing, Communications, and Applications
Publisher
Association for Computing Machinery (ACM)
Published
2023-04-15
Volume
19
Issue
5
Pages
1-19
ISSN
  • 1551-6857 (ISSN-L)
  • 1551-6865
DOI
10.1145/3579359
OpenAlex
W4315491145
Cited by
38
References
22
Citations by year
2026: 10 · 2025: 11 · 2024: 13 · 2023: 4
FWCI
4.08
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Transactions on Multimedia Computing Communications and Applications (publishedVersion)
  • Iris (University of Trento) (submittedVersion, other-oa)
Funders
National Key Research & Development Program of China (2018AAA0102902)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Human Motion and Animation
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
bidirectional transformer, generative adversarial networks, exposure bias, dual discriminator, Human3.6M

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review