CoChat
Sign Up
PreprintCited by 44Human Pose and Action RecognitionHand Gesture Recognition SystemsHuman Motion and Animation

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

Open access
Qiu, Helei, Hou, Biao, Ren, Bo, Zhang, Xiaohua·2022·arXiv
DOI 10.48550/arxiv.2201.02849arXiv 2201.02849
Open this paper in CoChat — free accountOpen PDF

Abstract

Capturing the dependencies between joints is critical in skeleton-based action recognition task. Transformer shows great potential to model the correlation of important joints. However, the existing Transformer-based methods cannot capture the correlation of different joints between frames, which the correlation is very useful since different body parts (such as the arms and legs in "long jump") between adjacent frames move together. Focus on this problem, A novel spatio-temporal tuples Transformer (STTFormer) method is proposed. The skeleton sequence is divided into several parts, and several consecutive frames contained in each part are encoded. And then a spatio-temporal tuples self-attention module is proposed to capture the relationship of different joints in consecutive frames. In addition, a feature aggregation module is introduced between non-adjacent frames to enhance the ability to distinguish similar actions. Compared with the state-of-the-art methods, our method achieves better performance on two large-scale datasets.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2201.02849

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 34

  1. [1]1. Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇ ci´ c, and Cordelia Schmid. Vivit: A video vision transformer. ArXiv , abs/2103.15691, 2021. 3 · arXiv
  2. [2]2. Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In The Thirty-eighth International Conference on Machine Learning , Jul 2021. 3
  3. [3]3. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV 2020) , pages 213-229, Cham, 2020. 3
  4. [4]4. Zhan Chen, Sicheng Li, Bing Yang, Qinghan Li, and Hong Liu. Skeleton-based action recognition with shift graph convolutional network. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1113-1122, 2021. 11
  5. [5]5. Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, and Hanqing Lu. Skeletonbased action recognition with shift graph convolutional network. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 180-189, 2020. 11
  6. [6]6. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv , abs/2010.11929, 2021. 3 · arXiv
  7. [7]7. Haodong Duan, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, and Bo Dai. Revisiting skeletonbased action recognition. ArXiv , abs/2104.13586, 2021. 1 · arXiv
  8. [8]8. Xiaoke Hao, Jie Li, Yingchun Guo, Tao Jiang, and Ming Yu. Hypergraph neural network for skeleton-based action recognition. IEEE Transactions on Image Processing , 30:2263-2275, 2021. 1
  9. [9]9. Jian-Fang Hu, Wei-Shi Zheng, Jianhuang Lai, and Jianguo Zhang. Jointly learning heterogeneous features for rgb-d activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(11):2186-2200, 2017. 3
  10. [10]10. Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. Learning clip representations for skeleton-based 3d action recognition. IEEE Transactions on Image Processing , 27(6):2842-2855, 2018. 11
  11. [11]11. Ce Li, Chunyu Xie, Baochang Zhang, Jungong Han, Xiantong Zhen, and Jie Chen. Memory attention networks for skeleton-based action recognition. IEEE Transactions on Neural Networks and Learning Systems , pages 1-15, 2021. 1
  12. [12]12. Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu. Skeleton-based action recognition with convolutional neural networks. In 2017 IEEE International Conference on Multimedia Expo Workshops (ICMEW) , pages 597-600, 2017. 1, 3
  13. [13]13. Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu. Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages 786-792, 2018. 11
  14. [14]14. Jianan Li, Xuemei Xie, Zhifu Zhao, Yuhan Cao, Qingzhe Pan, and Guangming Shi. Temporal graph modeling for skeleton-based action recognition. ArXiv , abs/2012.08804, 2020. 4 · arXiv
  15. [15]15. Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Symbiotic graph neural networks for 3d skeleton-based human action recognition and motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2021. 1
  16. [16]16. Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5457-5466, 2018. 11
  17. [18]18. Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C. Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(10):2684-2701, 2020. 8
  18. [20]20. Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 140-149, Seattle, WA, USA, Jun 2020. 1, 4, 11
  19. [21]21. Yuya Obinata and Takuma Yamamoto. Temporal extension module for skeleton-based action recognition. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 534-540, 2021. 4
  20. [24]24. Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1010-1019, 2016. 8
  21. [25]25. Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Skeleton-based action recognition with directed graph neural networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7904-7913, 2019. 11
  22. [26]26. Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12018-12027, Long Beach, CA, USA, Jun 2019. 1, 4
  23. [27]27. Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12018-12027, 2019. 11
  24. [28]28. Lei Shi, YiFan Zhang, Jian Cheng, and Hanqing Lu. Decoupled spatial-temporal attention network for skeleton-based action recognition. In Asian Conference on Cpmputer Vision , 2020. 1, 2, 6, 11
  25. [29]29. Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan. An attention enhanced graph convolutional lstm network for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1227-1236, 2019. 1,
  26. [30]30. Chenyang Si, Ya Jing, Wei Wang, Liang Wang, and Tieniu Tan. Skeleton-based action recognition with spatial reasoning and temporal stack learning. In Computer Vision - ECCV 2018 , pages 106-121, Cham, 2018. Springer International Publishing. 1, 3
  27. [32]32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 60006010, Red Hook, NY, USA, 2017. 1, 3
  28. [33]33. Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 588-595, 2014. 3
  29. [35]35. Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7794-7803, June 2018. 1
  30. [36]36. Shenghua Wei, Yonghong Song, and Yuanlin Zhang. Human skeleton tree recurrent neural network with joint relative motion feature for skeleton based action recognition. In 2017 IEEE International Conference on Image Processing (ICIP) , pages 91-95, 2017. 1, 3
  31. [37]37. Haiping Wu, Bin Xiao, Noel C. F. Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. ArXiv , abs/2103.15808, 2021. 3 · arXiv
  32. [39]39. Fanfan Ye, Shiliang Pu, Qiaoyong Zhong, Chao Li, Di Xie, and Huiming Tang. Dynamic gcn: Context-enriched topology learning for skeleton-based action recognition. In Proceedings of the 28th ACM International Conference on Multimedia , pages 55-63, New York, NY, USA, 2020. 11
  33. [40]40. Xikun Zhang, Chang Xu, and Dacheng Tao. Context aware graph convolution for skeletonbased action recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14321-14330, 2020. 4
  34. [41]41. Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. ArXiv , abs/2011.09315, 2020. 3 · arXiv

Cited by

Sorted by citations

Metadata

Authors
Qiu, H., Hou, B., Ren, B., & Zhang, X.
Affiliations
Xidian University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2201.02849
arXiv
2201.02849
OpenAlex
W4221162785
Cited by
44
References
1
Citations by year
2025: 8 · 2024: 20 · 2023: 14 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
Keywords
skeleton-based action recognition, self-attention, long jump

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review