View Enhanced Jigsaw Puzzle for Self-Supervised Feature Learning in 3D Human Action Recognition
Open accessAbstract
Self-supervised learning methods have received much attention in skeleton-based human action recognition. These methods rely on pretext tasks to utilize unlabeled data and learn an effective feature encoder. In this paper, a novel self-supervised learning method is proposed. First, we design a new pretext task called view enhanced jigsaw puzzle (VEJP) to improve the learning difficulty of the encoder. The VEJP introduces multi-view information into the jigsaw puzzle, thus forcing the encoder to learn view-independent high-level features of human skeletons. Based on the encoder trained by VEJP, we propose the view pooling encoder (VPE) to integrate the information of multiple views with the pooling mechanism, and the features extracted by VPE are more robust and distinguishable. In addition, by adjusting the difficulty of VEJP, the influence of the pretext task difficulty on the downstream task performance is studied, and the experimental results show that the pretext tasks should be moderately difficult to achieve effective feature learning. Our method achieves competitive results on representative benchmark datasets. It provides a strong baseline for the jigsaw puzzle task and shows advantages in situations where the number of labeled data is minimal.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
You and Wang proposed VEJP to recognize 3D human actions, which captures multi-view data and forces the encoder to obtain view-independent high-level features from the human skeleton.
“You and Wang [31] proposed using a view-enhanced jigsaw puzzle (VEJP) to recognize 3D human actions. VEJP captures multi-view data and forces the encoder to obtain view independent high-level features from the human skeleton.”
From Self-supervised learning: A succinct review (2023) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · see citing works
NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
Shahroudy, A., Liu, J., Ng, T.-T., & Wang, G. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.115 · Cited by 3,148 · see citing works
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition
Shi, L., Zhang, Y., Cheng, J., & Lu, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01230 · Cited by 2,181 · see citing works
Hierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., & Wang, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298714 · Cited by 1,962 · see citing works
View invariant human action recognition using histograms of 3D joints
Xia, L., Chen, C.-C., & Aggarwal, J. K. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6239233 · Cited by 1,512 · see citing works
Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group
Vemulapalli, R., Arrate, F., & Chellappa, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.82 · Cited by 1,506 · see citing works
Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition
Liu, J., Shahroudy, A., Xu, D., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_50 · Cited by 1,353 · see citing works
Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition
Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00371 · Cited by 1,285 · see citing works
Skeleton-Based Action Recognition With Shift Graph Convolutional Network
Cheng, K., Zhang, Y., He, X., Chen, W., Cheng, J., & Lu, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00026 · Cited by 1,046 · see citing works
An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
Si, C., Chen, W., Wang, W., Wang, L., & Tan, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00132 · Cited by 973 · see citing works
Enhanced skeleton visualization for view invariant human action recognition
Liu, M., Liu, H., & Chen, C. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.02.030 · Cited by 871 · see citing works
A New Representation of Skeleton Sequences for 3D Action Recognition
Ke, Q., Bennamoun, M., An, S., Sohel, F., & Boussaid, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.486 · Cited by 793 · see citing works
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
Zhang, R., Isola, P., & Efros, A. A. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.76 · Cited by 724 · see citing works
Cross-View Action Modeling, Learning, and Recognition
Wang, J., Nie, X., Yin, X., Wu, Y. N., & Zhu, S. C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.339 · Cited by 683 · see citing works
Going deeper into action recognition: A survey
Herath, S., Harandi, M. T., & Porikli, F. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.010 · Cited by 646 · see citing works
Human activity recognition from 3D data: A review
Aggarwal, J. K., & Xia, L. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.04.011 · Cited by 578 · see citing works
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
Zhang, P., Lan, C., Zeng, W.-J., Xing, J., Xue, J.-R., & Zheng, N. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr42600.2020.00119 · Cited by 564 · see citing works
Ensemble Deep Learning for Skeleton-Based Action Recognition Using Temporal Sliding LSTM Networks
Lee, I., Kim, D., Kang, S., & Lee, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.115 · Cited by 439 · see citing works
RGB-D-based human motion recognition with deep learning: A survey
Wang, P., Li, W., Ogunbona, P., Wan, J., & Escalera, S. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.04.007 · Cited by 405 · see citing works
Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks
Wang, P., Li, Z., Hou, Y., & Li, W. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967191 · Cited by 391 · see citing works
Boosting Self-Supervised Learning via Knowledge Transfer
Noroozi, M., Vinjimoor, A., Favaro, P., & Pirsiavash, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00975 · Cited by 298 · see citing works
Spatial Temporal Transformer Network for Skeleton-based Action Recognition
Plizzari, C., Cannici, M., & Matteucci, M. (2020) · ICPR Workshops · doi:10.1007/978-3-030-68796-0_50 · Cited by 274 · see citing works
A Comparative Review of Recent Kinect-Based Action Recognition Algorithms
Wang, L., Huynh, D. Q., & Koniusz, P. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2925285 · Cited by 261 · see citing works
PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition
Su, K., Liu, X., & Shlizerman, E. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00965 · Cited by 244 · see citing works
Symbiotic Graph Neural Networks for 3D Skeleton-Based Human Action Recognition and Motion Prediction
Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., & Tian, Q. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3053765 · Cited by 243 · see citing works
3D Human Action Representation Learning via Cross-View Consistency Pursuit
Li, L., Wang, M., Ni, B., Wang, H., Yang, J., & Zhang, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00471 · Cited by 236 · cited 3× · see citing works
“These methods are proven to be effective for skeleton feature learning [19].”
BackgroundMethodRao, H., Xu, S., Hu, X., Cheng, J., & Hu, B. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.023 · Cited by 235 · see citing works
MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition
Lin, L., Song, S., Yang, W., & Liu, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413548 · Cited by 223 · see citing works
Unsupervised Representation Learning With Long-Term Dynamics for Skeleton Based Action Recognition
Zheng, N., Wen, J., Liu, R., Long, L., Dai, J., & Gong, Z. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11853 · Cited by 201 · see citing works
SkeletonNet: Mining Deep Part Features for 3-D Action Recognition
Ke, Q., An, S., Bennamoun, M., Sohel, F. A., & Boussaïd, F. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2690339 · Cited by 172 · see citing works
Context Aware Graph Convolution for Skeleton-Based Action Recognition
Zhang, X., Xu, C., & Tao, D. (2020) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr42600.2020.01434 · Cited by 170 · see citing works
Skeleton-Contrastive 3D Action Representation Learning
Thoker, F. M., Doughty, H., & Snoek, C. G. M. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475307 · Cited by 159 · see citing works
2D Pose-Based Real-Time Human Action Recognition With Occlusion-Handling
Angelini, F., Fu, Z., Long, Y., Shao, L., & Naqvi, S. M. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2944745 · Cited by 99 · see citing works
Skeleton embedded motion body partition for human action recognition using depth sequences
Ji, X., Cheng, J. S., Feng, W., & Tao, D. (2017) · Signal Processing · doi:10.1016/j.sigpro.2017.08.016 · Cited by 95 · see citing works
Convolutional Multitimescale Echo State Network
Ma, Q., Chen, E., Lin, Z., Yan, J., Yu, Z., & Ng, W. W. Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2919648 · Cited by 59 · see citing works
Learning content and style: Joint action recognition and person identification from human skeletons
Wang, H., & Wang, L. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.03.030 · Cited by 57 · see citing works
IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos
Moon, G., Kwon, H., Lee, K. M., & Cho, M. (2020) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00372 · Cited by 32 · see citing works
Li, W., Liu, X., Liu, Z., Du, F., & Zou, Q. (2020) · IEEE Access · doi:10.1109/access.2020.3014445 · Cited by 27 · see citing works
SRNet: Structured Relevance Feature Learning Network From Skeleton Data for Human Action Recognition
Nie, W., Wang, W., & Huang, X. (2019) · IEEE Access · doi:10.1109/access.2019.2940281 · Cited by 24 · see citing works
How and What to Learn: Taxonomizing Self-Supervised Learning for 3D Action Recognition
Tanfous, A. B., Zerroug, A., Linsley, D. A., & Serre, T. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00294 · Cited by 17 · see citing works
Skeleton-Based Action Recognition With Low-Level Features of Adaptive Graph Convolutional Networks
Gang, J., Xiao, Y., Liu, S., et al. (2021) · IEEE Access · doi:10.1109/access.2021.3111633 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ''Attention is all you need,'' in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 5998-6008.
- N. Komodakis and S. Gidaris, ''Unsupervised representation learning by predicting image rotations,'' in Proc. Int. Conf. Learn. Represent., 2018, pp. 1-16.
- T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, ''A simple framework for contrastive learning of visual representations,'' in Proc. Int. Conf. Mach. Learn., 2020, pp. 1597-1607.
- J. Wang, X. Nie, Y. Xia, Y. Wu, and S. Zhu, ''Action modeling, learning and recognition,'' in Proc. IEEE Conf. Comput., Vis. Pattern Recognit., Jun. 2014, pp. 2649-2656.
- D. P. Kingma and J. Ba, ''ADAM: A method for stochastic optimization,'' in Proc. Int. Conf. Learn. Represent., 2014, pp. 1-15.
- XUE WANG (Senior Member, IEEE) received the M.S. degree in measurement and instrument from the Harbin Institute of Technology, Harbin, China, in 1991, and the Ph.D. degree in mechani- cal engineering from the Huazhong University of Science and Technology, Wuhan, China, in 1994. From 1994 to 1996, he was a Postdoctoral Fellow of electrical power system with the Huazhong Uni- versity of Science and Technology. He then joined the Department of Precision Instrument, Tsinghua University, Beijing, China, where he is currently a Professor and the Vice Dean of the Department of Precision Instrument. From 2001 to 2002, he was a Visiting Professor with the Department of Mechanical Engineering, University of Wisconsin-Madison. His research interests include topics in wireless sensor networks, cyber physical systems, intelligent biosignal processing, medical image processing, and smart energy utilization. He is a Senior Member of the IEEE Instrumentation and Measurement Society, the Computer Society, the Computational Intelligence Society, and the Commu- nications Society.
Cited by
Sorted by citations
Self-supervised learning: A succinct review
Rani, V., Nabi, S. T., Kumar, M., Mittal, A., & Kumar, K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09884-2 · Cited by 295 · see citing works
- Authors
- You, W., & Wang, X.
- Affiliations
- Tsinghua University (CN)
- Venue
- IEEE Access
- Published
- 2022
- Volume
- 10
- Pages
- 36385-36396
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4226194958
- Cited by
- 6
- References
- 47
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 3
- FWCI
- 0.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Key Research and Development Program of China (2018YFB2003200, 2018YFB2003500)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- self-supervised learning, skeleton-based action recognition, unlabeled data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
