Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“Previous skeleton-based action recognition methods utilize graph topologies or manually designed rules to transform the raw skeleton sequence into a grid-shape structure such as pseudo-image [10,17,20] or graph, and then feed it into RNNs, CNNs, or GCNs [36] to extract features.”

“In the particular context of skeleton-based action recognition, GCNs have been increasingly used [13,14,24– 26,35,36,42,43] as they explicitly model, with a better interpretability, the spatial and temporal connectivity among joints.”

“…Multi-Task Learning Network [17] , the FSNet [26] , the Skeleton Visualization [31] , the Two-Stream Attention LSTM [29] , the Multi-Task CNN with RotClips [18] , and the Body Pose Evolution Map method [32] , as well as GCN-based ones ST-GCN [49] and current best GCN method [41] implemented by us.”

“Besides, some approaches are based on the spatial graph which convolutions on the graph directly (Yan et al., 2018).”

“25% STGCN [69] 81.”

“Yan et al. [18] proposed the first GCN-based method called ST-GCN, where the graph is manually designed by the correlation between physically connected human joints.”

“Recent advances in keypoint detection techniques, particularly using deep neural networks, have led to us-ing keypoints to improve the performance of recognition tasks [64, 77].”

“As many real-world data show the spatial-temporal graph structure, these STGNN-based models have been widely applied and expressed competing performance, such as [4]–[8].”

“…which combines GCN and LSTM to establish spatial dependence and temporal correlation respectively; spatial-temporal convolution network (STGCN) [24], using GCN and CNN to build correlation; multi-component spatio-temporal graph convolutional network (MSTGCN) [25], which captures super long…”

“With advances of geometric signal processing [2], deep graph learning for the non-Euclidean data has been proposed and studied [37, 38].”

“The spatio-temporal graph convolutional network (ST-GCN) [6] with its dynamic skeleton model was the first graph convolution approach to skeleton-based action recognition tasks.”

“The relationships between time-series joints have been studied using a variety of skeleton-based methods [8,18,31,42] that primarily use GNNs.”

“In particular, ST-GCN (Yan et al., 2018) is a milestone work, using pre-defined graphs to extract the action patterns.”

“Differently from previous related works, which use either ST-GCN [10, 12, 13], or a Recurrent Neural Network [9] to encode the human motion, COSKAD implements its encoder as a Space-Time-Separable Graph Convolutional Network [14].”

“We follow approach [48] and use the estimated joint locations in the pixel coordinate system as the input to our pipeline.”

“From each clip we construct a spatio-temporal pose graph and apply a spatio-termporal graph convolutional network (ST-GCN) [13], a model for spatio-temporal pose graph, to learn and represent the clip.”

“In [86], they further proposed a coherence constrained graph LSTM with a temporal confidence gate and a spatial confidence gate to control the memory updating.”

“An efficient and accurate human pose estimation approach can benefit extensive real-life applications, including activity recognition (Yan, Xiong, and Lin 2018), human-computer interaction, virtual or augmented reality, AI Coach(Wang et al. 2019), and so on.”

“Recent years have witnessed successful deep architectures [5], [9], [23], [39], [52], [62] with promising results on benchmark datasets [3], [38].”

“SkeleTR samples a large number of short skeleton sequences instead of a few long ones (practice of prior works [64, 16]) from each video.”

“3D Human Pose Estimation (HPE) is the foundation of many research fields such as action understanding (Yan, Xiong, and Lin 2018; Duan et al. 2022), virtual humans (Guzov et al. 2021), and self-driving (Xu et al. 2021; Bouazizi et al. 2022).”

“The performance of this scheme is satisfactory, especially based on InceptionNet V3 [43], even better than some GCNbased methods [2, 6] (use one stream), and the computational cost is much lighter than GCN.”

“Nonetheless, the vast majority of the recent deep learning methods for co-evolving time series [21, 23, 24, 35, 37] have almost exclusively focused on a single mode.”

“We used ST-GCN [13] as our baseline, which was originally used for action recognition, and we modified it to test the sign language dataset.”

“In order to address this problem, Yan et al. [22] represent the human joints with a graph and use the spatial–temporal graph convolutional network (ST-GCN) for feature extraction.”

“As a fundamental task in computer vision, it is widely applied to human-robot interaction [15], action recognition [52], human tracking [32], etc.”

“A Spatio Temporal Graph Convolution Network (ST-GCN) [16] extracts features, which are encoded in semantic space by a continuous bag of words method.”

“ST-GCN (Yan, Xiong, & Lin, 2018) develops an automatic learning model to establish dynamic skeletons to address the limitations of hand-crafted features in spatial–temporal prediction.”

“Table 1 Content Guidance of This Article learning tasks involving diverse sets of non-Euclidean data [13], [18], [63], [64], [65].”

“Taking a similar approach to previous work on body pose estimation [24], each graph node at ED is connected to the corresponding node in ES to model the temporal connections.”

“Other applications of dynamic GNNs include fraud detection [31, 48, 52] and human motion prediction [55].”

“Recently GCN-based models [13] have been favored for the fine-grained modeling of the spatial structure by using graph representation, and have achieved more impressive performance.”

“Yan et al. [48] proposed Spatial-Temporal Graph Convolutional Networks (ST-GCN), which automatically learns both the spatial and temporal patterns from skeleton-based data.”

“Yan et al. [48] presented spatial-temporal graph convolution networks that can learn both the spatial and temporal patterns of dynamic skeletons.”

“The predicted results could be used for understanding people in images and videos, with wide potential applications such as action recognition [45,58] and animations [54].”

“The first work applying GCN to skeleton-based action recognition is ST-GCN (Yan, Xiong, and Lin 2018), which constructs a spatio-temporal graph with the joints as the nodes and skeletal connectivity as the edges to model joint dependencies.”

“In terms of GCN-based human motion modelling, the most related work is ST-GCN [44], which applies a”

“We select here a three-layer LSTM model as the main structure, and adopt the idea from ResNet [24], [25], that is adding the input of the first layer to the output of the second layer, and also as the input of the third layer LSTM.”

“[27] propose a spatial-temporal graph convolu-”

“For all models, including ours, we use a spatial-temporal graph convolution network (STGCN; Yan et al., 2018) to encode the adjacency matrix and the prediction and predict energy function values or score functions.”

“Method View Primitive Task Atomic Action Left-Hand Right-Hand Left-Hand Right-Hand Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 ST-GCN [1] Average 39.”

“Hence, the basic TGC can be performed similarly to the classical convolution operation [25].”

“Different from RNNs and CNNs, GCNs are more conducive to learning structural skeleton data by exploring the inter-relationships among body joints [19], [20], [38], [39], [44], [45], [46], [47].”

“Some of them try to learn temporal information along with local structures, such as HTNE [43], EvolveGCN [22], GraphWavenet [36], DGNN [19], EPNE [33] and ST-GCN [39], but they have the problem of high computation complexity.”

“The ST-GCN [56] is the GCN-based skeleton method, while the stacked Bi-LSTM [36] is the RNN-based method.”

“Unlike existing offline learning approaches [7, 37] that update the parameters of the fixed-structure network during learning, our Else-Net has the capabilities in dynamically and flexibly searching and activating only the most relevant learning block in each Elastic Unit.”

“Unlike STGCN and MSG3D which use raw joints and bones and rely on networks to learn good clas-sification features, SGN also employs semantic features, where different joint types are encoded in learning their patterns and correlations.”

“RNNs are able to efficiently leverage time-series information, while GNNs are able to incorporate the spatial constraints and relationships of human joints in their models.”

“More generally, spatio-temporal graphs have been explored in the field of action recognition (Jain et al., 2016; Sun et al., 2018; Wang & Gupta, 2018; Yan et al., 2018; Girdhar et al., 2019; Herzig et al., 2019; Materzynska et al., 2020).”

“An added benefit of many DL methods is the end-to-end learning and prediction setup, removing the need for feature engineering and validation [14], [15].”

“ST-GCN [43] with 16 hidden channels is used as the visual feature extractor and the extracted feature dimension is 256.”

“2) Appearance-Guided GCN: We use ST-GCN [38] as the backbone of posture feature extraction model and remove the temporal downsampling to make the temporal dimension of appearance feature and posture feature consistent.”

“, on Convolutional Neural Networks (CNNs) [6], [7] and Graph Neural Networks (GNNs) [7], [8].”

“Over the last decade, skeleton-based behaviour analysis has evolved from pseudo-images with CNNs [21, 24, 28, 35] and sequence coordinate vectors with RNNs [9, 26, 27, 34, 45], to the solid improvements of GCNs [6, 7, 33, 43, 47, 46] which models skeleton data as a spatiotemporal graph, which better represents the emb…”

“Over the past few years, graph neural networks have achieved unprecedented advances with various approaches being developed [21, 19, 63, 7, 57, 56, 25, 27, 48, 13, 24, 32, 26].”

“Graph Convolutional Networks (GCNs) [18], which generalize Convolutional Neural Networks (CNNs) to graphs, have been deeply studied to explore the relational data structure [19], contextual information [20], and geometric dependency [21].”

“To this end, inspired by recent success on skeleton-based action recognition [19]–[23], we propose a simple yet effective multitask training paradigm, namely Multitask NonAuToregressive model (mNAT), which is further empowered by the merit of action recognition.”

“Most well-established methods are graph convolutional neural networks (GCN)-based (Kipf and Welling 2016; Yan, Xiong, and Lin 2018; Shi et al. 2019; Cheng et al. 2020; Ye et al. 2020; Chen et al. 2021), more recent approaches leverage transformer architectures (Shi et al. 2020; Plizzari, Can-nici,…”

“Spatial-Temporal Graph Convolutional Networks (ST-GCN) [8] is the pioneering method to model the skeleton data, which breaks the limitations that the previous method cannot effectively extract spatial and temporal features at the same time.”

“58], virtual try-on [31], and mixed reality [5, 20, 34].”

“In addition, keypoint-based action recognition has drawn much attention [60, 75, 112, 119, 120].”

“GCNs have shown impressive capability on various tasks [2, 20, 23, 24, 45, 46].”

“In this subsection, we introduce a multi-layer graphical neural network model (GNN) which consider the global structure of the network that has been successfully applied in different domains [25, 28, 39].”

“In particular, spatio-temporal graph convolutional networks (ST-GCNs) 29,30 can help capture both spatial and temporal dependencies in nodes and links, enhancing the model ’ s ability to predict water quality changes over time 31 .”

“They also evaluated conventional action recognition methods, including 2D CNN based [11, 29, 53], 3D CNN based [6, 50], GCN-based [40, 58], and Transformer-based [41].”

“On the other hand, graph convolutional networks (GCNs) were applied to characterize two-dimensional spatial relationships of the input time-series subject to an underlying topology (a planar graph) [35].”

“Many recent highperformance network backbones can be considered as the special cases of multi-axial mixer based on what types of mixing functions are applied, which mainly include convolution [6][10][11], self-attention [12][13], and multi-layer perceptrons (MLP) [14] [15] as shown in Fig.”

“This task can be applied to a wide range of applications, such as skeleton action recognition [55, 61], motion retargeting [1], and human”

“Skeleton-based action recognition [22], [23] is to identify human movements through 3D key points, such as waving, touching the head, etc.”

“One reason for the successful use of GCN on vision-based skeleton data [17, 18, 24, 47-49] is that the human body can be naturally presented as a non-directed graph.”

“Approaches using human poses [10, 23, 44, 57] are effective in laboratory settings [33, 43, 55] but limited in real-world videos [13, 45].”

“It is inspired by the graph convolution from ST-GCN (Yan et al., 2018b), which uses a similar Graph Convolution formulation to the one proposed by Kipf and Welling (2016).”

“Many works [6, 35, 47] apply GCN to the spatial and temporal dimension [47] and achieves remarkable results in the supervised skeleton-based action recognition.”

“8 – – – – – ST-GCN [24] (AAAI 2018) 81.”