Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“Otherwise, we assign 1 to allow updating the low-confident estimate in the BF block.”

“GCNs have achieved state-of-the-art results in zero-shot recognition [50, 26], 3D understanding [18, 34, 53], and action recognition in video [24, 54, 55] by harnessing the flexibility of graphs representations.”

“Their results have been compared with Yan et al. (2018) that utilized spatial–temporal graph convolutional networks (ST-GCN) working on top of skeleton graphs.”

“Thirdly, to distinguish the importance of clips, SANet constructs a skeleton-based Graph Convolutional Network (GCN) for feature scaling, i.e., giving importance weight for each clip.”

“Most existing GCN-based methods [3,4] apply graph convolution to aggregate weighted features of joints according to a predefined graph topology, followed by temporal convolu-”

“[9], [16], [56], [50], which is more detailed bellow.”

“In order to integrate temporal information, graph representations are usually combined with recurrent-based ensembles such as Social-BiGAT [21], Social-STGCNN [22], GRIP++ [23], or adapted to allow learning temporal patterns (ST-GCN) [24].”

“Graph Neural Network (GNN) and its variants [27], [28] are powerful methods for data representation in the non-Euclidean space.”

“In [27], the authors introduced spatial temporal graph convolutional networks (ST-GCN, not to be confused with already mentioned STGCN) for skeleton-based action recognition.”

“To demonstrate the superiority of our model in the fine-grained assessment of the LA task, we compared the performance of our with the state-of-the-art skeleton-based action recognition models [20], [23], [25], [39].”

“Another line of work exploits the skeletal structure of humans to classify activities [49, 54, 43, 60, 39].”

“introduced the Spatial Temporal-Graph Convolutional Network (ST-GCN) [12].”

“Spatial-temporal GCN (ST-GCN) composes a spatial-temporal convolutional block by stacking at least 1D CNN layers and a GCN layer [28], [29].”

“In this paper [59], the authors present a new Spatial-Temporal Graph Convolutional Networks (ST-GCN) model for dynamic skeletons.”

“Yan et al. focused their efforts on extracting hand skeleton joint information using the OpenPose framework to facilitate sign language recognition (SLR) [18].”

“…[33], temporal convolutional network (TCN) [44], unified hand and object model (H + O) [48], transition forests (TF) [45], lie group [40], histogram of oriented 4D surface normals (HON4D) [39], spatial–temporal graph convolutional network (ST-GCN) [46], and temporal transformer network (TTN) [47].”

“3) : Spatial–Temporal Graph Neural Networks (STGNNs) aim to learn complex hidden patterns from spatial-temporal graphs: [Yan et al. , 2018; Wu et al. , 2019; Guo et al. , 2019; Jiang et al. , 2023; Li et al. , 2022]. utilized decision trees and support vector machines on data from a major national…”

“At the same time, due to the expressive power of graph structures, Graph convolutional Networks (GCNs) has been introduced to the skeleton-based HAR problem [51, 93, 104].”

“With the advancements in the field of Graph Convolutional Networks [19] and the availability of abundant 3D Pose data [41], many works have studied skeleton based action recognition [56, 47, 43, 44, 24, 30, 6].”

“The Spatial-Temporal Graph Convolutional Network (ST-GCN) [51] is one of the first proposed models for skeleton-based HAR that utilizes GCNs based on the propagation rule introduced by Kipf and Welling [19].”

“Graph Neural Networks in Visual Understanding The expressive power of GNNs in dealing with complex graph structures is shown to benefit many visual learning tasks [20, 16, 10, 50, 45, 12, 54, 55, 11, 6, 52].”

“Yan et al. [30] proposed the spatial-temporal graph on the sequence of skeleton key points for action recognition.”

“[8] attempted a new method of extracting useful information from dynamic skeletons in the search for a better predictor regarding human actions.”

“ther reviewed. For the tensor data, Yue et al. [20] proposed a tensor mixed-effects (TME) model to analyze massive high-dimensional Raman mapping data with a complex correlation structure. Yan et al. [21] and Si et al. [22] applied Graph Convolutional Neural Networks (GCN) to capture the spatial structure information…”

“Empowering the ubiquitous mobile devices with cameras (e.g., smartphones), PE enables or acts as a key building block for many AI applications, such as activity recognition [12, 53], human-computer interaction [21], and mixed reality [30, 47].”

“The ST-GCN proposed by (31) shows an excellent performance in extracting the features from skeleton sequences.”

“On NTU RGB+D dataset, we compare with one hand-crafted features method [22], four RNN-based methods [8], [10], [11], [54], four CNN-based methods [13], [24], [55], [56], three graph convolutional methods [14]–[16], one graph and LSTM hybridized method [9].”

“…Combining multiple modalities, such as raw RGB frames, optical flow, and audio [66, 21, 70, 73] 2) Spatio-temporal convolutions and 3D convolutions [65, 59, 22, 35, 50, 64], 3) Recurrent models and Long Short-Term Memory based methods [33, 38], and finally, 4) Video Representation methods, such as…”

“These models combine temporal modeling (via RNN [60, 61] or CNN [62, 63]) with spatial modeling through graph convolution.”

“This includes areas such as document classification [58], image classification [59], [60], person re-identification [61], [62], and action recognition [63], [64].”

“The skeleton-based dataset’s main advantage is its robust scene variation, light illumination, and partial occlusion [34,63–65].”

“Then many models based on Graph Convolution Network (GCN) were proposed [21]–[29].”

“[46] constructed a spatio-temporal graph to realize pose estimation and behavior classification of videos.”

“1) State-of-the-Art Action Recognizers: Modern action recognizers can be categorized into two groups based on how they process temporal information: 1) two-stream approaches such as ST-GCN [98] and 2) 3-”

“Huang et al. [12] propose a GCN-based approach in which they utilize the higher order part level graph for better pooling and aligning of nodes in the main skeleton graph.”

“Visual cues [48, 49] and skeleton estimations [35, 50, 51] are examples of such data.”

“Graph Neural Networks (GNNs) [4] apply neural layers on these graphical structures to learn representations [45, 74], classify nodes [64, 67] or generate new data [34, 66].”

“Input : { V } (cid:3)−→ Output : { A } Previous effective skeleton-based action recognition models like ST-GCN [4], 2s-AGCN [5], or SGN [7] can be adopted.”

“Recent works focus on neural networks on manifolds [13, 14, 15, 41] and on graphs [7, 21, 22, 53, 62].”

“Unlike the previous skeleton modeling trials that rely on hand-crafted parts or traversal rules [99], [106], the ST-GCN [107] firstly applies a GCN to skeleton-based human action recognition tasks.”

“These researchers assemble GCN with RNN [1], [2], [3], [4], [5], [6], CNN [7], [8], [9], or attention [10], [11], [12] to model spatiotemporal correlations.”

“In order to learn representation from the graph, graph convolutional network (GCN) [22] was proposed, and it has been widely applied in action recognition tasks [34] and has also been extended to sign language tasks [11, 17, 21].”

“Inspired by the excellent performance in modelling skeleton sequences (Yan, Xiong, and Lin 2018), we use a l g -layer GCN, consisting of spatial-GCN (S-GCN) and temporal GCN (T-GCN), to obtain uni-fied representations Y 2 R C 1 (cid:2) T (cid:2) V .”

“…1 × 1 convolutions to expand the data channels and then multiply the features by uniformly initialized (He et al. 2015) and learnable dependency matrices (which replace the adjacency matrices used in GCN-based methods (Yan, Xiong, and Lin 2018; Shi et al. 2019; Liu et al. 2020; Chen et al. 2021)).”

“On the other hand, the Motion Style Embedding module has 4 motion graph convolution block layers [33], 2 GRU layers, and 1 linear layer.”

“Moreover, the GCN-based methods [8, 30, 31, 42] take the body joints as the nodes and the bones based on the physical connection of the body as the edges to build the graph.”

“Furthermore, graph structure has been used considerably for describing scenes in video analysis [8], [35], [48], [66], [69] and image segmentation work [49], [70].”

“Similar to [39], a set of nodes connected to vt m can be represented as”

“For skeletal movement, ResNet18 [47] and ST-GCN [10] were selected as baseline models.”

“The spatial-temporal graph convolutional network (ST-GCN) [8] therefore models skeleton sequences as spatial-temporal graphs and uses graph convolutions.”

“Research in GNN has become a hot topic in deep learning, and it can be roughly categorized into graph convolution network [8, 20, 27], graph attention network [22, 40], graph auto-encoder [3, 41], graph generation network [2, 7] and graph spatial-temporal network [45, 47].”

“The GCN-based method represents a sequence of skeletons as a graph.”

“Each category contains at least 400 YouTube video clips, from which the skeleton data is extracted using OpenPose [6] in [43].”

“For example, Yan et al. exploited GNNs for skeleton-based HAR by introducing Spatial-Temporal GCNs (ST-GCNs) that can automatically learn both the spatial and temporal patterns from skeleton data [39].”

“5, PennAction).”

“Although RNN and CNN utilized joint coordinates, they did not explicitly exploit the structural topology of the joints; thus, Graph Convolution Network [35, 20, 24, 8] was introduced.”

“Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. tention [39, 21, 28, 7, 35].”

“We achieve the state of the art on large-scale NTU-60 [50], NTU-120 [39], Kinetics-skeleton [75] and UWA3D Multiview Activity II [49].”

“Spatial-Temporal Graph Convolutional Network (ST-GCN) is a graphical approach for learning spatial and temporal characteristics that can be used for action recognition [26].”

“In addition to face clustering, GCNs are also used in many tasks such as skeleton-based action recognition (Yan et al., 2018), knowledge graph (Schlichtkrull et al., 2018) and recommend system (Ying et al., 2018).”

“The sequential classifier can be any model including Graph Convolutional Neural Network [56, 64, 63], 3D Convolutional Neural Network [65, 67, 66, 68] once these models are welldesigned for sequential classification.”

“Importantly, contrastive pre-training is generally encoder-agnostic and can be employed with a variety of models, originally both CrosCLR [17] and AimCLR [16] used ST-GCNs [5] but subsequent works have successfully applied contrastive pre-training to other architectures such as STTFormer [10].”

“of ST-GCN in action recognition [39], the ST-GCN model is investigated to extract spatial and temporal Parkinsonian gait features from the pose graph sequences.”

“Yan [43] dynamically extracted ST features through multiple attention and masked multiple attention mechanisms and determined the significant influential parts of the road network by analyzing the attention weight matrix.”

“Other works have explored the benefits of using divided space-time feature extraction by using a Spatio-Temporal Graph Convolutional Network (ST-GCN), which models the spatial configuration and temporal dynamics of skeletons [39].”

“Yan, et al. [82] presented Spatial-Temporal Graph Convolutional Networks (ST-GCN) for action recognition which learns both the spatial and temporal patterns using GNN.”

“Anisotropic filtering Graph neural networks [42, 32, 18] have been used extensively for modeling dynamical systems [55, 35, 27, 19, 3].”

“This problem has numerous real applications such as traffic speed forecasting (Li et al. 2018), driver maneuver anticipation (Jain et al. 2016) and human action recognition (Yan, Xiong, and Lin 2018), and has attracted considerable research interest due to its importance.”

“Several possible improvements are left for future work, such as combining our PCN with the Graph Convolutional Network [9][43].”

“For human-related anomaly detection, poses can be extracted using OpenPose (Cao et al., 2017) then embedded into pose features via ST-GCN(Yan et al., 2018).”

“The STGC operation performed on the node v t i , n can be formulated as: where Z t i , n is a normalization factor (Yan et al., 2018) where Λ n , n ¼ P j A n , j þ 1 is a diagonal matrix and f 0 t i ℝ N (cid:1) M represents the M -channel output features.”

“Yan et al. [40] proposed ST-GCN for modeling spatial-temporal correlations adaptively by first constructing a joint-level graph and connecting frame-level nodes afterward.”

“The applications of GNNs span over online recommendation [13], traffic forecasting [14], action recognition [15], link prediction [16] and quantum chemistry [17].”

“Our method with local modeling for implementing policy network is short for Ours (Local), which outperforms ST-GCN and CTR-GCN by a large margin using all kinds of protocols.”

“Joint features are commonly the most popular feature in human action recognition [4], [6], [8].”

“Same as UESTC [34] in ACTOR [69], we use ST-GCN [105] as the feature extractor.”

“Especially using GCN, deep features [11,26,27] outperform handcrafted features [18,19] by a large margin.”

“The Eq.1 implicitly have a vertex partition strategy [33].”

“The graph representation is invariant to node and edge permutations and has proved its effectiveness of extracting relevant information of the human body skeleton in many tasks [27, 16, 28].”

“…of the underlying applications [100]–[102] (e.g., node-to-node relationships in social networks [85], edges in 3D modeling [14]–[16], [18], [19], protein connectivity in biological systems [97], [98], etc.) whilst other methods handcraft graph connections by modeling similarities between…”

“Given the input spatial features of i joint, fin(νi), the spatial graph convolution is defined as [30]:”