Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“It is the basis of other related tasks and various downstream vision applications, including video pose estimation[43], tracking[10,42] and human action recognition [22,39,44].”

“For instance, Kinect sensor [61, 33] provides high-quality RGB, depth, and skeleton sequences simultaneously, where both depth [1, 54] and skeleton [39, 44, 59] modalities have been demonstrated to provide effective and unique motion knowledge for action recognition.”

“[1] 90.5 80.5 Res-TCN [13] 91.1 87.3 STA-Res-TCN [13] 93.6 90.7 ST-GCN [39] 92.7 87.7 DG-STA (Ours) 94.4 90.7 Results on SHREC’17 Track Dataset .”

“To explore the architectural capacity question, we study the pose-based action recognition model further and experiment with the recent ST-GCN model (Yan et al. 2018) that makes use of graph convolutions.”

“For example, ST-GCN [38] applied spatial-temporal GCN to model human joints relations in both spatial and temporal dimension.”

“[47] first propose to treat joints and their natural connections as nodes and edges of a graph, and employ a Graph Convolutional Network (GCN) [20] on such a predefined graph to learn joint interactions.”

“The proposed TSA-Net achieves significant improvement in AQA-7 and MTL-AQA compared to pose-based methods such as Pose+DCT [27] and ST-GCN [41] as shown in Table 1 and 4.”

“We employ a hierarchical encoding strategy using spatial-temporal graph convolutions (STGCNs) [58].”

“Moreover, GNNs prove useful in human action recognition through joint-connected graphs [31, 71].”

“We adopt ST-GCN [57], sharing a similar configuration to the discriminator in the ActFormer’s training.”

“action recognition [38] and tracking [14].”

“Recently, GNN and GCN have demonstrated promising results in various computer vision tasks, including scene graph generation [67, 36, 19], point clouds classification and segmentation [30, 63], semantic segmentation [58, 48], action recognition [66] and visual reasoning and question answering [6, 44].”

“Benefiting from the rapid development in pose estimation [2, 7, 16, 28] and graph-based models (GCN [27, 35] and Transformer [23]), the pose-based methods have achieved fairly surprising results in some cases [30,31,37], e.”

“Yan et al. proposed ST-GCN [36], a generic graph-based formulation for modeling dynamic skeletons, which can capture the patterns embedded in the spatial configuration of the joints as well as their temporal dynamics.”

“The model introduced by Kipf and Welling [20] has been shown to work well on a variety of graph-based tasks [20, 29, 45] and has spawned variants including [1, 42].”

“After that, Graph Convolution Networks (GCNs) [20] is widely used in many domains, such as video classification [41] and action recognition [46].”

“Graph neural networks (GNNs) have achieved unprecedented advances in recent years, showing promising performance in handling graph data lying in the non-Euclidean domain [16, 37, 53, 44, 6, 41, 19, 23, 36, 13, 18, 25, 42].”

“of Graph Convolutional Networks (GCNs) [18], [19], [20], [21].”

“[73] is a graph convolution based approach.”

“As Method Backbone NTU-120 X-sub X-set CPM [59] ST-GCN 78.4 78.9 CrosSCLR [23] 3s-ST-GCN 80.5 80.4 AimCLR [49] 3s-ST-GCN 80.1 80.9 CrosSCLR [23] STTFormer 75.0 77.9 AimCLR [49] STTFormer 74.6 77.2 SkeletonMAE [53] STTFormer 76.8 79.1 MCC [47] 2s-AGCN 81.3 83.3 ViA [56] 2s-UINK 85.0 86.5 Hi-TRS [6] 3s-Transformer 85. w…”

“Among them, the literature based on RNN [30], [31] and the literature based on CNN [32]–[34].”

“We first use a feature pool composed of linear layers ϕ to project each vocabulary feature into a same-size embedding and then aggregate them together to produce the fused feature F :”

“In recent years, some researchers have focused on GCN for image classification (Chen et al. 2019), human action recognition (Yan, Xiong, and Lin 2018), semantic segmentation (Li and Gupta 2018), pedestrian analysis (Li et al. 2019b; 2019a) and so on.”

“1 Compared with the ST-GCN [16] in Table I, which is the first GCN-based method and is currently the most popular backbone for skeleton-based action recognition, our model outperforms by around 10% and 20% in accuracy on both benchmarks of both datasets, respectively.”

“Three state-of-the-art models for the skeleton-based action recognition task, ST-GCN (Yan et al. 2018), NAS-GCN (Peng et al. 2020) and MS-G3D (Liu et al. 2020) are provided as baselines that infer the emotional state based on the raw long instances.”

“…fusing angular information into the existing modern action recognition architectures, such as spatio-temporal graph convolutional network (STGCN) [39] and decoupling GCN [4], confusing action sequences can be classified more accurately, especially when the actions have very similar motion…”

“Because explicit action labels are not available in both GTA-IM and PROX, we change the encoder model from ST-GCN [32] model, which is used to compute FID score in [31], to ERD [6], which is a wellknown LSTM based method for motion prediction.”

“Note that STA-LSTM [53], TPN [19], VA-LSTM [82], and ST-GCN [75] are only for skeletons.”

“However, most previous works deal with spatial dimension and temporal dimension separately using different modules and mechanisms, and then fuse the results [18, 34].”

“Unlike the existing 3D action recognition method [7,47,18,12,44,29,30] learning the discriminative features through the designed effective networks, the goal of”

“After those processes, we employ a three-layer ST-GCN (Yan et al., 2018) to construct the short-term temporal encoder.”

“The first work using a GCN for skeleton-based action recognition is ST-GCN [16].”

“The first 4 rows in Table 3 compare the performance of different node partition strategies, where three of them are directly taken from the ST-GCN [57].”

“, point cloud classification [27], action recognition [37], and traffic forecasting [39].”

“Luo et al. [24] exploit stacked layers of ST-GCN [50] to accumulate joint information over the spatio-temporal dimensions of the frame and predict joints in the future.”

“While in Gait-D, the canonical polyadic decomposition algorithm is used to decompose features extracted from ST-GCN[26] blocks.”

“7 ST-GCN [39] skeleton 92.”

“This increases its flexibility to process intermediate features and generalizes the masking aggregation rule in [37], which can be seen as an EdgeNet of order K 1⁄4 1.”

“They use a CNN encoder to extract music features and use an ST-GCN [35] module to decode the dance motions.”

“Yan et al. (Yan, Xiong, and Lin 2018) make the first attempt to propose a spatial-temporal GCN for action recognition.”

“usion. In the final merging stage, the outputs of spatial path, temporal path and residual path are aggregated by summation. 3.3 Network Architecture To maintain consistent with state-of-the-art GCNs [52][42][39], we introduce ten ST inception blocks into our STIGCN. The overall architecture is illustrated in Fig. 2 a…”

“Graph-based architectures, as exemplified by ST-GCN [156], excel in this regard.”

“Note that ISC and the proposed CMD share the same BiGRU encoder, which is different from the STGCN [60] encoder in CrosSCLR.”

“[56] applied GCNs to model human movements and classify actions.”

“Our method obtains competitive results to ST-GCN [45], being slightly inferior at the Cross-Subject protocol, but superior at Cross-Setup.”

“Graph convolutional networks (GCNs) are developed primarily focusing on point clouds classification [17, 18], scene graph generation [37], and action recognition [38] in computer vision.”

“We adopt the ST-GCN and DSTA-Net (Shi et al. 2020) as backbones to evaluate our method.”

“3) Graph Convolution: In [47], the spatial graph convolution on vertex v i is defined as where f represents the feature map, v represents the vertex in the spatial graph, S i represents the convolution area for v i , w represents the weighting function similar to the traditional convolution…”

“[112] 2018 The authors developed the first strategy for collecting spatial and temporal data from skeleton data by encoding the skeleton data with GCN.”

“action recognition [40], anomaly detection [50], recommendation system [26] and supervised person re-ID [32].”

“For example, some researchers use the ST-GCN to predict traffic flow [26], [27], [28], [29], Yan et al. regarded a person as a graph to recognize actions, human joints are nodes in the graph and connections between joints are edges [59].”

“Further, considering the connections between joints, Graph convolution networks (GCNs) [102], a generalization of CNNs, are utilized to model the correlation between joints for the human motion prediction.”

“In order to capture discriminative spatial structure and temporal motion patterns, existing methods [27, 31, 36] usually rely on a shared spatio-temporal representation for the skeleton data at the original frame rate of input sequences.”

“to several research areas in computer vision, such as skeletonbased action recognition [50], object detection [48] and video classification [45].”

“While comparing the previous comparison models, we add ST-GCN [45] and DCRNN [46] models for comparison.”

“Several studies have demonstrated that GCNs possess a robust modeling capacity for spatial structural information [22], [26], yet they still lack sufficient ability to mine temporal information.”

“We train two fundamentally di erent types of machine learning methods on the ASL Citizen dataset: an appearance-based approach, I3D, which is based on a 3D convolutional network applied directly to the frames of the video (Carreira and Zisserman, 2017); and a pose-based approach, ST-GCN, which preprocesses the video t…”

“Most recent work in (supervised) skeleton-based action recognition (Chen et al., 2021a;b; Wang et al., 2020) adopts the ST-GCN model of Yan et al. (2018) for its simplicity and performance.”

“[88] proposed a spatial-temporal graph convolutional network to learn the spatial and temporal pattern from skeleton data.”

“Traditional GNN models such as Graph Convolutional Network (GCN) [16], GraphSAGE [13] and Graph Attention Network (GAT) [39] are widely used in computer vision tasks such as pose estimation [44, 51], panoptic segmentation [42], point cloud analysis [50], etc.”

“Many recent works attempt to model the interaction be-tween traffic participants by constructing a scene graph and utilizing graph neural networks to extract relational features [6].”

“Here, RGB modality pays greater attention to visual modalities (RGB and Depth), whereas Skeleton modality pays greater attention to non-visual modalities (Physical Sensors) for extracting complementary multimodal features. such as visual [13]–[15], skeleton [16]–[18], and wearable sensors [5], [19].”

“The response magnitude of all the joints (white dots) in a motion sequence in the last layer of ST-GCN [49] backbone.”

“The human body shows a natural graph structure, so that some advanced work constructed graph networks to address human pose related problems, such as action recognition [25], motion prediction [14], 3D pose regression [28,5].”

“GNNs have achieved significant success in human action recognition [115], [116], [117].”

“For further motion understanding task, we predict the fine-grained semantic action attributes using a spatial-temporal Graph Convolutional Network(ST-GCN), based on both the original motion embedding stream and the recovered 3D detailed motion stream of the whole video clip.”

“GNN-based models have achieved the state-of-the-art performance on several tasks, such as social influence prediction [30], traf-fic forecasting [38], recommender system [19], [39], action recognition [40] and event detection [41].”

“Yan et al [9] were first to propose a spatio-temporal graphical approach for skeleton-based action recognition.”

“Recently, GNN has been widely applied to many computer vision tasks, e.g., action detection (Yan, Xiong, and Lin 2018), image classification (Defferrard, Bresson, and Vandergheynst 2016) and objection tracking (Cui et al. 2018).”

“Since Kinetics only provides raw videos without the skeletons, ST-GCN [64] uses the publicly available OpenPose toolbox [2] to estimate and extract the location of 18 human body joints on every frame in the clips.”

“[30] 0.31 0.39 0.58 0.43 TSN[26] 0.34 0.23 0.72 0.46 JRG[16] 0.36 0.54 0.75 0.57 USDL[22] 0.64 0.63 0.61 0.63 MUSDL[22]”

“With action classes at hand, we train an STGCN [Yan et al. 2018] action recognition model.”

“It has been successfully applied in many graph-based applications, including molecule property prediction 31 , drug discovery 40 , skeleton-based human action recognition 41 , spatio-temporal forecasting 42 , etc.”

“However, the graph convolutional network (GCN) [43]–[46] algorithm mainly considers and constructs the relationships between nodes (nodes can represent a word in a sentence, a pixel in an image, a feature in a pixel, and so on).”

“Another method is to retrieve structure information of human body by constructing a graph with human joints as edges [40], which also based on CNNs.”

“In recent years, the modelling method based on graph structure has been applied in the 3D point cloud computing [22, 47], action recognition [25, 46] and other fields.”

“Mathematically, let X ( t ) ∈ R M × 3 be a pose carrying the 3D coordinates of M body joints at time t , X = [ X (1) , . . . , X ( T ) ] ∈ R T × M × 3 be a three-mode tensor that concatenates moving poses within T timestamps.”

“The ST-GCN model, which is based on the human skeleton, represents the general sequence of behavior recognition, overcoming the deficiency of the RGB-based model.”

“skeleton based action recognition, especially when extended to spatial-temporal graph convolutional networks (ST-GCN) [8], networks that can learn both spatial and temporal patterns in”

“…existing skeleton-based methods can be grouped into four categories, namely, recurrent neural network (RNN)-based method, convolutional neural network (CNN)-based method, graph convolutional network (GCN)-based method, and transformer-based method [12], [16], [21], [22], [23], [24], [25], [26].”

“Our method also out-performs recent state-of-the-art two-stage method SCG [38] by 0.42% map.”

“To get the classification scores and intermediate features, we follow [56] in training a ST-GCN-based classifier [57] with the ground truth training data.”