Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“Temporal dynamics for HAR were quickly tackled with CNN or RNN strategies [31, 22, 49], although these models lacked a proper learning of the spatial-temporal inter-play among keypoints in the skeleton.”

“Nevertheless, neural networks with GCN structures obtained better results than Lie groups in the same task [23].”

“In [39], [40], GCN is applied for capturing the correlation of human body skeletons to recognize human actions.”

“Firstly, we train the action recognition model based on a slightly modified version of ST-GCN [75].”

“The adopted baseline considers all keypoints as a whole and stacks three ST-GCN [48] layers with the distance partition strategy as the frame-wise feature extractor.”

“Accurate monocular 3D human pose estimation has found its wide applications in augmented reality [26], human-object interaction [7], and video action recognition [45].”

“In the field of trajectory prediction, [31] directly extracts spatio-temporal features from the graph representing G = ( V, A ) by the Spatio-Temporal Graph Convolution Neural Network (ST-GCNN) [39].”

“GNNs have shown great potentials in various domains, including Recommendation Systems (Ying et al., 2018; Wang et al., 2019a), Computer Vision (Yan et al., 2018; Qi et al., 2018), Natural Language Processing (Nguyen & Grishman, 2018; Yao et al., 2018), et al (Kipf & Welling, 2016; Hamilton et al., 2017).”

“…have been widely applied to several research topic in computer vision filed, such as person re-identification (Shen et al. 2018), skeleton-based action recognition (Yan, Xiong, and Lin 2018) and video action recognition and detection (Wang and Gupta 2018; Zhang et al. 2020, 2019; Zeng et al. 2019).”

“It is very shadow when compared with previous GCN-based models (e.g., 9 blocks in ST-GCN [8]).”

“[11] first introduced the GCN into skeleton-based action recognition and proposed the ST-GCN to model the joint spatial and temporal relations simultaneously.”

“Specifically, ST-GCN [45] uses a graph convolutional network to predict video action from pose estimation.”

“first proposed spatial-temporal graph convolutional network [48] for this task, which decouples the neural architecture into a GCN to”

“Spectral graph convolution based on spectral theory has achieved highly promising results on graph-structured data in recent years, such as 3D skeleton-based action recognition [20]”

“Dynamic GCN has been successfully applied for action recognition [27] and motion detection [28].”

“However, since there is no standard feature extractor for motion, we train an action classifier [Yan et al. 2018] instead to classify the motion content and calculate the FMD between the real and generated motion samples from the feature vectors that are extracted from its final pooling layer.”

“ST-GCN [40] is the pioneering work integrating graph convolutions for comprehensive spatial-temporal modeling.”

“For example, fixed spatial-temporal relation graphs are constructed based on some a priori rules or observations [38], [39], [40].”

“Graph neural networks have been widely used in many application such as human action recognition [23], [24], social relationship understanding [25], object parsing [26], multi-label image recognition [27], visual question answer [28], and fashion retrieval [29].”

“To the best of our knowledge, the methods proposed in [15, 4, 27] had achieved state-of-the-art performance for skill action assessment on JIGSAWS , and we used 4-fold cross validation on JIGSAWS by following [15].”

“Recently, some works use GCNs for 3D pose estimation [1] and skeleton-based action regnition [46,16,33].”

“As we know, most of the existing VSR models are derived from general backbone models used in other fields (e.g., action recognition [137], [138], audio speech recognition [135], etc.), and few are designed explicitly for VSR.”

“To solve these issues, in this paper, we propose to predict pedestrian crossing intention using spatial-temporal graph convolution networks (ST-GCNs), inspired by ST-GCN [22] that achieves good action recognition performance.”

“[43] divides human joints into several body parts and constructs a graph to capture joint dependencies.”

“If there are multiple persons, they are processed separately and the classification scores are averaged [38, 27].”

“The method ST-GCN [13] and related methods [29,96,110–113] all adopt one-hop convolution.”

“We implement three widely used positional augmenta- 651 tions [40], [50], which are described as follows.”

“For instance, graph convolutional networks [36]–[39] extended the convolutional operation from traditional data (i.e., images or grids) to graph data.”

“lutional [41] networks.”

“Calculating these metrics relies on a pre-trained action recognition model [87] and we train the recognition model on the HUMANISE dataset.”

“Consisting a spatio-temporal graph convolutional neural network (ST-GCNN) [37] and a temporal extrapolation convolutional neural network (TXP-TCN) [38], Social-STGCNN [11] model replaces the need for aggregation methods by capturing interactions as a graph.”

“On the other hand, we need to evaluate whether it is necessary to leverage ST-GCN to exploit the joint features over the temporal dimension instead of biGRU.”

“Inspired by Kipf and Welling (2016); Yan et al. (2018), we use the Graph Convolutional Network (GCN) to process the human pose P , which is defined as follows: where P is the coordinates of the keypoints’ positions, Eq.7 Eq.8 Eq.8 Eq.8 The mixture purpose transfer module.”

“Yan et al. 22 and Si et al. 23 applied Graph Convolutional Neural Networks to capture the spatial structure information of a body skeleton to recognize different actions in the video.”

“Graph-based approaches have shown to be an efficient way to reason relation in many computer vision tasks such as semi-supervised classification [14], video recognition [37], crowd counting [5], action recognition [26, 40] and semantic segmentation [6, 43].”

“Therefore, recently Graph Convolutional Network (GCN) has been widely applied to this task [5], [6], [7], [8], [9] and has achieved significant performance boost over conventional methods.”

“…studied by utilizing unimodal sensor data, such as visual (Ryoo et al. 2017; Zhang and Parker 2011; Fan et al. 2018), skeleton (Arzani et al. 2017; Ke et al. 2017; Yan, Xiong, and Lin 2018; Iqbal, Rack, and Riek 2016), and wearable sensors (Frank, Kubota, and Riek 2019; Batzianoulis et al. 2017).”

“[41], we solve this problem by partitioning the nodes into different subsets and using different parameters for different partitions, where the number of partitions is fixed but the number of nodes in each partition is variable.”

“It can be seen that on the NTU RGB+D dataset under the Cross Subject evaluation, our method not only achieves the best result when compared with the mainstream ST-GCNs [7,13,15], as well largely reduces the model parameters.”

“Spatio-temporal graph neural networks have a wide range of applications, e.g. traffic forecasting, action forecasting and wind speed forecasting [18–22].”

“Following the experimental setup by [58], we use OpenPose [2] to locate the 18 human body joints.”

“Graph learning, a deep learning subset, aids real-time decision-making and impacts diverse applications like computer vision [1], traffic forecasting [2], action recognition [3, 4], recommendation systems [5], and drug discovery [6].”

“Recent years have witnessed great success achieved by graph neural networks (GNNs) in graph representation learning, which has tremendously advanced various graph tasks [15, 26, 27, 39, 42, 49].”

“Different from the existing methods [15], [16], [17], [18], [19], [21], we propose a novel Motif-GC method to extract features from a motif graph defined for skeletal structures.”

“Recently, a graph-based approach that implements a Graph Convolutional Network (GCN) to model the dynamic patterns in skeleton data is first explored by ST-GCN [19].”

“Graph Convolutional Networks (GCNs) [9] have been successfully used to address various problems in computer vision and natural language processing, such as action recognition [10], object tracking [11] and text classification [12].”

“Joint coordinates extracted through pose tracking on standard videos have already been used successfully in applications of human action recognition [19].”

“Therefore, to handle topology and time respectively, DGs are often decomposed or transformed into components like equivalent static (sub)graphs [74,89,75], random walks [113,124,123,121,122], or sequences of matrices [95,86,93].”

“Runtime (ms) Total FPS ST-GCN [57] 30 pled from a normal distribution to the keypoint image coordinates.”

“Graph Neural Networks have been adapted to model temporal dynamics [32], by exploiting relationships between graphs proximal in time [33], or related by known temporal periodicities in data [34].”

“[54] first applied an ST-GCN [43] for a human action recognition task to a dynamic hand gesture recognition task.”

“We consider an extension of the notion of regularity studied in Yarotsky and Zhevnerchuk (2020) but in the Riemannian context.”

“The main-stream methods lie in three branches: (1) [33,30,26].”

“We build a graph convolutional network as the baseline model with reference to Spatial Temporal Graph Convolutional Network (ST-GCN) [29] , which is the first to apply graph convolutional networks to skeleton based action recognition.”

“Specifical-ly, ST-GCN [Yan et al. , 2018] and SEM-GCN [Zhao et al. , 2019] both follow the spatial perspective.”

“The structure of the graph follows the work of ST-GCN [26].”

“[1] proposes a new structure called Space Time Graph Convolutional Network (ST-GCN) for motion identification.”

“Graph convolutional networks (GCN) have been widely used in skeleton-based action recognition [23]–[28] in recent years.”

“Most of the existing temporal GNNmodels [2, 17, 20, 25, 45] generally discretize a temporal graph into multiple static snapshot in a timeline and use combinations of GNN models and recurrent models whereby the former handle graph information and the latter capture dynamism.”

“However, ST-GCN mainly explores the interaction or collaboration between local (e.g., first and second-order neighboring) joints and the topology of the graph representing the human body is fixed or static.”

“Many research fields consider certain natural graph structures, such as the traffic road network [16], the human skeleton points [17] and molecules structures in biology [18].”

“Spatial temporal graph convolutional network (ST-GCN) [26] is proposed to recognize human activities through skeleton data.”

“We now compare the performance of leranable adjacency with two fixed adjacency matrices: (i) Binary adjacency: a natural choice is a binary adjacency matrix as used for graph-based action recognition [13].”

“STGCN computes spatial neighborhoods as per the bone structure of the 3D poses and temporal neighborhoods according to the instances of the same joints across time steps and performs convolutions based on these neighborhoods to predict the output actions.”

“Most recent methods focus on full-supervised learning algorithms to build their frameworks: methods based on Convolutional Neural Networks (CNN) [2], methods based on Recurrent Neural Networks (RNN) [3], methods based on Graph Convolution Networks (GCN) [4] and methods based on Transformer [5] are widely applied in sk…”

“GNNs have also been used for tasks in computer vision such as, point cloud classification and segmentation [18, 45], as well as human action recognition [49].”

“The spatio-temporal graph (ST-GCN) [99], the first work utilizing GCNs for HAR, is constructed such that the graph nodes represent the joint and the edges encode natural connections in both human body and time.”

“Research in this field has led to new models [13, 54] and learning representations [10].”

“As part of the proposed framework, a pretrained action recognition model [61] was used to identify the frames with large temporal attention in the Kinetics-skeleton dataset [62] as the action frames to train the AGCN.”

“The dominant solutions to it in recent years are the graph convolutional networks (GCNs) [46], as they can model the irregular topology of the human skeleton.”

“1 in Section 2, a typical ST-GCN layer’s computation involves three consecutive PMult(ct , pt): spatial convolution (1× 1 plaintext kernel), matrix-matrix multiplication with normalized adjacency matrix J × J in plaintext, and the temporal convolution along the dimension T (K × 1 plaintext kernel).”

“C2D [50] I3D [4] SlowFast [12] H+O [47] H2O w/ ST-GCN [51] H2O w/ TA-GCN [27] Ours”

“ST-GCN [10] was the first GCN-based method proposed for skeleton-based action recognition.”

“[167] Faster R-CNN Skeleton-based (STGCN [168]) CAD120 Object detector as feature extractor; Dataset provides skeletal data.”

“ST-GCN [22] adds temporal edges to connect each node with itself at the previous and the next time steps, so that GCN can be applied in the unfolded graph with both spatial and temporal edges.”

“Our proposed approach outperforms existing methods in noise environments and is competitive in normal environments on public benchmark dataset Kinetics-skeleton [56] and NTURGBD [36].”

“[4] introduced the graph convolution operation into the skeleton-based action recognition task.”

“Our TRANS4SOAR (Small) has DKey: 1, Nhead : [2, 2, 2], Hdep : [2, 4, 4] and Cdim : [384, 512, 512] with 23 M parameters while TRANS4SOAR has DKey : 32, Nhead: [6, 9, 12], Hdep : [4, 4, 4] and Cdim : [384, 512, 768] with 43 M parameters, where DKey , Nhead, Hdep and Cdim denote dimension of Key, number of the attention…”