Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“[11] proposed a Spatial-Temporal Graph Convolutional Network (ST-GAN) that employs graph convolution to aggregate the joint features in the spatial dimension.”

“Note that we use GCN instead of ST-GCN in [43] or GAT in [17] since latent relationship have already fully captured in Relational Social Representation and we only need to propagate features here.”

“Recently, graph-based methods model the data as a graph with joints as vertexes and bones as edges [55,47,40].”

“[38] is the first attempt to use GCNs for skeleton based human action recognition.”

“The adjacency matrix should be first normalized to facilitate STGCN learning [37].”

“The first successful work is ST-GCN (Yan, Xiong, and Lin 2018), which learn the spatial and temporal patterns in skeleton data with graph convolution.”

“Unlike the traditional datasets used by previous works, such as the 3D NTU RGB+D [31], [25], or the KineticsSkeleton [22], [40], the MPOSE2021 dataset collects poses”

“With the development of 2D pose estimation, recent researches [142]–[145] tend to screen out the pose estimation task in action recognition and focus on skeleton-based action recognition directly with structured positions of human keypoints.”

“Previous works [28, 57, 53] find that decoupling the transformations of self-connections and other edges can significantly improve the 3D HPE performance, which we also observe.”

“Methods[31,19,29] based on GNNs are also applied to many areas in computer vision, since the effectiveness of these GNNs.”

“One approach is to use an LSTM [41], another approach is to use a spatial temporal graph CNN (ST-GCN) [74].”

“In this paper, we adopt the widely-used ST-GCN as the encoder to extract the skeleton features.”

“It is a fundamental task with broad applications in action recognition [48, 40], human-”

“GCNs have been used to model similar interactions in traffic networks [22] and activity recognition [55].”

“Many other forms of GNN have been presented since then, including, but not limited to, Graph Attention Networks [15], Graph Autoencoders [16–19], and Graph Spatial–Temporal Networks [20–23].”

“With the rapid advancement of neural networks, deep learning methods have become capable of capturing complex spatio-temporal features (Yu, Yin, and Zhu 2018; Yan, Xiong, and Lin 2018; Seo et al. 2018).”

“Note that in the previous work [20, 31, 12], the weight of the adjacency matrix is consistently fixed with optimization process, i.e ., A ∈ { e i,j = 1 } .”

“Although decoupling self-connections in a GCN has been studied in [38] and [40], we investigate it in different weight sharing methods.”

“Graph neural networks have received increasing attention [16, 27].”

“They are mostly used for trajectory predictions of road vehicles [197], the prediction of pedestrian behaviors for autonomous vehicles [198], [199], or pose/action recognition [200], [201].”

“To further explore whether our model design can be validated on outdoor actions, we conduct experiments on the Kinetics 400 dataset [40], which is based on OpenPose 2D skeleton samples available in [1] and downloadable videos of Kinetics 400 [40] at the time experiments were performed.”

“[39] were the first to propose a spatio-temporal graph convolutional network for action recognition, which was able to learn the temporal skeleton dynamics and classify them.”

“Similar to [61], we first represent human skeleton sequence as an undirected spatial-temporal”

“(5) We replace S-DGCN and T-DGCN by ST-GCN [47].”

“…determination (Kipf and Welling 2017; Defferrard, Bresson, and Van-dergheynst 2016; Zhou et al. 2019; Veli ˇ ckovi ´ c et al. 2018; Hu et al. 2019; Yan, Xiong, and Lin 2018), where G raph C onvolutional N etworks (GCNs) have been demonstrated to outperform traditional graph-based models due to…”

“Followup works [10, 7] addressed these drawbacks, and gradually, graph convolutional networks (GCN) can achieve state-of-the-art results on some tasks including skeleton-based human action recognition [42].”

“To validate the properness of using AGCN layers, we compare our model with the version implemented with ST-GCN [69] blocks, which leads to the ablation of AGCN.”

“RGB videos [17, 18, 95, 96] with additional information such as optical flow and estimated poses and 3D skeletons typically obtained from RGB-D sequences [89, 90, 105, 111] are the common input to existing methods.”

“Human pose estimation is an important task in computer vision with applications in activity recognition [54], human computer interaction [44], animation [4], gaming [42], health [10], and sports [51].”

“Our model outperforms the state-of-the-art approaches on this dataset without data augmentation as used in previous literatures (Ke et al. 2017; Kim and Reiter 2017; Yan, Xiong, and Lin 2018).”

“With graph neural network (GCN) being generalized to various natural language processing (NLP) and computer vision tasks [37], [38], [39], [40], [41], Xin et al. [33] proposed a GCN-based method which learns through the facial landmark graph.”

“We set S = 5 , L = 7 and d = 1 / 8 . k -hop ( k = 1 , 5 , 7 ) means aggregating the neighbors within the distance k (1-hop with a static graph is ST-GCN [42]).”

“Some works on action recognition but not assessment also model spatial-temporal joint relations through graphs [28, 33, 13].”

“Keskes and Noumier (2021) [106] proposed a fall detection system using spatial-temporal graph convolutional networks (ST-GCN) [114].”

“Consequently, research in this domain has predominantly gravi-tated towards Graph Neural Networks (GNNs) [42,69] and Graph Convolutional Networks (GCNs) [3–6,10,19–21,24,36,43,46,47,61,67,68].”

“It is noted that all comparative methods adopt 2.8 19.2 ST-GCN [13] 3.1 16.7 MS-G3D [31] 6.4 98.0 2s-AGCN [29] 6.9 37.4 AS-GCN [14] 7.2 35.5 2s-AAGCN [35] 7.6 39.1 DGNN [80] 8.1 71.1 DeCoup-GCN [81] 13.7 102.3 MAC-Learning (Ours) 8.1 40.7 The abbreviations “M” and “G” denote Mega and Giga,…”

“Some of them capture temporal information along with local structure so as to improve the expressive ability of themodel, with typical examples like DynamicTriad [34], HTNE [36], EPNE [27], EvolveGCN [18], DynamicGCN [4], DGNN [14] and ST-GCN [31].”

“However, since human motion has both spatial and temporal characteristics, it is necessary to consider both when dealing with latent representation learning [56, 50].”

“Based on the ST-GCN [20], many variants were explored [21]–[26], which typically intro-”

“The idea of relation modeling behind GCN has been extended to various real-world applications such as recommendation [53], [55], [61] and skeleton based action recognition [25], [58].”

“Also, some papers represented skele-tons as graphs and utilized graph neural network (GNN) for action recognition [38, 13].”

“Later, the graph convolution operator, which was defined directly in the spatial domain to represent a graph by aggregating neighboring vertex features (Gilmer et al. 2017; Hamilton et al. 2017; Yan et al. 2018; Zhu et al. 2020), became mainstream because of decreased computational complexity.”

“Thus, they must iterate the propagation process and therefore they have limitations regarding the propagation time and gradient explosion or vanishing problems [175, 176, 177].”

“, action recognition [38, 39, 53], action synthesis [52] and pose tracking [32].”

“X-Sub X-View STA-LSTM [42] 73.4 81.2 ST-GCN [47] 81.5 88.3 HCN [26] 86.5 91.1 AS-GCN [27] 86.8 94.2 2S-AGCN [39] 88.5 95.1 Shift-GCN [5] 91.5 96.5 DGNN [38] 89.9 96.1 MS-G3D [32] 91.5 96.2 MS-AAGCN [41] 90.0 96.2 DSTA-Net [40] 91.5 96.4 ST-TR [36] 89.9 96.1 STST (ours) 91.9 96.8”

“Graph models are utilized in several computer vision tasks, and Graph Neural Networks (GNN) is introduced in [35] to model the relations between graph nodes, and a large number of the variants [36]–[41] are proposed.”

“[48] proposes the spatial-temporal graph convolutional networks for skeleton-based action recognition.”

“The field of action feature extraction has recently gained a lot of interest due to emerging high-efficiency posture estimate techniques that leverage deep learning [28–30], and this is currently an important study area in human activity recognition [31].”

“data and construct spatio-temporal graphs for traffic forecasting [4], [23], human action recognition [24], [25], etc.”

“However, conventional GCNs [10], [11], [12], [13] are all single-pass”

“In the work of [89], [90], the authors used spatial-temporal neural networks to learn human motion pattern by examining the time series of human skeleton in the video clip.”

“In addition, spatial-temporal graphs have recently been thoroughly explored in the content of video-related tasks [40], [41], [42], [43]; here, the goal is to encode the spatial and temporal information, and the links between objects from video frames.”

“First and foremost, the area of skeleton-based action recognition pioneered most of the current graph-based spatiotemporal GCN architectures, including the ones used in this paper [22, 29].”

“Due to the attractive universality and efficiency of GCN, innovative applications using GCN to process multiple types of data have appeared in various fields [14, 40-46].”

“CNN-GCN [51] overcomes this problem through a complete convolutional structure which interleaves 1-D-CNN with GCN. ST-GCN [52] extends a temporal flow as graph edges and then extracts both temporal and spatial features using a unified GCN to perform human action recognition.”

“Spatio-temporal attention has been successful in many application scenarios that can be modeled as dynamic graphs (Xu et al. , 2022; Yan et al. , 2018; Zheng et al. , 2020; Guo et al. , 2019; Fang et al. , 2020).”

“Pose-extractors are useful across sign languages and also other tasks such as action recognition (Yan, Xiong, and Lin 2018; Liu et al. 2020), and can be trained to high accuracy.”

“[19], graph convolutional networks (GCNs) and their variants like Graph Attentional Networks (GAT) [35], have seen massive applications in Computer Vision, including modeling video proposal relations in action localization [27, 45], object relations in visual relation reasoning [26], joint relations in skeleton-based…”

“ate on graph structure, has achieved superior performance in graph analysis and shown promise in various applications such as visual question answering [48], image captioning [90], action recognition [85], point clouds classification and segmentation [39, 63, 71, 74], text classification [18, 82, 89], named entity ext…”

“In Table 7, we present the Top1 accuracy among different inputs to train the popular networks [51,63].”

“The ST-GCN [90] and MS-G3D [46] yield the similar results with the pose data as input.”

“Based on spatial–temporal graph convolutional networks (ST-GCNs) [7], the second skeleton part elementwise maximizes action representations of the human body topology.”

“action recognition [48], person Re-ID [49,23] and point cloud learning [22].”

“T , A key pre-requisite for using GCNs is to define the adjacency between the nodes in the graph [5, 29, 49].”

“Early works, such as [45], used a fixed topology based on bone connectivity, demonstrating the effectiveness of GCNs in action recognition.”

“We then train the ST-GCN model on Student-Train in a fully supervised manner using the noisy labels as ground-truth.”

“Another work that combines spatio-temporal modeling more closely is STGCN [37], where they encode both the spatial connection of human joints in a single frame and the temporal connection of the same joint between frames into a single adjacency matrix of GCN.”

“ST-GCN [21] 62.3 2s-AGCN [16] 68.2 HD-GCN [8] 70.6 CTR-GCN [2] 74.5 MS-G3D [10] 76.8 frames.”

“model dynamics of human body skeletons via graphs to retain information for human actions [151].”

“Specifically, we follow the graph structure proposed by [Yan et al. 2018], summarized below.”

“As mentioned in the previous section, pose estimation is a standard preprocessing step and several researchers have presented algorithms that can infer the type of action based on 2D or 3D joint positions [237], [238], [239], [240], [241], [242], [243].”

“objects [19, 26, 28, 56, 62], poses or skeletons [29, 54, 65–67].”

“Spatial-Temporal Graph Convolutional Networks [29] takes advantage of the dynamics of human body skeletons convey significant information for human action recognition.”

“, optical flow [17, 20, 37, 55], audio [23, 36], human skeleton estimates [15, 76].”

“To benefit from the rich semantics of the pose data, we use spatio-temporal graph convolutions (ST-GCN [43]).”

“For example, [46] presents a spatial-temporal GCN to learn both spatial and temporal patterns from skeleton data.”

“Yan et al. [97] proposed a scheme to obtain the sample function and weight function.”

“The previous methods usually compress vertex features by means of average pooling in both the temporal domain and the spatial domain (Yan et al., 2018; Shi et al., 2019; Li et al., 2019b), which will inevitably result in the loss of important spatial-temporal information.”

“S. Yan et al. [149] propose a Spatial-Temporal GCN (ST-GCN) for action recognition, which is similar to 3D convolu-tionalnetworksbutexecutedonskeletongraph,achieving30.”

“The encoder f(·) is based on ST-GCN [40] with hidden channels of size 16, which is a quarter the size of the original model.”

“However, different from [22], we employ the distance partitioning, which can be applied to any graph rather than skeletons.”

“Thanks to the powerful representation ability in nonEuclidean space, Graph Neural Networks (GNNs) are widely applied in the trajectory prediction task [27, 67, 70, 72, 76] recently.”