Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“For ST-GCN, we train it on skeletal sequences from NTU and Kinetics [36] datasets following the standard protocols.”

“Yan et al. (2018) modeled human skeletons across time as graphs and proposed a temporal graph network for human activity recognition.”

“the stacked-layers scheme. Recent approaches use an extension of the graph convolutional networks (GCN) in order to allow the network learning spatio-temporal features more efficiently. Yan et al. in [79] proposed a spatio-temporal graph convolutional network (ST-GCN) which automatically learns both the spatial and tem…”

“Recently, graph convolutional networks (GCNs) have been widely applied in computer vision community, such as video classification [31] and action recognition [32].”

“It not only paves the way for some important downstream tasks such as action recognition [37,40,43,44,51] and human-computer interaction [1,18], but also enables a wide range of applications, e.g . sports analysis [5, 30] and virtual avatar animation [47,64].”

“…the graph convolutional layer [20]; 3) modifying the original LSTM or GRU, which is to replace the fully connected layer in LSTM or GRU by graph convolution [21]; 4) representing temporal correlations as new edges of the graph and constructing a new graph with spatial-temporal correlations [22].”

“To perform this experiment, training and validation data were firstly occluded by the occlusions augmentation technique, while testing data were occluded with the proposed equation (21).”

“We also compare to STGCN (Yan et al. 2018) that uses OpenPose to estimate the pose, i.”

“Many ISR approaches leverage deep learning, adapting action recognition models (Carreira & Zisserman, 2017; Tran et al., 2018; Yan et al., 2018; Jiang et al., 2021; Albanie et al., 2020) on datasets such as WLASL (Li et al., 2020a), MSASL (Joze & Koller, 2018) and AUTSL (Sincan & Keles, 2020).”

“It serves as a foundation for many down-stream tasks such as AR/VR, action recognition [21,65], and medical diagnosis [11].”

“Spatial-temporal graph convolution networks (ST-GCNs) have been proposed for working on the skeleton graph (Yan, Xiong, and Lin 2018; Si et al. 2018; Chen et al. 2021; Cheng et al. 2020b; Liu et al. 2020; Zhang et al. 2020; Shi et al. 2019b).”

“This paper provides a comparison of different methods such as [66,116,119–121] with the evaluation criteria such as MAD, Mean Absolute Percentage Error (MAPE) and RMSE scores.”

“Various representations that focus on the spatial and temporal features of the skeleton modality like recurrent neural network (Liu et al. 2017b), and graph convolutional network (Yan, Xiong, and Lin 2018; Li et al. 2019; Shi et al. 2019a) have been proposed.”

“[72] Sijie Yan, Yuanjun Xiong, and Dahua Lin.”

“Learning from spatial and temporal 53 components of data using deep learning can also be seen in various non54 biological domains [16, 17, 18]; such approaches, however, usually rely on a 55 single spatio-temporal convolutional block that creates low-dimensional em56 beddings at each timestep, instead of performing a…”

“…RNN [39] 71.3 79.5 STA-LSTM [11] 73.4 81.2 Ensemble TS-LSTM [40] 74.6 81.3 Visualization CNN [31] 76.0 82.6 VA-LSTM [38] 79.4 87.6 ST-GCN [45] 81.5 88.3 HCN [32] 86.5 91.1 PB-GCN [44] 87.5 93.2 AGC-LSTM [46] (Part) 87.5 93.8 JSR + JMR + BSR + BMR [36] 85.6 92.0 SR-TSL [13] 84.8 92.4…”

“Due to its wide-spread use in graph convolution network based action recognition, we select ST-GCN [8] as our target model, and launch our attack against it.”

“Yan et al. applied a temporal feature enhancement-based ST-GCN model, extracting both spatial and temporal features, as shown in Figure 20 [168].”

“HOI detection is steadily attracting attention in the computer vision field, due to the high potential for use in the down-stream tasks such as action recognition [46], scene understanding [44], and image captioning [47].”

“Pose modality is a compact and high-level representation of human action and contains physical connection among skeleton joints (Li et al. 2018; Ng et al. 2021; Yan, Xiong, and Lin 2018).”

“Graph neural networks (GNN) have attracted high attention in various research areas, which are designed to apply deep learning architectures on graph-structured data (Yan et al. 2018; Wang et al. 2019; Wu et al. 2020).”

“This means that a causal convolutional layer can be used to model time-series data [51].”

“The authors used ST-GCN to extract visual features from the 3-”

“[36] have constructed a spatial graph based on the natural connections of joints in the human body.”

“We may also treat human skeleton data as a sparse point cloud, and construct graphs over the skeleton data by connecting human body joints both spatially and temporally [107], [118], [119].”

“Reconstructing the locations of human joints or other body landmarks in 3D space from a single RGB camera has huge potential, with applications in computer animation [39, 38, 42], action recognition [34, 56, 30, 23, 5], or posture and motion analysis [54, 41, 52].”

“State-of-the-art approaches model the skeleton sequences as a spatial-temporal graph [5], [6].”

“Finally, the optimal number of multi-heads is three, which may be attributed to the centritic and centrifugal choices in constructing the adjacency matrix [39].”

“…NTU-RGB+D [28] and NTU-RGB+D120 [15], and is compared with state-of-the-art RNN-, CNN-and GCN-based human action and interaction recognition methods, including Co-LSTM [39], ST-LSTM [18], GCA-LSTM [21], 2s-GCA [20], FSNET [16], VA-LSTM [35], LSTM-IRN [24], ST-GCN [33], AS-GCN [12] and CTR-GCN [3].”

“Spatial-temporal graph convolution is widely-used in skeletonbased action generation and recognition [49, 50], which usually combines spatial graph convolution and temporal convolution to model spatial-temporal information effectively.”

“The goal is to improve gesture and action recognition using manifold learning [24] and GCN [36].”

“Skeleton-based models use Graph Convolutional Networks (GCN) [55] to model the joints in the skeleton graph, while Spatial–Temporal GCN (STGCN) [56] is used for action recognition, which is executed on a skeleton graph.”

“Topologymodelingisthedesignfocus ofGCN-basedmethods[7],[8],[9],[13],[19],[20],[22],[44], [45],[46],[47].”

“Therefore, we choose [115] as CNN, [114], [117] as LSTM, and [87], [109], [112], [113] as graph-based regressors.”

“[12] proposed a method based on Graph Convolutional Network (GCN), namely stacked-STGCN, which extend STGCN [37] for action detection.”

“We first explore the effectiveness of the cross-domain spatial residual layer, using STGCN [41] as a baseline.”

“Neural Network on Graph Structure: Learning features on irregular graphs has become popular in many applications such as 3D geometric data analysis [23, 44], social network analysis [24], action recognition [49], and pose estimation [12, 8].”

“In addition, graph convolutional network [257] has also been proposed for tasks such as exercise evaluation [13] and pose estimation [265].”

“Its robustness against the variation of appearance and backgrounds has attracted many researchers to explore fully supervised methods for action recognition, such as applying CNN (Ke et al. 2017), RNN (Du, Wang, and Wang 2015) or GCN (Yan, Xiong, and Lin 2018; Shi et al. 2019) on skeleton data.”

“Inspired by the work of spatio-temporal graph convolutional network (ST-GCN) developed in [85,86] to predict the graph structure, ref.”

“Many studies adopt spatial-temporal graph neural networks (STGNNs) for the sequence prediction task, such as action recognition [48, 35], taxi demand prediction [50], and traffic prediction [49].”

“Inspired by previous work [49], [65], we also add a learnable edge importance weighting mask on the graph convolution of this step considering that messages from different joints have different contributions to the feature refinement of the target joint.”

“6M dataset Method MPJPE(mm) aGCN [31] 82.9 ST-GCN [30] 57.4 Martinez et al. [15] 45.5 SemGraph [33] 43.8 Ours 53.1 components designed in our method contribute to the final performance.”

“Yan et al. [22] utilized an improved model, STGCNN, to recognize human interactions.”

“Many of current skeleton-based works [5], [40], [42] for action recognition resort to supervised learning paradigms to learn action representations, which require massive annotated samples for training.”

“Graphs have been widely applied to vision tasks, including point cloud segmentation [17, 41], human action recognition [42], and image classification [12, 32].”

“For instance, works from Peng et al. (2020a, 2020b) and Yan et al. (2018) also endow graph convolutional network with the residual connections, capturing a better representations for skeleton graphs.”

“Some work [13, 33, 41, 51, 53] modifies it to apply GCN into different categories, such as knowledge base construction and text classification.”

“Different with previous methods, Yan et al. [76] proposed a multi-layer graph neural networks, where the graph nodes consist of joint coordinates and estimation confidences, to automatically learn the spatial-temporal pattern of the skeleton data.”

“Furthermore, OD flows can be integrated with epidemiological models to better predict the spatial and temporal dynamics of disease transmission.”

“On the basis of these works, many alternative GNN approaches have been developed, including graph auto-encoders (GAEs) [43], graph generative networks (GGNs) [44], and spatial–temporal graph convolutional networks (STGCNs) [45].”

“Following the convention (Yan et al. 2018 Peng et al. 2020; Shi et al. 2020; Liu et al. 2020), we fix the max number of frames in each sample (i.e., max_sample ).”

“GCNs [4][29][30] can be divided into two domains: spectral and non-spectral methods.”

“On both NTU-60 and NTU-120 datasets the fully fine-tuned CPM outperforms the supervised ST-GCN [39], demonstrating the effectiveness of the unsupervised pretraining.”

“In computer vision, GCNs have been successfully applied to scene graph generation [22, 31, 38, 52, 56], 3D understanding [16, 29, 49, 51], and action recognition in video [20, 53, 55].”

“Thus, the features extracted by GCN contains high-level spatial structural information about the human skeleton structure.”

“To fully utilize it, some recent approaches rely on fully convolutional structures to extract spatial-temporal features from spatial-temporal graphs [41], [42], [44], [48], [49], [50].”

“We first briefly review the pipeline of ST-GCN [53], whose architectural design is widely followed by subsequent works.”

“Prominent examples include the ST-GCN framework introduced by Yan et al. (2018) and variants Wu et al. (2019); Song et al. (2019).”

“One example using temporal-spatial graphs is the action recognition problem [63] from the computer vision field.”

“These models push the boundary of existing variants of graph representation models, such as graph convolutional network, graph attention network and graph recurrent network, through encoding the spatial and temporal information of the nodes (Yan et al 2018).”

“We test all the baselines and proposed algorithm on one popular dataset, ModelNet40 [27] using the state-of-the-art model architecture Dynamic graph convolutional neural network (DGCNN) [29].”

“…consist of Faster R-CNN [41], region-based fully convolutional networks (R-FCNS) [50], single-shot multibox detector (SSD) [51], multiscale spatial and channel-wise attention model (MSCA) [18], spatial temporal graph convolutional net-TIAN work (ST-GCN) [52], and you only look once (YOLOv2) [53].”

“Graph convolutional networks (GCNs) [16] have been used in several areas of computer vision such as skeleton-based action recognition [36], video classification [11,35], visual object tracking [5], and temporal action localization [39].”

“[12] propose a Spatial-Temporal Graph Convolutional Networks (ST-GCN) for action recog-”

“After that, GCN draws more attention of researchers and then was employed in a variety of applications such as webscale recommendation systems [34], Skeleton-based action recognition [35], and traffic forecasting [36].”

“More recently, methods have been proposed to treat the connectivity of joints as a graph to be used as input for deep learning models [55, 45].”

“ST-GCN [ Yan et al. , 2018 ] is a widely used GCN-based human skeleton recognition model in recent years.”

“E.g., For NTU-RGB+D 60, compared with ST-GCN [8], which is one of the most influential GCN-based models, our results outperform it by 8.6% (90.1% vs 81.5”

“Particular notable examples of that approach are the Spatial-Temporal Graph Convolution Network (ST-GCN), [Yan et al., 2018], and the Semantic Graph Convolution Network (SemGCN), [Zhao et al.”

“In this work, we adopt a 2D convolution as reported in [38], to extract the temporal information of the observed trajectories.”

“Recent studies [50,38,40] have achieved remarkable performance improvement by learning skeleton encoders in a fully-supervised manner.”

“We apply spectral graph convolution [Kipf and Welling, 2016] with the following modified propagation rule including learnable edge weights [Yan et al., 2018] : fk+1 = σ(D̃ − 12ω(A+ I)D̃− 1 2 fkWk), (1) where I represents the identity matrix, D̃ii = ∑ j(A ij + Iij), ω = {ωij} are learnable edge…”

“Skeleton-based: a MS-G3D ensemble [31] and STGCN [57].”

“…(Kipf and Welling 2017; Hamilton, Ying, and Leskovec 2017; Xu et al. 2019; Veliˇckovi´c et al. 2018; Li et al. 2019b) and action recognition (Yan, Xiong, and Lin 2018; Li et al. 2019c; Shi et al. 2019b,a; Li et al. 2019a; Liu et al. 2020), it became clear that social relations can be also…”

“works [37], [39], [42], which fuse features of every node in different graphs, we only focus on ego node.”

“In [38, 32] the temporal modeling is further improved by tracking feature points or body joints over time.”