Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“ST-GCN [20]: This article proposes a novel model called ST-GCN, which is used to handle dynamic skeletal architectures and compensates for the shortcomings of most previous methods by automatically learning spatial and temporal patterns.”

“Unlike ST-GCN(Yan, Xiong, and Lin 2018), the proposed JTG focuses on constructing the topological structure between joints within a certain spatio-temple period.”

“Our proposed approach FSPN achieves the best average correlation (0.8724) compared with existing approaches, i.e., ST-GCN [11], C3D-LSTM [2], C3D-SVR [2], USDL [5], CoRe [38], adaptive approaches [9] and [31] approaches.”

“[58] 1.”

“He et al. [71] extracted the trajectory, velocity, and acceleration features from a video of human motion and input them into a two-branch ST-GCN [115] to extract global and local features, respectively.”

“To encode the generalized joint positions and forces from the input skeleton sequence, we construct a spatial-temporal graph convolution network (ST-GCN) [59].”

“In this expanded scope, we have incorporated two more GCN-based methods as feature extractors, namely ST-GCN [25] and CTR-GCN [55].”

“In recent years, spatio-temporal networks were used to cope with problems such as traffic prediction [37] and skeleton-based action cognition [38], but those studies did not include the intrinsic relationship cognition for fierce confrontation problems, which often present more complex temporal dynamics.”

“[38], we propose MS-GCN, which is structured as a hierarchy of stacked blocks.”

“M is a learnable parameter matrix measuring the importance between edges, which has the same dimension as A and is initialized to all 1 [64].”

“We build up a temporal convolution network (TCN) module by referring the temporal convolution architecture proposed in [23], [32].”

“The graph convolution (Yan et al., 2018) can be formulated as”

“[10] first applied a graph convolution network to human skeleton action recognition, establishing the spatial graph of natural human connections, added time edges between corresponding joints in continuous time frames, and proposed a spatiotemporal graph convolution neural network.”

“[17] used Spatial Temporal Graph Convolutional Networks (ST-GCN) to recognize bone-based motions.”

“Yan et al. [34] completed skeleton-based action recognition with spatial-temporal graph convolutional networks (STGCNs) that auto-learn spatial and temporal patterns.”

“[34] processed spatial information in the body along with space by operating on bone connections using GCN.”

“The authors in [25] proposed a spatio-temporal graph convolutional network (ST-GCN).”

“Spatial links, as well as the temporal connectivity, are designed to formulate the spatial-temporal graph model [33].”

“The spatial domain methods [36], [44], [45] directly conduct the convolution operation based on receptive fields which contain the central node and its neighbors on the graph.”

“In addition, Yan et al. [12] used graph convolutional networks (GCN), suggesting exploiting both LSTM and graph neural networks to represent temporal and spatial data, respectively.”

“Finally, another interesting direction is to study the proposed explanationbased weakly-supervised method in other domains such as situation recognition [21], video recognition [45], segmentation [35], chemistry [9] and biology [42].”

“…GCN-based (BGC-LSTM [42] ) approach, our method is 1.2% higher with CV and 0.6% higher with CS. Previous P-LSTM [11] 62.9 70.7 methods TS-LSTM [17] 74.6 81.3 Res-TCN [25] 74.3 83.1 VA-LSTM [33] 79.4 87.6 ST-GCN [20] 81.5 88.3 ST-BBMIM [15] 80.0 84.2 EleAtt-GRU [18] 80.7 88.4 BGC-LSTM [42] 81.”

“9,10 However, the application of GCNs in the digital pathology domain is limited.”

“To evaluate the performance of the skeleton-based ST-GCN classification model for fall detection, the UP-Fall dataset containing multiple inertial sensors data was used as the experimental dataset.”

“Inspired by the success of skeleton-based action recognition [23], Zhang et al. proposed a pose skeleton-based model [10] to leverage spatial-temporal graph convolutional network (STGCN) for reasoning the crossing intention.”

“In this work, we use a spatial-temporal graph [41] to represent the relations of multiple joints and frames.”

“It serves as a fundamental task for action recognition [1], [2], [3], [4], [5], [6], human-robot interactions [7], [8], etc.”

“After the graph convolution operations capture neighboring information in the spatial dimensions, we apply a 2D convolution layer, as reported in [47], to extract the temporal features by merging the information at the neighboring time slice.”

“Graph convolutional networks are currently widely employed in node embedding [37], node classification [38], node clustering [39], graph classification [40], and link prediction [41].”

“Recently, many researchers used Graph Convolutional Networks (GCNs) [39] to address the challenges of pose estimation [9, 41, 43, 4] and shape reconstruction [2, 28, 38].”

“It is relevant to a variety of applications, such as action recognition [34], human-machine interaction [40], and sign language understanding [19].”

“In addition, the GCN-based [27, 29, 30] action recognition approaches achieve applaud performance due to GCNs are specialize on finding spatio-temporal information in raw 3D skeleton sequence.”

“represents the physical structure of the human body [57] or a fully-learned adjacency matrix without supervision of graph representations [43].”

“Inspired by [25], we address the challenging problem of CSLR from videos by applying a skeleton-based action recognition method that relies on ST-GCNs.”

“• Spatial-temporal similarity-based models: Based on the similarity among sub-nets in the spatial-temporal domain, graph convolutional network (GCN) based models are employed to distill spatial-temporal features for traffic forecasting [25], [26], weather forecasting [27], skeletonbased action recognition [28], and ma…”

“To reduce the computational cost of the visual stream, we propose to use face landmarks together with a spatio-temporal graph CNN [39].”

“[242] presented a skeleton-based action recognition model using a spatial–temporal graph convolutional network.”

“Of particular note, for this paper, are methods which translate models used for video understanding [29], especially graph convolution networks (GCN) used in skeleton-based action recognition [5].”

“Graph neural networks (GNNs) have the advantage of aggregating node neighborhood contexts to generate spatial representations, which is earliest method introduced to spatiotemporal graph modeling [3], [11], [21].”

“With the rise of deep learning, more researchers turned their attention to neural networks, giving birth to many classical methods based on CNNs[26], RNNs[27] and GCNs[8, 9, 7, 28, 29, 30].”

“Additionally, GCNs process human joints’ graphs for activity recognition [4, 22, 39].”

“Our method involves training a content classifier as a feature extractor using [82].”

“In addition, most existing methods cannot provide informative feedback about motion defects, so it is difficult to obtain effective guidance for patients to improve their exercise performance [26], [27], [28], [29], [30], [31], [32], [33], [34].”

“Graph convolutional networks (GCN) [25, 40] can capture the underlying structures and encode how nodes are connected.”

“ST-GCN [7] 81.5 88.3 AAAI’18 DGNN [13] 89.9 96.1 CVPR’19 DC-GCN+ADG [15] 90.8 96.6 ECCV’20 Shift-GCN [4] 90.7 96.5 CVPR’20 MS-G3D [16] 91.5 96.2 CVPR’20 Dynamic GCN [17] 91.5 96.0 ACMMM’20 SRCA [5] 89.4 94.7 ICME’21 GCHN [2] 90.6 96.4 ICME’21 CTR-GCN [3] 92.4 96.8 ICCV’21 MKE-GCN (Ours) 92.5 96.9 -…”

“We broadly classify the methods into RNN-based methods [10, 12–15], CNN-based methods [19–21] and GCN-based methods [22–32, 34–38, 46–51], and distinguish them with a line.”

“On the other hand, GNN-based methods (GCRN [10], ST-GCN [11], CGCN [8]) enable the modeling of complex relationships between entities through the use of graphs.”

“Yan et al. [36] introduced the graph convolution operation to skeleton-based action recognition and proposed a novel spatial-temporal graph convolutional network to learn the spatial and temporal pattern from skeleton data automatically.”

“Joint-based action recognition [10], [11], [12], [13], [14] acquires reliable 3D joints with Kinect or similar RGB-D sensors and generates action predictions based on the joint sequences.”

“GNN was also used to model the different parts of a human or other objects for action recognition [45] and object tracking [8].”

“The typical applications include road traf­ fic forecasting [7]—[9], wind speed forecasting [10], action recognition [11], and driver maneuver anticipation [12].”

“Although deep learning methods have advanced rapidly in the field of skeleton-based human action recognition, with many excellent approaches emerging, these methods primarily focus on proposing different network architectures for single homogeneous data.”

“NTU-RGB-D Methods ETRI Wang et al.[45] 84.2% Xie et al. [48] 82.4% Xu et al. [46] 85.1% Xu et al. [46] 83.0% Li et al. [47] 86.5% Li et al. [49] 83.3% Jang et al. [8] 88% Yan et al. [50] 86.8% Proposed System = 89.5% Proposed System = 87.5%”

“ST-GCN [47], the first to apply a spatio-temporal graph convolutional network for action recognition, encodes motions across the human kinetic chain.”

“TSM [13] RGB 58.77 VSwin-T [14] RGB 59.97 VSwin-S [14] RGB 57.83 VSwin-B [14] RGB 61.73 ST-GCN [15] Skeleton ( 𝐽 ) 46.38 ST-GCN++ [16] Skeleton ( 𝐽 ) 49.56 StrongAug [16] Skeleton ( 𝐽 ) 53.13 AAGCN [17] Skeleton ( 𝐽 ) 54.73 CTR-GCN [18] Skeleton ( 𝐽 ) 53.02 DG-STGCN [19] Skeleton ( 𝐽 ) 49.56…”

“1) Transformer for Temporal Modeling: In temporal information modeling, previous works [26] used temporal convo-lutions with fixed kernel sizes.”

“To extract both the spatial and temporal structural features from skeleton data, Yan et al. [22] proposed spatial-temporal graph convolutional networks.”

“3D skeleton-based action recognition pipelines often use GCNs (Kipf amb Welling, 2017), e.g. , spatio-temporal GCN (ST-GCN) (Yan et al., 2018), Attention enhanced Graph Convolutional LSTM network (AGC-LSTM) (Si et al., 2019), Actional-Structural GCN (AS-GCN) (Li et al., 2019), Dynamic Directed GCN…”

“We also test the FST-GCN pathway based on the whole body skeleton graph and the spatial configuration partitioning strategy [24], namely FST-GCN (25 joints).”

“Since our model is built on the ST-GCN [58], we only compare computational cost and model parameters with the ST-GCN.”

“To avoid the computational efficiency issue, UGCN [43] adopts the spatial-temporal graph convolution proposed by [46] to model motion in multiple time scales in the input sequence.”

“Specifically, the feature representation learner includes nine layers of SGCN units (SGCN units) [25], as illustrated in Fig.”

“While for skeleton tasks, both GCN [48, 49, 50] and 2DCNN [51, 52] are widely used.”

“It is a fundamental task of action recognition [5,10,47], 3D human pose estimation [16,25,29,32,55], etc.”

“Based on [16], the spatial GCN calculation formula of the skeleton sequence at the t th frame can be expressed as in (1): where f in ( · ) and f out ( · ) denote the mapping rules of input and output, respectively.”

“A is the skeleton graph proposed in [13] More higher order polynomial is also used in many works, such as [1], [30], and [32].”

“[37], instead of providing much complicated and costly appearance features or not easily accessible AU features, we provide a simple and much efficient way that directly takes landmark coordinate-based geometric features as node features.”

“Hence, we adopt the estimated poses provided by [68] extracted by OpenPose [1] to study the skeletonbased UDA task.”

“One state of the art example of this category is ST-GCN (Yan et al. 2018).”

“i) As a means of general video representation, deeplearning-based action recognition can be generally summarized to stream-based methods [10, 12, 17, 19, 26, 52, 65, 79,87,88,90,104] and skeleton-based [21,80,92,98] methods.”

“Inspired by [70] in 2D pose recognition, we first use a graph convolution layer to learn intra-skeleton joint features.”

“OurSequentialPointNet achievesastate-of-the-artperformanceamongallmethodson thecross-viewtestsettingandresultsinsimilarrecognition accuracy as PSTNet on the cross-subject test setting.”