Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

“In tasks such as image object recognition, the hierarchical representation and locality are usually achieved by the intrinsic properties of convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012), rather than manually assigning object parts.”

“In the single frame case, ST-GCN with the first partitioning strategy can be implemented with the following formula (Kipf and Welling 2017) where Here the weight vectors of multiple output channels are stacked to form the weight matrix W .”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”

“Skeleton and joint trajectories of human bodies are robust to illumination change and scene variation, and they are easy to obtain owing to the highly accurate depth sensors or pose estimation algorithms (Shotton et al. 2011; Cao et al. 2017a).”

“More detailed explanation and other applications of this formulation can be found in (Dai et al. 2017).”

“…graphs of arbitrary structures, have received increasing attention and successfully been adopted in a number of applications, such as image classification (Bruna et al. 2014), document classifi-cation (Defferrard, Bresson, and Vandergheynst 2016), and semi-supervised learning (Kipf and Welling 2017).”

“In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al. 2016).”

“Weevaluate the recognition performance by top-1 and top-5 classification accuracy as recommended by the dataset authors (Kay et al. 2017).”

“We train the standard TSN (Wang et al. 2016) models from scratches on Kinetics with RGB and optical flow models.”

“skeleton-based sequences contains compact joints positions information [17], which is robust to variations of viewpoints, body scales, and action speeds.”

“We train a skeleton representation model (Shift-GCN [4] or ST-GCN [26], depending on experimental settings) on the seen classes using standard cross-entropy loss.”

“Traditional pose-based methods aim to extract motion patterns directly from the poses for trajectory-prediction [27, 50], pose estimation [6, 74], action recognition [10, 66].”

“Recently, with the success of GNN in action recognition tasks [35], some text classification researchers have also turned their attention to GNN.”

“Yan et al.[24] presented a method for skeleton motion detection based on Spatio-Temporal graph convolutional neural networks.”

“Spatial-temporal graph convolutional network (STGCN) [41] has been applied in action recognition and achieved good performance.”

“In recent years, graph convolutional networks have also been applied to data containing time series, such as traffic prediction [15], [36], [37], [38], and action recognition [39], [40].”

“For the skeleton feature learning, STGCN [44] first applied GCN in the skeleton-based action recognition, which constructed the edge between spatial and temporal adjacent nodes; inspired by shift CNN, ShiftGCN [45] proposed an efficient GCN model for skeleton-based action recognition, which gained comparable results w…”

“Following the work of ST-GCN (Yan et al., 2018), we construct a spatial-temporal graph, where nodesrepresenthumanjointsandedgesrepresentnaturalconnectionsalongboththespa-tial and temporal dimension.”

“Motion Representation A variety of methods have been proposed to learn (mostly human) motion representation from video frames [3, 23, 28, 53], depth maps [14, 22, 27, 37, 47], keypoints/skeletons [4, 21, 24, 26, 29, 33, 41, 52, 55, 58], or point clouds [6, 7].”

“Recently, to accommodate both graph constitution and graph geometry, there have been efforts to extend GNNs by incorporating 3D/2D node coordinates in graph convolutions [13, 25, 30, 44, 46].”

“It is based on Spatio-temporal Graph Convolution Network (ST-GCN) [43] which encodes the body keypoints to pose features, which are then fed into the Transformer decoder where each block contains the self-attention, cross-attention, and feed-forward modules.”

“To process graphs using deep learning algorithms, graph convolution networks(GCN) have been used for a number of different applications including action recognition [Yan et al., 2018, Ghosh et al., 2020 , 2017].”

“We would like to employ GCN that can encapsulate both spatial and temporal attributes at the same time, e.g. , ST-GCN [48], to model traffic predictions.”

“Deep neural networks (DNNs) have played an astonishing role in the evolution of modern machine learning by achieving state-of-the-art performance on many challenging tasks [22, 41].”

“Based on the question-specific object graph constructed in Section 3.2 and the attention mechanism mentioned above, we can learn object representations by spatial graph convolution networks [36].”

“From the input side, human videos, either from third-person view [24, 50, 54, 55, 58] or first-person view [5–7, 59 ], are used for this task.”

“II, previous supervised methods, i.e., ST-GCN [66] and PSLR [58] show inferior performance compared with RGB-based methods.”

“They utilize the ST-GCN architecture [58] to capture the spatio-temporal features from sequences of skeletons.”

“(Bhattacharya et al. 2020a) propose a classifier network based on Spatial Temporal Graph Convolutional Network (ST-GCN) (Yan, Xiong, and Lin 2018) to detect human emotion from gait.”

“STGCN uses GCN for the spatial domain and 1D dilated convolutional operations for the temporal domain, but may lose local information due to the dilation operation, while its absence of attention-based operations and limited capability of convolutional operations in modeling high-dimensional spatial, temporal, and spa…”

“…variants where ‘LORIS w/o V’ and ‘LORIS w/o R’ drop the visual and rhythm conditioning, respectively, and ‘LORIS w/ V m ’ denotes using a trainable ST-GCN (Yan et al., 2018) network to encode human poses and then con-Method catenate the extracted motion features with I3D features as the visual…”

“[23] presented a spatialtemporal GCN (ST-GCN) to recognize the human action from the dynamic data of human skeletons.”

“In recent years, Graph Neural Networks (GNNs) [43] and the variants [9], [44], [45], [46] have been applied to a series of tasks, including image classification [47], document classification [44], and action recognition [48], [49] due to its robust performance in relation modeling.”

“With the prevalence of deep learning techniques [1, 2, 3], MPPE has achieved remarkable progress and played an important role in many other vision tasks, such as activity recognition [4, 5, 6, 7] and pose tracking [8, 9].”

“Compared to the GCN&RNN pipeline and STGCN [13], our RGCN has the following advantages: 1) the hidden state Ht of RGCN in Eq.”

“We compare our method with other self-supervised methods and some supervised methods, such as ST-GCN, which has all the same structure and parameters as ours.”

“After rearranging the coordinates of the joints, we put the preliminary features into ST-GCN [4] and designed convolutional layers separately for refining and formatting, and the concatenation of the outputs of two-stream network is then fed into the classifier to obtain the final result.”

“In existing graph-centered networks [43], [45], [46], the topology structure of graph reflected by the node connectivity A is unweighted and artificially defined.”

“HUMAN poses or named as human skeletons convey significant information, they facilitate many high-level computer vision tasks: human action recognition [1], [2], video”

“There are many supervised representation learning methods that achieve good performances for downstream tasks [25, 22, 30, 60] on several data domains such as images and graphs.”

“For the Cross-Subject accuracy, this work’s accuracy is lower than some research [16], [55].”

“STGCN [51]: It designs multi-scale convolutional filters, each of which is composed of receptive field computation and signal mapping, to encode dynamic graphs.”

“…methods, including AAC [11], MMAPM [18], IBOW [17], DBOW [17], DeepSCN [4], Lai et al. [5], MSSC [20], Lan et al. [21], Yuge et al. [24], Harshala et al. [25], MTSSVM [16] LieNet [60], ST-LSTM [61], AAPNet [62], PA-DRL [22], RGN-KF [63], I3D [64], LieNet [60], ST-GCN [65], and AS-GCN [66].”

“NTU-RGBD.”

“For ST-GCN, follow the [66], we utilize its official toolbox(1) to generate skeleton locations for frames using OpenPose.”

“Many works have been proposed with different input modalities, including RGB video [1], [2], [9], [10], optical flow [4], [11] and human 2D/3D skeletons [8], [12], [13] (the optical flow and human skeletons can be estimated directly from the RGB video).”

“Many remarkable papers are worthy to refer, such as graph attention network [24], In the graph neural network area, the most related work is spatio-temporal graph convolutional neural network [25]–[27].”

“Both feed-forward neural networks such as CNNs [38], [41], [75] and several variants of recurrent neural networks”

“Yan et al. [27] firstly proposed the spatial-temporal GCN model (ST-GCN) for the action recognition.”

“Drawing inspiration from skeleton-based action recognition models [13, 60], we propose a graph-based IMU modeling technique, where we construct a graph with nodes as IMU features.”

“2D skeleton data has been widely applied to tasks relating to human behavior understanding, such as action recognition [41, 42], action quality assessment [24, 25] and beyond.”

“Some works apply Kalman ltering [13, 29] or stacked spatial-temporal graph convolution networks (STGCN) [41] to create dynamic graph embeddings, and then use simple MLPs as decoders to perform the prediction task.”

“For the person split, VA-CNN and AGCN perform similarly well, with ST-GCN being slightly worse.”

“Compared to supervised models that heavily rely on action labels [1, 2, 3], recognition without manual labeling is considered more efficient and more comprehensive to learn representative features with large-scale data.”

“Yan et al. [43] provided a novel view to aggregate the spatial and temporal features of skeleton joints for action recognition.”

“DCNN for image classification, action recognition applications utilizing ST-GCN for processing the skeleton data obtained from RGB-D sensors [23], [24].”

“, action recognition [7,46,61], human pose estimation [6,28,65,68,71] and hand pose estimation [2, 9, 11, 14, 25].”

“There are three partitioning strategies proposed in [21]:”

“…achieved state-of-the-art performance for tasks in different domains such as point clouds classification and action recognition in computer vision [27]–[29], recommendation and spam detection in social networks [30]–[32], and molecular fingerprints and drug discovery in chemistry [24], [33], [34].”

“Params FLOPs Two-stream 3D CNN [37] 0.91M 1.53G SEM-MEM+WAL [38] 2.36M 1.35G ST-GCN [4] 1.73M 1.68G Liu et al. [5] 2.05M 1.46G HAN 0.53M 0.04G HAN-2S 0.94M 0.05G based on manifold learning with SVM classifier, our method is suboptimal.”

“For example, in skeleton-based human action recognition, we can model a sequence of 3D joint locations as data supported on skeleton graphs across time, which reflects both the human physical constraints and temporal consistency (Yan et al. (2018)).”

“In order to regress the scores based on the extracted joint sequences (taken from the pose estimation module trained on our dataset), we develop the wellknown Spatial Temporal Graph Convolutional Networks (ST-GCN) [19] to also learn dependencies between unconnected joints.”

“redicate,object> triplet. Whilst computer vision has experienced extraordinary advances in object detection [6,21,29], human pose estimation [5,25], scene segmentation [13], and action recognition [17,38]; the harder problem of HOI has progressed less so, as it is a more a b Instance Detection Interaction Inference…”

“A pre-trained graph convolutional network for skeleton-based action recognition [29] classified the scene as feeding birds because of the absence of training material”

“The model consists of ST-GCN [60] and BiLSTM layers and segments sign language video into subtitle units.”

“There are many techniques for spatio-temporal feature extraction, such as 3D Pose Estimation [3], 3D Convolutional Neural Networks [4], and Graph-Based Approaches [5].”

“(Ghaleb et al. 2021) proposed a classifier network based on Spatial-Temporal Graph Convolutional Networks (Yan, Xiong, and Lin 2018) to recognize emotions from body movements.”

“The cross-modal analysis of 3D human motion and natu-ral language has opened up new avenues for tasks such as motion recognition [10, 38, 46, 49, 50, 52, 54, 60] and text-to-motion synthesis [3, 36, 37], which can benefit the applications like animating avatars or humans [18, 53].”

“Their common practice is extracting visual features from skeleton sequences with ST-GCN (Yan et al., 2018) or Shift-GCN (Cheng et al., 2020) and obtaining semantic embeddings from category names or action descriptions with Word2Vec (Mikolov et al., 2013b) or Sentence-Bert (Reimers & Gurevych, 2019).”

“They have been successfully applied to many files like action recognition [26], [29], [33], [35], [41], [42], [43], [44], [45], [46], [47], [48], action localization [49] and motion prediction [50], [51], [52].”

“Table 4 compares the performance with the other well-known machine learning models of LSTM, KNN, Decision Tree, AdaBoost, and ST–GCN.”

“Our framework takes the mask of an avatar pose as the input image for the full-body gesture recognition program [34] to compute the result.”

“To encode a raw data sample as a graph, the majority of existing approaches [11], [13], [15], [16] manually define the vertices, the topology of the graph (i.e., edges’ presence) Manuscript received September 19, 2022. as well as each edge’s features based on pre-defined rules.”

“Graph convolutional networks [10] arewidely used for non-Euclidean structures, which have recently become increasingly popular in computer vision, such as skeleton-based action recognition [35], person re-identification [25], and video classification [33].”

“As a long-standing problem, human-centric perception has been studied for decades, ranging from sparse prediction tasks, such as human action recognition [8, 27, 42, 50], 2D keypoints detection [2, 26, 43, 48] and 3D pose estimation [22, 31, 40], to dense prediction tasks, such as human parsing [7, 11, 12, 25] and Den…”

“In particular, the ST-GCN model proposed by Yan et al. [17] is the first construct, a spatial-temporal graph that offers a new partitioning strategy.”

“[42] proposed the spatiotemporal convolution network model ST-GCN, it solves the problem of human motion recognition which is based on the key points of the human skeleton.”