Deep learning for time series classification: a review

In the paper

“Finally, we analyze how poor random initializations can have a significant effect on a DNN's performance.”

In the paper

“We can see that FCN does not hold any pooling nor a regularization operation.”

In the paper

“Similarly to the FCN model, the layers (except the final one) in the ResNet architecture have an invariant number of parameters across different datasets.”

In the paper

“The ResNet significantly outperforms the other approaches with an average rank of almost 2.”

In the paper

“This suggestion is also supported by the fact that Time-CNN is the smallest model: it contains a very small number of parameters by design with only 18 filters compared to the 512 filters of FCN.”

In the paper

“Again, ResNet and FCN still dominate with not much of a difference.”

In the paper

“our empirical study strongly suggests to use ResNet instead of any other deep learning algorithm - it is the most accurate one with similar runtime to FCN (the second most accurate DNN).”

In the paper

“Although the datasets used in our experiments are relatively small compared to the billions of labeled images (such as ImageNet (Russakovsky et al., 2015) and OpenImages (Krasin et al., 2017) challenges), the deepest networks did reach competitive accuracies on the UCR/UEA archive benchmark.”

In the paper

“The MCNN and t-LeNet architectures yielded very low accuracies with only one win for the Earthquakes dataset.”

In the paper

“FCN wins on 36 datasets whereases Encoder wins only on 17”

In the paper

“ResNet and FCN achieved the worst accuracy (30%) on this dataset while Time-CNN reached the best accuracy (95%).”

In the paper

“However, TWIESN did reach competitive accuracies on several long time series datasets such as reaching a 96.8% accuracy on Meat whose time series length is equal to 448.”

In the paper

“Overall, our results show that, on average, ResNet is the best architecture with FCN and Encoder following as second and third respectively.”

In the paper

“In other words, FCN's learned representation has too many data points close to the decision boundary whereas ResNet's hidden features enables projecting data points further away from the decision boundary.”

In the paper

“which is evident by the relatively low accuracy of both models FCN and ResNet which is equal respectively to 58.7% and 74.4%.”

In the paper

“In other words, FCN's CAM contains much more smoother regions with cyan, green and yellow regions, whereas ResNet's CAM contains more dark red and blue subsequences”

In the paper

“ResNet performed very well in general except for the ECG datasets where it was outperformed by FCN.”

In the paper

“In other words, the weight's initial value can easily decrease the accuracy of FCN whereas ResNet maintained a relatively high accuracy when taking the worst initial weight values.”

In the paper

“We found small variance between the approaches that replace the GAP layer with an FC dense layer (MCDCNN, CNN) which also showed similar performance to TWIESN and MLP.”

In the paper

“We note that both deep learning models (FCN and ResNet) managed to project the data from GunPoint into a linearly separable space”

In the paper

“we observed in our experiments, that DNNs for TSC suffer from a significant decrease (increase) in accuracy when initialized with bad (good) random weights.”

In the paper

“DNNs have indeed revolutionized the field of computer vision especially with the advent of novel deeper architectures such as Residual and Convolutional Neural Networks.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “Deep Convolutional Neural Networks (CNNs) have revolutionized the field of computer vision (Krizhevsky et al., 2012).”

What the sources say

• Not checked yetcites Krizhevsky et al., 2012

In the paper

“To achieve its high accuracy, HIVE-COTE becomes hugely computationally intensive and impractical to run on a real big data mining problem (Bagnall et al., 2017).”

• Also stated in: § 1 Introduction · p. 2 · hedged

  “The approach requires training 37 classifiers as well as cross-validating each hyperparameter of these algorithms, which makes the approach infeasible to train in some situations (Lucas et al., 2018).”

What the sources say

• Not checked yetcites Bagnall et al., 2017 source
• Not checked yetcites Lucas et al., 2018

In the paper

“Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “Following the success of deep neural networks (DNNs) in computer vision, a huge amount of research proposed several DNN architectures to solve natural language processing (NLP) tasks such as machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), learning word embeddings (Mikolov et al., 2013; Mikolov et al., 2013) and document classification (Le and Mikolov, 2014; Goldberg, 2016). DN”

What the sources say

• Not checked yetcites Sutskever et al., 2014
• Not checked yetcites Hinton et al., 2012 source
• Not checked yetcites Le and Mikolov, 2014

In the paper

“Once we have established that a statistical difference exists within the classifiers' performance, we followed the pairwise post-hoc analysis recommended by Benavoli et al. (2016) where the average rank comparison is replaced by a Wilcoxon signed-rank test (Wilcoxon, 1945) with Holm's alpha correction (Holm, 1979; Garcia and Herrera, 2008).”

What the sources say

• Not checked yetcites Benavoli et al., 2016
• Not checked yetcites Wilcoxon, 1945 source
• Not checked yetcites Holm, 1979

In the paper

“The multivariate evaluation shows another benefit of deep learning models, which is the ability to handle the curse of dimensionality (Bellman, 2010; Keogh and Mueen, 2017) by leveraging different degrees of smoothness in compositional function (Poggio et al., 2017) as well as the parallel computations of the GPUs (Lu et al., 2015).”

What the sources say

• Not checked yetcites Bellman, 2010 source
• Not checked yetcites Poggio et al., 2017 source
• Not checked yetcites Lu et al., 2015

In the paper

“Apart from time series forecasting, we found that these neural networks were rarely applied for time series classification which is mainly due to three factors: (1) the type of this architecture is designed mainly to predict an output for each element (time stamp) in the time series (L¨ angkvist et al., 2014);”

What the sources say

• Not checked yetcites L¨ angkvist et al., 2014

In the paper

“(3) RNNs are considered hard to train and parallelize which led the researchers to avoid using them for computational reasons (Pascanu et al., 2013).”

What the sources say

• Not checked yetcites Pascanu et al., 2013

In the paper

“They were designed to mitigate the challenges of RNNs by eliminating the need to compute the gradient for the hidden layers which reduces the training time of these neural networks thus avoiding the vanishing gradient problem.”

What the sources say

• Not checked yetcites Gallicchio and Micheli, 2017

In the paper

“This is surprising as deep learning has seen very successful applications in the last years.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “After having established the current state-of-the-art of non deep classifiers for TSC (Bagnall et al., 2017), we discuss the success of Deep Learning (LeCun et al., 2015) in various classification tasks which motivated the recent utilization of deep learning models for TSC (Wang et al., 2017b).”

What the sources say

• Not checked yetcites LeCun et al., 2015 source
• Not checked yetcites Wang et al., 2017b

In the paper

“Deep learning approaches for TSC can be separated into two main categories: the generative and the discriminative models (as proposed in L¨ angkvist et al. (2014)).”

What the sources say

• Not checked yetcites L¨ angkvist et al., 2014

In the paper

“As for comparing the classifiers over multiple datasets, we followed the recommendations in Demˇ sar (2006) and used the Friedman test (Friedman, 1940) to reject the null hypothesis.”

What the sources say

• Not checked yetcites Demˇ sar, 2006
• Not checked yetcites Friedman, 1940 source

In the paper

“Generative models usually exhibit an unsupervised training step that precedes the learning phase of the classifier (L¨ angkvist et al., 2014).”

What the sources say

• Not checked yetcites L¨ angkvist et al., 2014

In the paper

“Finally, adding to the huge runtime of HIVE-COTE, the decision taken by 37 classifiers cannot be interpreted easily by domain experts, since researchers already struggle with understanding the decisions taken by an individual classifier.”

In the paper

“MLP does not have any layer whose number of parameters is invariant across time series of different lengths (denoted by # invar in Table 1) which means that the transferability of the network is not trivial: the number of parameters (weights) of the network depends directly on the length of the input time series.”

In the paper

“One final note is that the careful design of Encoder's attention mechanism enabled the invariance across all layers which encouraged the authors to implement a transfer learning approach.”

What the sources say

• Not checked yetcites Serr` a et al., 2018

In the paper

“Residual Networks (ResNets) (Wang et al., 2017b; Geng and Luo, 2018) which add linear shortcut connections for the convolutional layers potentially enhancing the model's accuracy (He et al., 2016).”

What the sources say

• Not checked yetcites He et al., 2016

In the paper

“To measure the performance of the model on the test data (generalization), we adopted the accuracy measure (similarly to Bagnall et al. (2017)).”

What the sources say

• Not checked yetcites Bagnall et al., 2017 source

In the paper

“Although ESNs were originally proposed for time series forecasting, Tanisaro and Heidemann (2016) proposed a variant of ESNs that uses directly the raw input time series and predicts a probability distribution over the class variables.”

What the sources say

• Not checked yetcites Tanisaro and Heidemann, 2016

In the paper

“In addition, one of the main characteristics of this architecture is the replacement of the traditional final FC layer with a Global Average Pooling (GAP) layer which reduces drastically the number of parameters in a neural network while enabling the use of the CAM (Zhou et al., 2016) that highlights which parts of the input time series contributed the most to a certain classification.”

What the sources say

• Not checked yetcites Zhou et al., 2016

In the paper

“The latter approach reached higher accuracy thus benefiting from the transfer learning technique.”

What the sources say

• Not checked yetcites Serr` a et al., 2018

In the paper

“In contrast to feature engineering, end-to-end deep learning aims to incorporate the feature learning process while fine-tuning the discriminative classifier (Nweke et al., 2018).”

What the sources say

• Not checked yetcites Nweke et al., 2018 source

In the paper

“which is probably due to their robustness and the relatively small amount of training time compared to complex architectures such as RNNs or MLPs.”

In the paper

“For memory usage limitations over a single GPU, we left the MTS dataset Performance Measurement System (PeMS) out of our experimentations.”

In the paper

“Unlike ResNet and FCN, this approach does not have much invariant layers (2 out of 4) due to the use of an FC layer instead of a GAP layer, thus increasing drastically the number of parameters needed to be trained which also depends on the length of the input time series.”

In the paper

“This has also been the consensus in the human activity recognition literature, where the accuracy of deep learning methods depends highly on the quality of the extracted features (Nweke et al., 2018).”

What the sources say

• Not checked yetcites Nweke et al., 2018 source

In the paper

“We also evaluated all deep learning models on Baydogan's archive (Baydogan, 2015) that contains 13 MTS classification datasets.”

What the sources say

• Not checked yetcites Baydogan, 2015

In the paper

“However, any deep learning algorithm can benefit from this type of modification.”

In the paper

“That being said, we can easily pre-train a model on a source dataset, then transfer and fine-tune it on a target dataset without having to modify the hidden layers of the network.”

In the paper

“This archive also exhibits datasets with different characteristics such as the length of the time series which, unlike the UCR/UEA archive, varies among the same dataset. This is due to the fact that the datasets in the UCR/UEA archive are already re-scaled to have an equal length among one dataset (Bagnall et al., 2017).”

What the sources say

• Not checked yetcites Bagnall et al., 2017 source

In the paper

“FCNs are mainly convolutional networks that do not contain any local pooling layers which means that the length of a time series is kept unchanged throughout the convolutions.”

In the paper

“Cui et al. (2016) were the first to introduce the Window Slicing (WS) method as a data augmentation technique.”

What the sources say

• Not checked yetcites Cui et al., 2016

In the paper

“Actually, the difference with the usual convolutions (such as in FCNs) is that a linear shortcut is added to link the output of a residual block to its input thus enabling the flow of the gradient directly through these connections, which makes training a DNN much easier by reducing the vanishing gradient effect (He et al., 2016).”

What the sources say

• Not checked yetcites He et al., 2016

In the paper

“The problem with an MLP approach is that temporal information is lost and the features learned are no longer time-invariant.”

In the paper

“We made this choice because we believe that the main goal of deep learning approaches is to remove the bias due to manually designed features (Ord´ o¨ nez and Roggen, 2016), thus enabling the network to learn the most discriminant useful features for the classification task.”

What the sources say

• Not checked yetcites Ord´ o¨ nez and Roggen, 2016

In the paper

“This form of pre-processing has also been used by Ratanamahatana and Keogh (2005) to show that the length of a time series is not an issue for TSC problems.”

What the sources say

• Not checked yetcites Ratanamahatana and Keogh (2005)

In the paper

“By using an FC layer before the softmax classifier, the transferability of this network is limited to the first and second convolutional layers.”

In the paper

“The first characteristic of Time-CNN is the use of the mean squared error (MSE) instead of the traditional categorical cross-entropy loss function, which has been used by all the deep learning approaches we have mentioned so far. Hence, instead of a softmax classifier, the final layer is a traditional FC layer with sigmoid as the activation function, which does not guarantee a sum of probabilities”

In the paper

“Unlike GAP, local pooling introduces invariance to small perturbations in the activation map (the result of a convolution) by taking the maximum value in a local pooling window.”

In the paper

“Since we did not train nor test any of the state-of-the-art non deep learning algorithms, it is possible that we allowed much more training time for the described DNNs.”

In the paper

“our findings are in agreement with the deep learning for computer vision literature where deeper neural networks are much more successful than shallower architectures (He et al., 2016).”

What the sources say

• Not checked yetcites He et al., 2016

In the paper

“Interestingly, DiatomSizeReduction is the smallest datasets in the UCR/UEA archive (with 16 training instances), which suggests that ResNet and FCN are easily overfitting this dataset.”

In the paper

“This step is very important for deep learning models whose architecture depends on the length of the input time series (such as a MLP) and for parallel computation over the GPUs.”

In the paper

“In addition to the huge training time, HIVE-COTE's classification time is bounded by a linear scan of the training set due to employing a nearest neighbor classifier, whereas the trivial GPU parallelization of DNNs provides instant classification.”

In the paper

“since increasing the number of convolutional layers will increase the path length viewed by the CNN model (Vaswani et al., 2017), ResNet and FCN managed to outperform other approaches whose filter length is longer (21) such as Encoder.”

What the sources say

• Not checked yetcites Vaswani et al., 2017

In the paper

“This FC layer reduces the effect of learning time invariant features which explains why MLP, Time-CNN and MCDCNN exhibit very similar performance.”

In the paper

“Therefore, we believe that allowing a much more thorough search of DTW's warping window would lead to a fairer comparison between deep learning approaches and other state-of-the-art TSC algorithms.”

In the paper

“unlike HIVE-COTE, ResNet's hyperparameters were not tuned for each dataset but rather the same architecture was used for the whole benchmark”

In the paper

“One final observation that is in agreement with the deep learning literature is that in order to achieve high accuracies while training a DNN, a large training set is needed.”

In the paper

“suggesting that these methods are no longer competitive with current state-of-the-art algorithms for TSC.”

In the paper

“suggesting further investigation of these hyperparameters should improve DNNs' accuracy for TSC.”

In the paper

“it is only natural to think that discovering patterns in a one dimensional space (time) should be an easier task for CNNs thus requiring less data to learn from.”

In the paper

“which enables us to take the mean accuracy over the 10 runs in order to reduce the bias due to the weights' initial values.”

In the paper

“for a lazy machine learning algorithm such as NN-DTW, training time is zero when allowing maximum warping whereas it has been shown that judicially setting the warping window Dau et al. (2017) can lead to a significant increase in accuracy.”

What the sources say

• Not checked yetcites Dau et al., 2017

In the paper

“These observations reinforces the need to have an equally large MTS classification archive in order to evaluate hybrid univariate/multivariate time series classifiers.”

In the paper

“but the fact that ResNet is not significantly different than COTE suggests that more datasets would give a better insight into these performances (Demˇ sar, 2006).”

What the sources say

• Not checked yetcites Demˇ sar, 2006

In the paper

“It is worthwhile noting that cliques formed by the Wilcoxon Signed Rank Test with Holm's alpha correction do not necessary reflect the rank order (Lines et al., 2018).”

What the sources say

• Not checked yetcites Lines et al., 2018 source

In the paper

“Although Time-CNN and MCDCNN are the only architectures originally proposed for MTS data”

In the paper

“We believe that the success of ResNet is highly due to its deep flexible architecture.”

In the paper

“especially since normalization is known to have a huge effect on DNNs' learning capabilities (Zhang et al., 2017).”

What the sources say

• Not checked yetcites Zhang et al., 2017

In the paper

“which probably explains why FCN is less accurate than ResNet on this dataset.”

In the paper

“In fact, most DNNs architectures, with the categorical cross-entropy as their cost function, employ mainly the same classifier: softmax which is basically designed for multi-class classification.”

In the paper

“we stress that the shapelet transformation classifier (Hills et al., 2014) is an ensemble approach, which makes unclear how the shapelets affect the decision taken by the individual classifiers whereas for an end-to-end deep learning model we can directly explain the classification by using the Class Activation Map.”

What the sources say

• Not checked yetcites Hills et al., 2014 source

In the paper

“On the other hand, the part where the actor raises his/her arm seems to be also a discriminative part of the data which suggests that the deep learning algorithms are identifying more 'shapelets'.”

In the paper

“Finally, another interesting observation is the ability of CNNs to localize a given discriminative shape regardless where it appears in the time series, which is evidence for CNNs' capability of learning time-invariant warped features.”

In the paper

“We should note that this observation cannot confirm which classifier extracted the most discriminative subsequences especially because all algorithms achieved similar accuracy on GunPoint dataset.”