The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances
Open accessAbstract
In the last 5 years there have been a large number of new time series classification algorithms proposed in the literature. These algorithms have been evaluated on subsets of the 47 data sets in the University of California, Riverside time series classification archive. The archive has recently been expanded to 85 data sets, over half of which have been donated by researchers at the University of East Anglia. Aspects of previous evaluations have made comparisons between algorithms difficult. For example, several different programming languages have been used, experiments involved a single train/test split and some used normalised data whilst others did not. The relaunch of the archive provides a timely opportunity to thoroughly evaluate algorithms on a larger number of datasets. We have implemented 18 recently proposed algorithms in a common Java framework and compared them against two standard benchmark classifiers (and each other) by performing 100 resampling experiments on each of the 85 datasets. We use these results to test several hypotheses relating to whether the algorithms are significantly more accurate than the benchmarks and each other. Our results indicate that only nine of these algorithms are significantly more accurate than both benchmarks and that one classifier, the collective of transformation ensembles, is significantly more accurate than all of the others. All of our experiments and results are reproducible: we release all of our code, results and experimental details and we hope these experiments form the basis for more robust testing of new algorithms in the future.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HIVE-COTE is currently considered the state-of-the-art algorithm for TSC when evaluated over the 85 datasets from the UCR/UEA archive.
“HIVE-COTE is currently considered the state-of-the-art algorithm for time series classification (Bagnall et al., 2017) when evaluated over the 85 datasets from the UCR/UEA archive.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
HIVE-COTE is hugely computationally intensive and impractical to run on a real big data mining problem.
“To achieve its high accuracy, HIVE-COTE becomes hugely computationally intensive and impractical to run on a real big data mining problem (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
Only a few TSC methods have considered Deep Neural Networks (DNNs) to perform TSC.
“Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
The paper used a common framework in Python, Keras and Tensorflow to train deep learning models on a cluster of more than 60 GPUs for fair comparison.
“Similarly to Bagnall et al. (2017), in order to have a fair comparison between the tested approaches, we developed a common framework in Python, Keras (Chollet, 2015) and Tensorflow (Abadi et al., 2015) to train the deep learning models on a cluster of more than 60 GPUs.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
The time series in the UCR/UEA archive are already z-normalized.
“Note that the time series in this archive are already z-normalized (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
This archive also exhibits datasets with different characteristics such as the length of the time series which, unlike the UCR/UEA archive, varies among the same dataset. This is due to the fact that the datasets in the UCR/UEA archive are already re-scaled to have an equal length among one dataset.
“This archive also exhibits datasets with different characteristics such as the length of the time series which, unlike the UCR/UEA archive, varies among the same dataset. This is due to the fact that the datasets in the UCR/UEA archive are already re-scaled to have an equal length among one dataset (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
We tested each algorithm on the whole UCR/UEA archive which contains 85 univariate time series datasets.
“In order to have a thorough and fair experimental evaluation of all approaches, we tested each algorithm on the whole UCR/UEA archive (Chen et al., 2015b; Bagnall et al., 2017) which contains 85 univariate time series datasets.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
The informal consensus in the literature is that generative models are usually less accurate than direct discriminative models.
“The informal consensus in the literature is that generative models are usually less accurate than direct discriminative models (Bagnall et al., 2017; Nguyen et al., 2017);”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
The paper adopted accuracy to measure model performance on test data.
“To measure the performance of the model on the test data (generalization), we adopted the accuracy measure (similarly to Bagnall et al. (2017)).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
Given the small sample size of only 7 ECG datasets, one cannot conclude that FCN will almost always outperform the ResNet model for ECG datasets.
“However, given the small sample size (only 7 datasets), we cannot conclude that FCN will almost always outperform the ResNet model for ECG datasets (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
Euclidean Distance is usually not able to strongly separate raw time series, as evident by the low accuracy of nearest neighbor when coupled with ED.
“This straightforward approach supposes that the ED is able to strongly separate the raw time series, which is usually not the case evident by the low accuracy of the nearest neighbor when coupled with ED (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · see citing works
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 11,966 · see citing works
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · see citing works
Experiencing SAX: A novel symbolic representation of time series
Lin, J., Keogh, E., Wei, L., & Lonardi, S. (2007) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-007-0064-z · Cited by 1,618 · see citing works
Self-citationQuerying and mining of time series data
Ding, H., Trajcevski, G., Scheuermann, P., Wang, X., & Keogh, E. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454226 · Cited by 1,353 · see citing works
Self-citationGarcía, S., & Herrera, F. (2008) · Cited by 1,277 · see citing works
On the Need for Time Series Data Mining Benchmarks: A Survey and Empirical Demonstration
Keogh, E., & Kasetty, S. (2003) · Data Mining and Knowledge Discovery · doi:10.1023/a:1024988512476 · Cited by 860 · see citing works
Self-citationExperimental comparison of representation methods and distance measures for time series data
Wang, X., Mueen, A., Ding, H., Trajcevski, G., Scheuermann, P., & Keogh, E. (2012) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-012-0250-5 · Cited by 846 · see citing works
Self-citationOn The Marriage of Lp-norms and Edit Distance
Chen, L., & Ng, R. T. (2004) · Elsevier eBooks · doi:10.1016/b978-012088469-8.50070-x · Cited by 792 · see citing works
Algorithms for the Longest Common Subsequence Problem
Hirschberg, D. S. (1977) · Journal of the ACM · doi:10.1145/322033.322044 · Cited by 752 · see citing works
Weighted dynamic time warping for time series classification
Jeong, Y., Jeong, M. K., & Omitaomu, O. A. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.022 · Cited by 697 · see citing works
A time series forest for classification and feature extraction
Deng, H., Runger, G. C., Tuv, E., & Vladimir, M. (2013) · Information Sciences · doi:10.1016/j.ins.2013.02.030 · Cited by 632 · see citing works
Word image matching using dynamic time warping
Rath, T. M., & Manmatha, R. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2003.1211511 · Cited by 584 · see citing works
The BOSS is concerned with time series classification in the presence of noise
Schäfer, P. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0377-7 · Cited by 564 · see citing works
Classification of time series by shapelet transformation
Hills, J., Lines, J., Baranauskas, E., Mapp, J., & Bagnall, A. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-013-0322-1 · Cited by 538 · see citing works
Self-citationTime series classification with ensembles of elastic distance measures
Lines, J., & Bagnall, A. J. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0361-2 · Cited by 513 · see citing works
Self-citationLearning time-series shapelets
Grabocka, J., Schilling, N., Wistuba, M., & Schmidt-Thieme, L. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623613 · Cited by 489 · see citing works
Time-Series Classification with COTE: The Collective of Transformation-Based Ensembles
Bagnall, A. J., Lines, J., Hills, J., & Bostrom, A. G. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2416723 · Cited by 420 · see citing works
Self-citationFast Shapelets: A Scalable Algorithm for Discovering Time Series Shapelets
Rakthanmanon, T., & Keogh, E. (2013) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972832.74 · Cited by 408 · see citing works
Self-citationTime series shapelets: a novel technique that allows accurate, interpretable and fast classification
Ye, L., & Keogh, E. (2011) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0179-5 · Cited by 406 · see citing works
Self-citationLocally adaptive dimensionality reduction for indexing large time series databases
Chakrabarti, K., Keogh, E., Mehrotra, S., & Pazzani, M. J. (2002) · ACM Transactions on Database Systems · doi:10.1145/568518.568520 · Cited by 404 · see citing works
Self-citationThree Myths about Dynamic Time Warping Data Mining
Ratanamahatana, C. A., & Keogh, E. (2005) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972757.50 · Cited by 398 · see citing works
Self-citationClustering Sequences with Hidden Markov Models
Smyth, P. (1996) · Cited by 381 · see citing works
A Bag-of-Features Framework to Classify Time Series
Baydogan, M. G., Runger, G., & Tuv, E. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.72 · Cited by 361 · see citing works
CID: an efficient complexity-invariant distance for time series
Batista, G. E. A. P. A., Keogh, E., Tataw, O. M., & Souza, V. M. A. (2013) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-013-0312-3 · Cited by 361 · see citing works
Self-citationUsing dynamic time warping distances as features for improved time series classification
Kate, R. J. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0418-x · Cited by 347 · see citing works
Highly Comparative Feature-Based Time-Series Classification
Fulcher, B., & Jones, N. S. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2316504 · Cited by 346 · see citing works
Time Warp Edit Distance with Stiffness Adjustment for Time Series Matching
Marteau, P.-F. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.76 · Cited by 337 · see citing works
Should we really use post-hoc tests based on mean-ranks?Preprint
BenavoliAlessio, CoraniGiorgio, & MangiliFrancesca (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02288 · Cited by 318 · see citing works
A database of Caenorhabditis elegans behavioral phenotypes
Yemini, E., Jucikas, T., Grundy, L. J., Brown, A. E., & Schafer, W. R. (2013) · Nature Methods · doi:10.1038/nmeth.2560 · Cited by 309 · see citing works
Rotation-invariant similarity in time series using bag-of-patterns representation
Lin, J., Khade, R., & Li, Y. (2012) · Journal of Intelligent Information Systems · doi:10.1007/s10844-012-0196-5 · Cited by 292 · see citing works
Mueen, A., Keogh, E., & Young, N. E. (2011) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020587 · Cited by 282 · see citing works
Self-citationSAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model
Senin, P., & Malinchik, S. (2013) · Proceedings · doi:10.1109/icdm.2013.52 · Cited by 254 · see citing works
Addressing Big Data Time Series
Rakthanmanon, T., Campana, B. J. L., Mueen, A., Batista, G., Westover, M. B., Zhu, Q., Zakaria, J., & Keogh, E. (2013) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2500489 · Cited by 245 · see citing works
Self-citationTime series clustering and classification by the autoregressive metric
Corduas, M., & Piccolo, D. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2007.06.001 · Cited by 199 · see citing works
The Move-Split-Merge Metric for Time Series
Stefan, A., Athitsos, V., & Das, G. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.88 · Cited by 189 · see citing works
Deformable Markov model templates for time-series pattern matching
Ge, X., & Smyth, P. (2000) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/347090.347109 · Cited by 179 · see citing works
Time series representation and similarity based on local autopatterns
Baydoğan, M. G., & Runger, G. C. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0425-y · Cited by 176 · see citing works
Maharaj, E. A. (2000) · Journal of Classification · doi:10.1007/s003570000023 · Cited by 171 · see citing works
Using derivatives in time series classification
Górecki, T., & Łuczak, M. (2012) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-012-0251-4 · Cited by 153 · see citing works
Model-based kernel for efficient time series analysis
Chen, H., Tang, F., Peter Tiňo, B., & Yao, X. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2487700 · Cited by 108 · see citing works
Time Series Classification Using Compression Distance of Recurrence Plots
Silva, D. F., Souza, V. M. A., & Batista, G. E. A. P. A. (2013) · Proceedings · doi:10.1109/icdm.2013.128 · Cited by 103 · see citing works
Non-isometric transforms in time series classification using DTW
Górecki, T., & Łuczak, M. (2014) · Knowledge-Based Systems · doi:10.1016/j.knosys.2014.02.011 · Cited by 95 · see citing works
Support vector machines of interval-based features for time series classification
Rodríguez, J. J., Alonso, C. J., & Maestro, J. A. (2005) · Knowledge-Based Systems · doi:10.1016/j.knosys.2004.10.007 · Cited by 90 · see citing works
Binary Shapelet Transform for Multiclass Time Series Classification
Bostrom, A. G., & Bagnall, A. J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22729-0_20 · Cited by 80 · see citing works
Self-citationSAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model
Senin, P., & Malinchik, S. (2013) · doi:10.21236/ada603196 · Cited by 65 · see citing works
Automatic segmentation of diatom images for classification
Jalba, A. C., Wilkinson, M. H. F., & Roerdink, J. B. T. M. (2004) · Microscopy Research and Technique · doi:10.1002/jemt.20111 · Cited by 49 · see citing works
A Run Length Transformation for Discriminating Between Auto Regressive Time Series
Bagnall, A. J., & Janacek, G. J. (2013) · Journal of Classification · doi:10.1007/s00357-013-9135-6 · Cited by 39 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342 · see citing works
“However, given the small sample size (only 7 datasets), we cannot conclude that FCN will almost always outperform the ResNet model for ECG datasets (Bagnall et al., 2017).”
BackgroundMethodResultInfluentialInceptiontime: finding alexnet for time series classification
Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., & Petitjean, F. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00710-y · Cited by 1,788 · see citing works
A Transformer-based Framework for Multivariate Time Series Representation Learning
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., & Eickhoff, C. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467401 · Cited by 1,278 · see citing works
Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., & Keogh, E. (2019) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2019.1911747 · Cited by 1,128 · see citing works
Self-citationDempster, A., Petitjean, F., & Webb, G. I. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00701-z · Cited by 1,080 · see citing works
k-Nearest Neighbour Classifiers - A Tutorial
Cunningham, P., & Delany, S. J. (2021) · ACM Computing Surveys · doi:10.1145/3459665 · Cited by 1,033 · see citing works
Theissler, A., Pérez-Velázquez, J., Kettelgerdes, M., & Elger, G. (2021) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2021.107864 · Cited by 581 · see citing works
Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
Strodthoff, N., Wagner, P., Schaeffter, T., & Samek, W. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3022989 · Cited by 540 · see citing works
Dempster, A., Schmidt, D. F., & Webb, G. I. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467231 · Cited by 526 · see citing works
Ruiz, A. P., Flynn, M., Large, J., Middlehurst, M., & Bagnall, A. J. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00727-3 · Cited by 480 · see citing works
Self-citationcatch22: CAnonical Time-series CHaracteristics
Lubba, C. H., Sethi, S. S., Knaute, P., Schultz, S. R., Fulcher, B., & Jones, N. S. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00647-x · Cited by 447 · see citing works
TapNet: Multivariate Time Series Classification with Attentional Prototypical Network
Zhang, X., Gao, Y., Lin, J., & Lu, C.-T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6165 · Cited by 352 · see citing works
Forecast evaluation for data scientists: common pitfalls and best practices
Hewamalage, H., Ackermann, K., & Bergmeir, C. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00894-5 · Cited by 344 · see citing works
HIVE-COTE 2.0: a new meta ensemble for time series classification
Middlehurst, M., Large, J., Flynn, M., Lines, J., Bostrom, A., & Bagnall, A. J. (2021) · Machine Learning · doi:10.1007/s10994-021-06057-9 · Cited by 343 · see citing works
Self-citationData Augmentation for Time Series Classification using Convolutional Neural Networks
Guennec, A. L., Malinowski, S., & Tavenard, R. (2016) · HAL · Cited by 341 · see citing works
Lines, J., Taylor, S., & Bagnall, A. (2018) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3182382 · Cited by 333 · see citing works
Self-citationTransfer learning for time series classification
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2018) · 2018 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2018.8621990 · Cited by 319 · see citing works
Fast and Accurate Time Series Classification with WEASEL
Schäfer, P., & Leser, U. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3132980 · Cited by 308 · see citing works
Unsupervised Scalable Representation Learning for Multivariate Time Series
Franceschi, J.-Y., Dieuleveut, A., & Jaggi, M. (2019) · arXiv · doi:10.48550/arxiv.1901.10738 · Cited by 305 · see citing works
Mobile sensor data anonymization
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2019) · Proceedings of the International Conference on Internet of Things Design and Implementation · doi:10.1145/3302505.3310068 · Cited by 298 · see citing works
Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey
Foumani, N. M., Miller, L., Tan, C. W., Webb, G. I., Forestier, G., & Salehi, M. (2024) · ACM Computing Surveys · doi:10.1145/3649448 · Cited by 296 · see citing works
A review on distance based time series classification
Abanda, A., Mori, U., & Lozano, J. A. (2018) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-018-0596-4 · Cited by 295 · see citing works
shapeDTW: Shape Dynamic Time Warping
Zhao, J., & Itti, L. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.09.020 · Cited by 294 · see citing works
Tan, C. W., Dempster, A., Bergmeir, C., & Webb, G. I. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00844-1 · Cited by 255 · see citing works
Improving position encoding of transformers for multivariate time series classification
Foumani, N. M., Tan, C. W., Webb, G. I., & Salehi, M. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00948-2 · Cited by 254 · see citing works
sktime: A Unified Interface for Machine Learning with Time SeriesPreprint
Löning, M., Bagnall, A. J., Ganesh, S., Kazakov, V., Lines, J., & Király, F. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07872 · Cited by 247 · see citing works
Self-citationMulti-scale Attention Convolutional Neural Network for time series classification
Chen, W., & Shi, K. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.01.001 · Cited by 245 · see citing works
Electricity price forecasting on the day-ahead market using machine learning
Tschora, L., Pierre, E., Plantevit, M., & Robardet, C. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.118752 · Cited by 233 · see citing works
Joint Activity Recognition and Indoor Localization With WiFi Fingerprints
Wang, F., Feng, J., Zhao, Y., Zhang, X., Zhang, S., & Han, J. (2019) · IEEE Access · doi:10.1109/access.2019.2923743 · Cited by 230 · see citing works
Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series
Bianchi, F. M., Scardapane, S., Løkse, S., & Jenssen, R. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3001377 · Cited by 229 · see citing works
TS-CHIEF: a scalable and accurate forest algorithm for time series classification
Shifaz, A., Pelletier, C., Petitjean, F., & Webb, G. I. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00679-8 · Cited by 216 · see citing works
Bake off redux: a review and experimental evaluation of recent time series classification algorithms
Middlehurst, M., Schäfer, P., & Bagnall, A. J. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-024-01022-1 · Cited by 216 · see citing works
Self-citationFast and Accurate Time-Series Clustering
Paparrizos, J., & Gravano, L. (2017) · ACM Transactions on Database Systems · doi:10.1145/3044711 · Cited by 213 · see citing works
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions
Theissler, A., Spinnato, F., Schlegel, U., & Guidotti, R. (2022) · IEEE Access · doi:10.1109/access.2022.3207765 · Cited by 211 · see citing works
Detecting and interpreting myocardial infarction using fully convolutional neural networks
Strodthoff, N., & Strodthoff, C. (2018) · Physiological Measurement · doi:10.1088/1361-6579/aaf34d · Cited by 206 · see citing works
Methods and tools for causal discovery and causal inference
Nogueira, A. R., Pugnana, A., Ruggieri, S., Pedreschi, D., & da Gama, J. M. P. (2022) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1449 · Cited by 206 · see citing works
Porumb, M., Stranges, S., Pescapè, A., & Pecchia, L. (2020) · Scientific Reports · doi:10.1038/s41598-019-56927-5 · Cited by 198 · see citing works
Riaboff, L., Shalloo, L., Smeaton, A. F., Couvreur, S., Madouasse, A., & Keane, M. T. (2021) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2021.106610 · Cited by 192 · see citing works
RTFN: A robust temporal feature network for time series classification
Xiao, Z., Xu, X., Xing, H., Luo, S., Dai, P., & Zhan, D. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.053 · Cited by 187 · see citing works
Predicting mortgage default using convolutional neural networks
Kvamme, H., Sellereite, N., Aas, K., & Sjursen, S. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.02.029 · Cited by 184 · see citing works
Proximity Forest: an effective and scalable distance-based classifier for time series
Lucas, B., Shifaz, A., Pelletier, C., O’Neill, L., Zaidi, N., Goethals, B., Petitjean, F., & Webb, G. I. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00617-3 · Cited by 183 · see citing works
Park, D., Kim, S., An, Y., & Jung, J. (2018) · Sensors · doi:10.3390/s18072110 · Cited by 174 · see citing works
Densely Knowledge-Aware Network for Multivariate Time Series Classification
Xiao, Z., Xing, H., Qu, R., Feng, L., Luo, S., Dai, P., Zhao, B., & Dai, Y. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3342640 · Cited by 170 · see citing works
ShapeNet: A Shapelet-Neural Network Approach for Multivariate Time Series Classification
Li, G., Choi, B., Xu, J., S Bhowmick, S., Chun, K.-P., & Wong, G. L.-H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.17018 · Cited by 169 · see citing works
A Large Comparison of Normalization Methods on Time Series
Lima, F. T., & Souza, V. M. A. (2023) · Big Data Research · doi:10.1016/j.bdr.2023.100407 · Cited by 161 · see citing works
Yu, W., Kim, I. Y., & Mechefske, C. K. (2020) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2020.107322 · Cited by 160 · see citing works
Adversarial Attacks on Deep Neural Networks for Time Series Classification
Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., & Müller, P.-A. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851936 · Cited by 158 · see citing works
Protecting Sensory Data against Sensitive Inferences
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2018) · Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems · doi:10.1145/3195258.3195260 · Cited by 154 · see citing works
Chen, W., & Shi, K. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.032 · Cited by 147 · see citing works
Multivariate Time Series Classification with WEASEL+MUSEPreprint
Schäfer, P., & Leser, U. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11343 · Cited by 142 · see citing works
The Canonical Interval Forest (CIF) Classifier for Time Series Classification
Middlehurst, M., Large, J., & Bagnall, A. J. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378424 · Cited by 141 · see citing works
Self-citationCausal inference for time series analysis: problems, methods and evaluation
Moraffah, R., Sheth, P., Karami, M., Bhattacharya, A., Wang, Q., Tahir, A., Raglin, A. J., & Liu, H. (2021) · Knowledge and Information Systems · doi:10.1007/s10115-021-01621-0 · Cited by 136 · see citing works
Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher
Iwana, B. K., & Uchida, S. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412812 · Cited by 131 · see citing works
Evaluation of post-hoc interpretability methods in time-series classification
Turbé, H., Bjelogrlic, M., Lovis, C., & Mengaldo, G. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00620-w · Cited by 130 · see citing works
A realistic and public dataset with rare undesirable real events in oil wells
Vargas, R. E. V., Munaro, C. J., Ciarelli, P. M., Medeiros, A. G., do Amaral, B. G., Barrionuevo, D. C., de Araújo, J. C. D., Ribeiro, J. L., & Magalhães, L. P. (2019) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2019.106223 · Cited by 129 · see citing works
Revisiting a Methodology for Efficient CNN Architectures in Profiling Attacks
Wouters, L., Arribas, V., Gierlichs, B., & Preneel, B. (2020) · IACR Transactions on Cryptographic Hardware and Embedded Systems · doi:10.46586/tches.v2020.i3.147-168 · Cited by 129 · see citing works
Hoffmann, M. W., Wildermuth, S., Gitzel, R., Boyaci, A., Gebhardt, J., Kaul, H., Amihai, I., Forg, B., Suriyah, M., Leibfried, T., et al. (2020) · Sensors · doi:10.3390/s20072099 · Cited by 127 · see citing works
Time series extrinsic regression
Tan, C. W., Bergmeir, C., Petitjean, F., & Webb, G. I. (2021) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00745-9 · Cited by 127 · see citing works
DTCM: Deep Transformer Capsule Mutual Distillation for Multivariate Time Series Classification
Xiao, Z., Xu, X., Xing, H., Zhao, B., Wang, X., Song, F., Qu, R., & Li, F. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3370219 · Cited by 124 · see citing works
Approaches and Applications of Early Classification of Time Series: A Review
Gupta, A., Gupta, H. P., Biswas, B., & Dutta, T. (2020) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2020.3027279 · Cited by 123 · see citing works
Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series
Pelletier, C., Webb, G. I., & Petitjean, F. (2019) · Remote Sensing · doi:10.3390/rs11050523 · Cited by 122 · see citing works
Towards a Universal Neural Network Encoder for Time Series
Joan, S., Santiago, P., & Karatzoglou, A. (2018) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-61499-918-8-120 · Cited by 119 · see citing works
Convolutional neural network-based wind-induced response estimation model for tall buildings
Oh, B. K., Glišić, B., Kim, Y., & Park, H. S. (2019) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12476 · Cited by 118 · see citing works
Cuesta–Frau, D. (2019) · Entropy · doi:10.3390/e21121167 · Cited by 118 · see citing works
MERLIN: Parameter-Free Discovery of Arbitrary Length Anomalies in Massive Time Series Archives
Nakamura, T., Imamura, M., Mercer, R., & Keogh, E. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00147 · Cited by 118 · see citing works
Self-citationTimeCLR: A self-supervised contrastive learning framework for univariate time series representation
Yang, X., Zhang, Z., & Cui, R. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108606 · Cited by 114 · see citing works
Hydra: competing convolutional kernels for fast and accurate time series classification
Dempster, A., Schmidt, D. F., & Webb, G. I. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00939-3 · Cited by 113 · see citing works
Nguyen, T. L., Gsponer, S., Ilie, I., O’Reilly, M., & Ifrim, G. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00633-3 · Cited by 111 · see citing works
Deep learning for brake squeal: Brake noise detection, characterization and prediction
Stender, M., Tiedemann, M., Spieler, D., Schoepflin, D., Hoffmann, N., & Oberst, S. M. (2020) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2020.107181 · Cited by 108 · see citing works
A review and evaluation of elastic distance functions for time series clustering
Holder, C., Middlehurst, M., & Bagnall, A. J. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01952-0 · Cited by 105 · see citing works
Self-citationZou, X., Wang, Z., Li, Q., & Sheng, W. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.08.023 · Cited by 104 · see citing works
Deep Neural Network Ensembles for Time Series Classification
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852316 · Cited by 103 · see citing works
Functional echo state network for time series classification
Ma, Q., Shen, L., Chen, W., Wang, J., Wei, J., & Yu, Z. (2016) · Information Sciences · doi:10.1016/j.ins.2016.08.081 · Cited by 102 · see citing works
A probabilistic classifier ensemble weighting scheme based on cross-validated accuracy estimates
Large, J., Lines, J., & Bagnall, A. J. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00638-y · Cited by 99 · see citing works
Self-citationMajumdar, S., & Laha, A. K. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113868 · Cited by 99 · see citing works
Time–frequency time–space LSTM for robust classification of physiological signals
Pham, T. D. (2021) · Scientific Reports · doi:10.1038/s41598-021-86432-7 · Cited by 98 · see citing works
Optimizing dynamic time warping’s window width for time series data mining applications
Dau, H. A., Silva, D. F., Petitjean, F., Forestier, G., Bagnall, A. J., Mueen, A., & Keogh, E. (2018) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-018-0565-y · Cited by 97 · see citing works
Self-citationTrend attention fully convolutional network for remaining useful life estimation
Fan, L., Chai, Y., & Chen, X. (2022) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2022.108590 · Cited by 93 · see citing works
The Temporal Dictionary Ensemble (TDE) Classifier for Time Series Classification
Middlehurst, M., Large, J., Cawley, G., & Bagnall, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67658-2_38 · Cited by 91 · see citing works
Self-citationConditional GAN for timeseries generationPreprint
Smith, K. E., & Smith, A. O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16477 · Cited by 90 · see citing works
ConvTimeNet: A Pre-trained Deep Convolutional Neural Network for Time Series Classification
Kashiparekh, K., Narwariya, J., Malhotra, P., Vig, L., & Shroff, G. M. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852105 · Cited by 87 · see citing works
Classification of Malware by Using Structural Entropy on Convolutional Neural Networks
Gibert, D., Mateu, C., Planes, J., & Vicens, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11409 · Cited by 85 · see citing works
The lernaean hydra of data series similarity search
Echihabi, K., Zoumpatianos, K., Palpanas, T., & Benbrahim, H. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3282495.3282498 · Cited by 84 · see citing works
Echihabi, K., Zoumpatianos, K., Palpanas, T., & Benbrahim, H. (2019) · Proceedings of the VLDB Endowment · doi:10.14778/3368289.3368303 · Cited by 84 · see citing works
Langone, R., Cuzzocrea, A., & Skantzos, N. (2020) · Data & Knowledge Engineering · doi:10.1016/j.datak.2020.101850 · Cited by 81 · see citing works
Scalable Dictionary Classifiers for Time Series Classification
Middlehurst, M., Vickers, W., & Bagnall, A. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33607-3_2 · Cited by 81 · see citing works
Self-citationZhang, Y., Hou, Y., Ouyang, K., & Zhou, S. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108385 · Cited by 81 · see citing works
Time Series Classification Using Time Warping Invariant Echo State Networks
Tanisaro, P., & Heidemann, G. (2016) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2016.0149 · Cited by 79 · see citing works
Robust Single Accelerometer-Based Activity Recognition Using Modified Recurrence Plot
Lu, J., & Tong, R. K. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2911204 · Cited by 78 · see citing works
Time2Graph: Revisiting Time Series Modeling with Dynamic Shapelets
Cheng, Z., Yang, Y., Wang, W., Hu, W., Zhuang, Y., & Song, G. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5769 · Cited by 76 · see citing works
Bridging the gap between mechanistic biological models and machine learning surrogates
Gherman, I. M., Abdallah, Z. S., Pang, W., Gorochowski, T. E., Grierson, C., & Marucci, L. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010988 · Cited by 76 · see citing works
Farahani, M. A., McCormick, M. R., Harik, R., & Wuest, T. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102839 · Cited by 75 · see citing works
Time series classification based on multi-feature dictionary representation and ensemble learning
Bai, B., Li, G., Wang, S., Wu, Z., & Yan, W. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114162 · Cited by 72 · see citing works
Towards Robotic Feeding: Role of Haptics in Fork-Based Food Manipulation
Bhattacharjee, T., Lee, G., Song, H., & Srinivasa, S. S. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2894592 · Cited by 72 · see citing works
Efficient Shapelet Discovery for Time Series Classification
Li, G., Choi, B. K. K., Xu, J., Bhowmick, S. S., Chun, K. P., & Wong, G. L.-H. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2995870 · Cited by 71 · see citing works
Data-Driven Modeling of Pregnancy-Related Complications
Espinosa, C. A., Becker, M., Marić, I., Wong, R. J., Shaw, G. M., Gaudilliere, B. L., Aghaeepour, N., Stevenson, D. K., Stelzer, I. A., Peterson, L. S., et al. (2021) · Trends in Molecular Medicine · doi:10.1016/j.molmed.2021.01.007 · Cited by 71 · see citing works
Automated location of steel truss bridge damage using machine learning and raw strain sensor data
Parisi, F., Mangini, A. M., Fanti, M. P., & Adam, J. M. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104249 · Cited by 71 · see citing works
On time series classification with dictionary-based classifiers
Large, J., Bagnall, A. J., Malinowski, S., & Tavenard, R. (2019) · Intelligent Data Analysis · doi:10.3233/ida-184333 · Cited by 71 · see citing works
Self-citationLahreche, A., & Boucheham, B. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114374 · Cited by 71 · see citing works
Time Series Classification: A Review of Algorithms and Implementations
Faouzi, J. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1004810 · Cited by 70 · see citing works
- Authors
- Bagnall, A., Lines, J., Bostrom, A., Large, J., & Keogh, E.
- Affiliations
- University of East Anglia (GB)
- University of California, Riverside (US)
- Corresponding author
- Bagnall, Anthony
- Venue
- Data Mining and Knowledge Discovery
- Published
- 2016-11-23
- Volume
- 31
- Issue
- 3
- Pages
- 606-660
- ISSN
- 1384-5810 (ISSN-L)
- 1573-756X
- PMC
- PMC6404674
- OpenAlex
- W2555077524
- Cited by
- 1,377
- References
- 48
- Citations by year
- 2026: 90 · 2025: 133 · 2024: 166 · 2023: 185 · 2022: 187 · 2021: 195 · 2020: 199 · 2019: 130 · 2018: 69 · 2017: 21 · 2016: 1
- FWCI
- 81.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Data Mining and Knowledge Discovery (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/M015087/1)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- time series classification, benchmark dataset, algorithm comparison, classification accuracy, reproducibility, Java implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
