The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances
Open accessAbstract
In the last 5 years there have been a large number of new time series classification algorithms proposed in the literature. These algorithms have been evaluated on subsets of the 47 data sets in the University of California, Riverside time series classification archive. The archive has recently been expanded to 85 data sets, over half of which have been donated by researchers at the University of East Anglia. Aspects of previous evaluations have made comparisons between algorithms difficult. For example, several different programming languages have been used, experiments involved a single train/test split and some used normalised data whilst others did not. The relaunch of the archive provides a timely opportunity to thoroughly evaluate algorithms on a larger number of datasets. We have implemented 18 recently proposed algorithms in a common Java framework and compared them against two standard benchmark classifiers (and each other) by performing 100 resampling experiments on each of the 85 datasets. We use these results to test several hypotheses relating to whether the algorithms are significantly more accurate than the benchmarks and each other. Our results indicate that only nine of these algorithms are significantly more accurate than both benchmarks and that one classifier, the collective of transformation ensembles, is significantly more accurate than all of the others. All of our experiments and results are reproducible: we release all of our code, results and experimental details and we hope these experiments form the basis for more robust testing of new algorithms in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“HIVE-COTE is currently considered the state-of-the-art algorithm for time series classification (Bagnall et al., 2017) when evaluated over the 85 datasets from the UCR/UEA archive.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“To achieve its high accuracy, HIVE-COTE becomes hugely computationally intensive and impractical to run on a real big data mining problem (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“Similarly to Bagnall et al. (2017), in order to have a fair comparison between the tested approaches, we developed a common framework in Python, Keras (Chollet, 2015) and Tensorflow (Abadi et al., 2015) to train the deep learning models on a cluster of more than 60 GPUs.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“Note that the time series in this archive are already z-normalized (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“This archive also exhibits datasets with different characteristics such as the length of the time series which, unlike the UCR/UEA archive, varies among the same dataset. This is due to the fact that the datasets in the UCR/UEA archive are already re-scaled to have an equal length among one dataset (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“In order to have a thorough and fair experimental evaluation of all approaches, we tested each algorithm on the whole UCR/UEA archive (Chen et al., 2015b; Bagnall et al., 2017) which contains 85 univariate time series datasets.”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“The informal consensus in the literature is that generative models are usually less accurate than direct discriminative models (Bagnall et al., 2017; Nguyen et al., 2017);”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“To measure the performance of the model on the test data (generalization), we adopted the accuracy measure (similarly to Bagnall et al. (2017)).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“However, given the small sample size (only 7 datasets), we cannot conclude that FCN will almost always outperform the ResNet model for ECG datasets (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
In the citing paper
“This straightforward approach supposes that the ED is able to strongly separate the raw time series, which is usually not the case evident by the low accuracy of the nearest neighbor when coupled with ED (Bagnall et al., 2017).”
From Deep learning for time series classification: a review (2019) · cites this paper as Bagnall et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 12,051
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Experiencing SAX: A novel symbolic representation of time series
Lin, J., Keogh, E., Wei, L., & Lonardi, S. (2007) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-007-0064-z · Cited by 1,618
Self-citationQuerying and mining of time series data
Ding, H., Trajcevski, G., Scheuermann, P., Wang, X., & Keogh, E. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454226 · Cited by 1,353
Self-citationGarcía, S., & Herrera, F. (2008) · Cited by 1,277
On the Need for Time Series Data Mining Benchmarks: A Survey and Empirical Demonstration
Keogh, E., & Kasetty, S. (2003) · Data Mining and Knowledge Discovery · doi:10.1023/a:1024988512476 · Cited by 860
Self-citationExperimental comparison of representation methods and distance measures for time series data
Wang, X., Mueen, A., Ding, H., Trajcevski, G., Scheuermann, P., & Keogh, E. (2012) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-012-0250-5 · Cited by 846
Self-citationOn The Marriage of Lp-norms and Edit Distance
Chen, L., & Ng, R. T. (2004) · Elsevier eBooks · doi:10.1016/b978-012088469-8.50070-x · Cited by 792
Algorithms for the Longest Common Subsequence Problem
Hirschberg, D. S. (1977) · Journal of the ACM · doi:10.1145/322033.322044 · Cited by 752
Weighted dynamic time warping for time series classification
Jeong, Y., Jeong, M. K., & Omitaomu, O. A. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.022 · Cited by 697
A time series forest for classification and feature extraction
Deng, H., Runger, G. C., Tuv, E., & Vladimir, M. (2013) · Information Sciences · doi:10.1016/j.ins.2013.02.030 · Cited by 632
Word image matching using dynamic time warping
Rath, T. M., & Manmatha, R. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2003.1211511 · Cited by 584
The BOSS is concerned with time series classification in the presence of noise
Schäfer, P. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0377-7 · Cited by 564
Classification of time series by shapelet transformation
Hills, J., Lines, J., Baranauskas, E., Mapp, J., & Bagnall, A. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-013-0322-1 · Cited by 538
Self-citationTime series classification with ensembles of elastic distance measures
Lines, J., & Bagnall, A. J. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0361-2 · Cited by 513
Self-citationLearning time-series shapelets
Grabocka, J., Schilling, N., Wistuba, M., & Schmidt-Thieme, L. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623613 · Cited by 489
Time-Series Classification with COTE: The Collective of Transformation-Based Ensembles
Bagnall, A. J., Lines, J., Hills, J., & Bostrom, A. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2416723 · Cited by 433
Self-citationFast Shapelets: A Scalable Algorithm for Discovering Time Series Shapelets
Rakthanmanon, T., & Keogh, E. (2013) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972832.74 · Cited by 408
Self-citationTime series shapelets: a novel technique that allows accurate, interpretable and fast classification
Ye, L., & Keogh, E. (2011) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0179-5 · Cited by 406
Self-citationLocally adaptive dimensionality reduction for indexing large time series databases
Chakrabarti, K., Keogh, E., Mehrotra, S., & Pazzani, M. J. (2002) · ACM Transactions on Database Systems · doi:10.1145/568518.568520 · Cited by 404
Self-citationThree Myths about Dynamic Time Warping Data Mining
Ratanamahatana, C. A., & Keogh, E. (2005) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972757.50 · Cited by 398
Self-citationClustering Sequences with Hidden Markov Models
Smyth, P. (1996) · Cited by 381
Highly Comparative Feature-Based Time-Series Classification
Fulcher, B., & Jones, N. S. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2316504 · Cited by 363
A Bag-of-Features Framework to Classify Time Series
Baydogan, M. G., Runger, G., & Tuv, E. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.72 · Cited by 361
CID: an efficient complexity-invariant distance for time series
Batista, G. E. A. P. A., Keogh, E., Tataw, O. M., & Souza, V. M. A. (2013) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-013-0312-3 · Cited by 361
Self-citationUsing dynamic time warping distances as features for improved time series classification
Kate, R. J. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0418-x · Cited by 355
Time Warp Edit Distance with Stiffness Adjustment for Time Series Matching
Marteau, P.-F. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.76 · Cited by 337
Should we really use post-hoc tests based on mean-ranks?Preprint
BenavoliAlessio, CoraniGiorgio, & MangiliFrancesca (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02288 · Cited by 318
A database of Caenorhabditis elegans behavioral phenotypes
Yemini, E., Jucikas, T., Grundy, L. J., Brown, A. E., & Schafer, W. R. (2013) · Nature Methods · doi:10.1038/nmeth.2560 · Cited by 309
Rotation-invariant similarity in time series using bag-of-patterns representation
Lin, J., Khade, R., & Li, Y. (2012) · Journal of Intelligent Information Systems · doi:10.1007/s10844-012-0196-5 · Cited by 305
Mueen, A., Keogh, E., & Young, N. E. (2011) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020587 · Cited by 282
Self-citationSAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model
Senin, P., & Malinchik, S. (2013) · Proceedings · doi:10.1109/icdm.2013.52 · Cited by 268
Addressing Big Data Time Series
Rakthanmanon, T., Campana, B. J. L., Mueen, A., Batista, G., Westover, M. B., Zhu, Q., Zakaria, J., & Keogh, E. (2013) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2500489 · Cited by 245
Self-citationTime series clustering and classification by the autoregressive metric
Corduas, M., & Piccolo, D. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2007.06.001 · Cited by 199
The Move-Split-Merge Metric for Time Series
Stefan, A., Athitsos, V., & Das, G. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.88 · Cited by 189
Deformable Markov model templates for time-series pattern matching
Ge, X., & Smyth, P. (2000) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/347090.347109 · Cited by 179
Time series representation and similarity based on local autopatterns
Baydoğan, M. G., & Runger, G. C. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0425-y · Cited by 176
Maharaj, E. A. (2000) · Journal of Classification · doi:10.1007/s003570000023 · Cited by 171
Using derivatives in time series classification
Górecki, T., & Łuczak, M. (2012) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-012-0251-4 · Cited by 153
Model-based kernel for efficient time series analysis
Chen, H., Tang, F., Peter Tiňo, B., & Yao, X. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2487700 · Cited by 108
Time Series Classification Using Compression Distance of Recurrence Plots
Silva, D. F., Souza, V. M. A., & Batista, G. E. A. P. A. (2013) · Proceedings · doi:10.1109/icdm.2013.128 · Cited by 103
Non-isometric transforms in time series classification using DTW
Górecki, T., & Łuczak, M. (2014) · Knowledge-Based Systems · doi:10.1016/j.knosys.2014.02.011 · Cited by 95
Support vector machines of interval-based features for time series classification
Rodríguez, J. J., Alonso, C. J., & Maestro, J. A. (2005) · Knowledge-Based Systems · doi:10.1016/j.knosys.2004.10.007 · Cited by 90
Binary Shapelet Transform for Multiclass Time Series Classification
Bostrom, A., & Bagnall, A. J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22729-0_20 · Cited by 85
Self-citationSAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model
Senin, P., & Malinchik, S. (2013) · doi:10.21236/ada603196 · Cited by 65
Automatic segmentation of diatom images for classification
Jalba, A. C., Wilkinson, M. H. F., & Roerdink, J. B. T. M. (2004) · Microscopy Research and Technique · doi:10.1002/jemt.20111 · Cited by 49
A Run Length Transformation for Discriminating Between Auto Regressive Time Series
Bagnall, A. J., & Janacek, G. J. (2013) · Journal of Classification · doi:10.1007/s00357-013-9135-6 · Cited by 39
Self-citation
Cited by
Sorted by citations
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342
“However, given the small sample size (only 7 datasets), we cannot conclude that FCN will almost always outperform the ResNet model for ECG datasets (Bagnall et al., 2017).”
BackgroundMethodResultInfluentialInceptiontime: finding alexnet for time series classification
Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., & Petitjean, F. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00710-y · Cited by 1,788
A Transformer-based Framework for Multivariate Time Series Representation Learning
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., & Eickhoff, C. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467401 · Cited by 1,278
Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., & Keogh, E. (2019) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2019.1911747 · Cited by 1,128
Self-citationDempster, A., Petitjean, F., & Webb, G. I. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00701-z · Cited by 1,165
Metadata
- Authors
- Bagnall, A., Lines, J., Bostrom, A., Large, J., & Keogh, E.
- Affiliations
- University of East Anglia (GB)
- University of California, Riverside (US)
- Corresponding author
- Bagnall, Anthony
- Venue
- Data Mining and Knowledge Discovery
- Published
- 2016-11-23
- Volume
- 31
- Issue
- 3
- Pages
- 606-660
- ISSN
- 1384-5810 (ISSN-L)
- 1573-756X
- PMC
- PMC6404674
- OpenAlex
- W2555077524
- Cited by
- 1,377
- References
- 48
- Citations by year
- 2026: 90 · 2025: 133 · 2024: 166 · 2023: 185 · 2022: 187 · 2021: 195 · 2020: 199 · 2019: 130 · 2018: 69 · 2017: 21 · 2016: 1
- FWCI
- 81.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Data Mining and Knowledge Discovery (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/M015087/1)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- time series classification, benchmark dataset, algorithm comparison, classification accuracy, reproducibility, Java implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
