Cited by 1,377Time Series Analysis and ForecastingMusic and Audio ProcessingImage Retrieval and Classification Techniques

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.

The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances

Open access
Bagnall, Anthony, Lines, Jason, Bostrom, Aaron, Large, James, Keogh, Eamonn·2017·Data Mining and Knowledge Discovery

Abstract

In the last 5 years there have been a large number of new time series classification algorithms proposed in the literature. These algorithms have been evaluated on subsets of the 47 data sets in the University of California, Riverside time series classification archive. The archive has recently been expanded to 85 data sets, over half of which have been donated by researchers at the University of East Anglia. Aspects of previous evaluations have made comparisons between algorithms difficult. For example, several different programming languages have been used, experiments involved a single train/test split and some used normalised data whilst others did not. The relaunch of the archive provides a timely opportunity to thoroughly evaluate algorithms on a larger number of datasets. We have implemented 18 recently proposed algorithms in a common Java framework and compared them against two standard benchmark classifiers (and each other) by performing 100 resampling experiments on each of the 85 datasets. We use these results to test several hypotheses relating to whether the algorithms are significantly more accurate than the benchmarks and each other. Our results indicate that only nine of these algorithms are significantly more accurate than both benchmarks and that one classifier, the collective of transformation ensembles, is significantly more accurate than all of the others. All of our experiments and results are reproducible: we release all of our code, results and experimental details and we hope these experiments form the basis for more robust testing of new algorithms in the future.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10618-016-0483-9

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Four cases from two classes of the dataset FiftyWords. The top two series show class 30, the bottom two class 50. The common pattern is clear, but only detectable with reallignment
Fig. 1 Four cases from two classes of the dataset FiftyWords. The top two series show class 30, the bottom two class 50. The common pattern is clear, but only detectable with reallignmentCC BY · PubMed Central
Fig. 2 An example dataset where interval methods should do well. The noise in the early part of the series may confound whole series methods
Fig. 2 An example dataset where interval methods should do well. The noise in the early part of the series may confound whole series methodsCC BY · PubMed Central
Fig. 3 Average ranks of published results for TSF, LPS and TSBF in a critical difference diagram (explained in detail in Sect. 3)
Fig. 3 Average ranks of published results for TSF, LPS and TSBF in a critical difference diagram (explained in detail in Sect. 3)CC BY · PubMed Central
Fig. 4 Example matching between a shapelet and three series of different classes from NonInvasiveFetalECGThorax2. The scale of the shapelet is different on each series to reflect that distance is measured with normalised subseries
Fig. 4 Example matching between a shapelet and three series of different classes from NonInvasiveFetalECGThorax2. The scale of the shapelet is different on each series to reflect that distance is measured with normalised subseriesCC BY · PubMed Central
Fig. 5 Average ranks of the published results for three shapelet algorithms Fast Shapelets (FS), Shapelet Transform (ST) and Learned Shapelets (LS) on the 33 datasets they have in common
Fig. 5 Average ranks of the published results for three shapelet algorithms Fast Shapelets (FS), Shapelet Transform (ST) and Learned Shapelets (LS) on the 33 datasets they have in commonCC BY · PubMed Central
Fig. 6 An example of the need for detecting recurring patterns rather than unique patterns from WormsTwoClass. The top two series are the motion of normal worms, the bottom two mutant worms. The candidate subseries would not necessarily be a good shapelet, because there are close matches in both mutant and non mutant series
Fig. 6 An example of the need for detecting recurring patterns rather than unique patterns from WormsTwoClass. The top two series are the motion of normal worms, the bottom two mutant worms. The candidate subseries would not necessarily be a good shapelet, because there are close matches in both mutant and non mutant seriesCC BY · PubMed Central
Fig. 7 Average ranks of published results on 19 data sets or BOP, SAXVSM, BOSS
Fig. 7 Average ranks of published results on 19 data sets or BOP, SAXVSM, BOSSCC BY · PubMed Central
Fig. 8 Summary information for the 85 datasets in the archive
Fig. 8 Summary information for the 85 datasets in the archiveCC BY · PubMed Central
Fig. 9 Critical difference diagram for 11 potential benchmark classifiers
Fig. 9 Critical difference diagram for 11 potential benchmark classifiersCC BY · PubMed Central
Fig. 10 Critical difference diagram for the nine classifiers significantly better than both benchmark classifiers
Fig. 10 Critical difference diagram for the nine classifiers significantly better than both benchmark classifiersCC BY · PubMed Central
Fig. 11 The rank of BOSS against series length, grouped into sets with lengths 1–100, 101–200, etc. The last group contains all series of length 1000 or more. The dotted line is the linear regression fit
Fig. 11 The rank of BOSS against series length, grouped into sets with lengths 1–100, 101–200, etc. The last group contains all series of length 1000 or more. The dotted line is the linear regression fitCC BY · PubMed Central
Fig. 12 Critical difference diagram for ten elastic distance measures. The elastic ensemble (EE) is significantly more accurate than its constituents
Fig. 12 Critical difference diagram for ten elastic distance measures. The elastic ensemble (EE) is significantly more accurate than its constituentsCC BY · PubMed Central
Fig. 13 Time to classify ten test instances (averaged over 100 parameter options) for varying number of train instances of the problem StarlightCurves. The legend is ordered from fastest algorithm (ED) to slowest algorithm (TWE)
Fig. 13 Time to classify ten test instances (averaged over 100 parameter options) for varying number of train instances of the problem StarlightCurves. The legend is ordered from fastest algorithm (ED) to slowest algorithm (TWE)CC BY · PubMed Central
Fig. 14 Texas sharp shooter plot for MSM against DTW. The top right quadrant contains the problems where both the train and test accuracy for MSM is higher than DTW
Fig. 14 Texas sharp shooter plot for MSM against DTW. The top right quadrant contains the problems where both the train and test accuracy for MSM is higher than DTWCC BY · PubMed Central
Fig. 15 Critical difference diagram for three interval based techniques. There is no significant difference between them
Fig. 15 Critical difference diagram for three interval based techniques. There is no significant difference between themCC BY · PubMed Central
Fig. 16 Example series from ToeSegmentation1, left is class 1 (normal walking) and right is class 2 (abnormal walking)
Fig. 16 Example series from ToeSegmentation1, left is class 1 (normal walking) and right is class 2 (abnormal walking)CC BY · PubMed Central
Fig. 17 Example series from LargeKitchenApplicances. The top three are Washing Machine (class 3), the middle three Tumble Dryers (class 2) and the bottom three Dishwashers (class 1)
Fig. 17 Example series from LargeKitchenApplicances. The top three are Washing Machine (class 3), the middle three Tumble Dryers (class 2) and the bottom three Dishwashers (class 1)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Cited by

Sorted by citations

Items 401–500 of 1,000

Metadata

Authors
Bagnall, A., Lines, J., Bostrom, A., Large, J., & Keogh, E.
Affiliations
  • University of East Anglia (GB)
  • University of California, Riverside (US)
Corresponding author
Bagnall, Anthony
Venue
Data Mining and Knowledge Discovery
Published
2016-11-23
Volume
31
Issue
3
Pages
606-660
ISSN
  • 1384-5810 (ISSN-L)
  • 1573-756X
OpenAlex
W2555077524
Cited by
1,377
References
48
Citations by year
2026: 90 · 2025: 133 · 2024: 166 · 2023: 185 · 2022: 187 · 2021: 195 · 2020: 199 · 2019: 130 · 2018: 69 · 2017: 21 · 2016: 1
FWCI
81.75
Type
journal article
Language
en
Open access
Locations
  • Data Mining and Knowledge Discovery (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UEA Digital Repository (University of East Anglia) (submittedVersion, cc-by)
  • eScholarship (California Digital Library) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
Engineering and Physical Sciences Research Council (EP/M015087/1)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
time series classification, benchmark dataset, algorithm comparison, classification accuracy, reproducibility, Java implementation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review