The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Bake off redux: a review and experimental evaluation of recent time series classification algorithms
Open accessAbstract
In 2017, a research paper (Bagnall et al. Data Mining and Knowledge Discovery 31(3):606-660. 2017) compared 18 Time Series Classification (TSC) algorithms on 85 datasets from the University of California, Riverside (UCR) archive. This study, commonly referred to as a ‘bake off’, identified that only nine algorithms performed significantly better than the Dynamic Time Warping (DTW) and Rotation Forest benchmarks that were used. The study categorised each algorithm by the type of feature they extract from time series data, forming a taxonomy of five main algorithm types. This categorisation of algorithms alongside the provision of code and accessible results for reproducibility has helped fuel an increase in popularity of the TSC field. Over six years have passed since this bake off, the UCR archive has expanded to 112 datasets and there have been a large number of new algorithms proposed. We revisit the bake off, seeing how each of the proposed categories have advanced since the original publication, and evaluate the performance of newer algorithms against the previous best-of-category using an expanded UCR archive. We extend the taxonomy to include three new categories to reflect recent developments. Alongside the originally proposed distance, interval, shapelet, dictionary and hybrid based algorithms, we compare newer convolution and feature based algorithms as well as deep learning approaches. We introduce 30 classification datasets either recently donated to the archive or reformatted to the TSC format, and use these to further evaluate the best performing algorithm from each category. Overall, we find that two recently proposed algorithms, MultiROCKET+Hydra (Dempster et al. 2022) and HIVE-COTEv2 (Middlehurst et al. Mach Learn 110:3211-3243. 2021), perform significantly better than other approaches on both the current and new TSC problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · cited 1×
“Spatial pyramids (Lazebnik et al., 2006) are a technique used in computer vision to retain some temporal information back into the bag-of-words paradigm.”
MethodThe Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500769 · Cited by 7,210
Fukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342 · cited 7×
“We extend and encompass recent experimental evaluations (e.g. (Ruiz et al., 2021; Bagnall et al., 2020a; Middlehurst et al., 2021; Fawaz et al., 2019)) to provide insights into the current state of the art in the field and highlight future directions.”
BackgroundMethodInfluentialTime series classification from scratch with deep neural networks: A strong baseline
Wang, Z., Yan, W., & Oates, T. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966039 · Cited by 2,299
Inceptiontime: finding alexnet for time series classification
Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., & Petitjean, F. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00710-y · Cited by 1,788 · cited 6×
“InceptionTime Fawaz et al. (2020) is included since it is, to the best our knowledge, best in category for deep learning.”
BackgroundInfluentialExperiencing SAX: A novel symbolic representation of time series
Lin, J., Keogh, E., Wei, L., & Lonardi, S. (2007) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-007-0064-z · Cited by 1,618
Bagnall, A., Lines, J., Bostrom, A., Large, J., & Keogh, E. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-016-0483-9 · Cited by 1,377
Self-citationTime series shapelets: a novel technique that allows accurate, interpretable and fast classification
Ye, L., & Keogh, E. (2011) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0179-5 · Cited by 406
Cited by
Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey
Mohammadi Foumani, N., Miller, L., Tan, C. W., Webb, G. I., Forestier, G., & Salehi, M. (2024) · ACM Computing Surveys · doi:10.1145/3649448 · Cited by 268
Metadata
- Authors
- Middlehurst, M., Schäfer, P., & Bagnall, A.
- Venue
- Data Mining and Knowledge Discovery
- Published
- 2024-04-19
- Volume
- 38
- Issue
- 4
- Pages
- 1958-2031
- ISSN
- 1384-5810 (ISSN-L)
- 1573-756X
- OpenAlex
- W4394957446
- Cited by
- 197
- References
- 12
- FWCI
- 35.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bake off redux: a review and experimental evaluation of recent time series classification algorithms. https://app.cochat.ai/papers/10.1007/s10618-024-01022-1
- The paper
- Middlehurst, M., Schäfer, P., & Bagnall, A. (2024). Bake off redux: a review and experimental evaluation of recent time series classification algorithms. Data Mining and Knowledge Discovery. https://doi.org/10.1007/s10618-024-01022-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.