The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
On the need for time series data mining benchmarks: a survey and empirical demonstration
Abstract
In the last decade there has been an explosion of interest in mining time series data. Literally hundreds of papers have introduced new algorithms to index, classify, cluster and segment time series. In this work we make the following claim. Much of this work has very little utility because the contribution made (speed in the case of indexing, accuracy in the case of classification and clustering, model accuracy in the case of segmentation) offer an amount of "improvement" that would have been completely dwarfed by the variance that would have been observed by testing on many real world datasets, or the variance that would have been observed by changing minor (unstated) implementation details.To illustrate our point, we have undertaken the most exhaustive set of time series experiments ever attempted, re-implementing the contribution of more than two dozen papers, and testing them on 50 real world, highly diverse datasets. Our empirical results strongly support our assertion, and suggest the need for a set of time series benchmarks and more careful empirical evaluation in the data mining community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771
Efficient similarity search in sequence databases
Agrawal, R., Faloutsos, C., & Swami, A. (1993) · Lecture Notes in Computer Science · doi:10.1007/3-540-57301-1_5 · Cited by 1,979
Fast subsequence matching in time-series databases
Faloutsos, C., Ranganathan, M., & Manolopoulos, Y. (1994) · Proceedings of the 1994 ACM SIGMOD international conference on Management of data · doi:10.1145/191839.191925 · Cited by 1,709
Efficient time series matching by wavelets
Chan, K.-P., & Fu, A. W.-C. (1999) · Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337) · doi:10.1109/icde.1999.754915 · Cited by 1,031
Locally adaptive dimensionality reduction for indexing large time series databases
Keogh, E., Chakrabarti, K., Pazzani, M., & Mehrotra, S. (2001) · Proceedings of the 2001 ACM SIGMOD international conference on Management of data · doi:10.1145/375663.375680 · Cited by 866
Self-citationFast Time Sequence Indexing for Arbitrary Lp Norms
Yi, B.-K., & Faloutsos, C. (2018) · doi:10.1184/r1/6605618 · Cited by 650
Fast Similarity Search in the Presence of Noise, Scaling, and Translation in Time-Series Databases
Agrawal, R., Lin, K.-I., Sawhney, H., & Shim, K. (1995) · Cited by 649
Rule discovery from time series
Das, G., Lin, K.-I., Mannila, H., Renganathan, G., & Smyth, P. (1998) · Cited by 637
Exact indexing of dynamic time warping
Keogh, E. (2002) · Cited by 594
Self-citationKeogh, E., & Pazzani, M. J. (1998) · Cited by 496
Self-citation
Cited by
A symbolic representation of time series, with implications for streaming algorithms
Lin, J., Keogh, E., Lonardi, S., & Chiu, B. (2003) · Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery · doi:10.1145/882082.882086 · Cited by 1,799
Self-citationToward accurate dynamic time warping in linear time and space
Salvador, S., & Chan, P. (2007) · Intelligent Data Analysis · doi:10.3233/ida-2007-11508 · Cited by 1,705
Experiencing SAX: A novel symbolic representation of time series
Lin, J., Keogh, E., Wei, L., & Lonardi, S. (2007) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-007-0064-z · Cited by 1,618
Self-citationA review on time series data mining
Fu, T.-C. (2011) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2010.09.007 · Cited by 1,500
Robust and fast similarity search for moving object trajectories
Chen, L., Özsu, M. T., & Oria, V. (2005) · Proceedings of the 2005 ACM SIGMOD international conference on Management of data · doi:10.1145/1066157.1066213 · Cited by 1,336
A review of unsupervised feature learning and deep learning for time-series modeling
Längkvist, M., Karlsson, L., & Loutfi, A. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.01.008 · Cited by 1,243
Esling, P., & Agon, C. (2012) · ACM Computing Surveys · doi:10.1145/2379776.2379788 · Cited by 997
(2015) · Cited by 989
Time series shapelets: a new primitive for data mining
Ye, L., & Keogh, E. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557122 · Cited by 987
Self-citationOn The Marriage of Lp-norms and Edit Distance
Chen, L., & Ng, R. (2004) · Proceedings 2004 VLDB Conference · doi:10.1016/b978-012088469-8.50070-x · Cited by 774
HOT SAX: Efficiently Finding the Most Unusual Time Series Subsequence
Keogh, E., Lin, J., & Fu, A. (2006) · Fifth IEEE International Conference on Data Mining (ICDM'05) · doi:10.1109/icdm.2005.79 · Cited by 696
Self-citationCharacteristic-Based Clustering for Time Series Data
Wang, X., Smith, K., & Hyndman, R. (2006) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-005-0039-x · Cited by 679
Probabilistic discovery of time series motifs
Chiu, B., Keogh, E., & Lonardi, S. (2003) · Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/956750.956808 · Cited by 573
Self-citationTowards parameter-free data mining
Keogh, E., Lonardi, S., & Ratanamahatana, C. A. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014077 · Cited by 567
Self-citationThe BOSS is concerned with time series classification in the presence of noise
Schäfer, P. (2015) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0377-7 · Cited by 560
A brief survey on sequence classification
Xing, Z., Pei, J., & Keogh, E. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1882471.1882478 · Cited by 553
Self-citationClustering of time-series subsequences is meaningless: implications for previous and future research
Keogh, E., & Lin, J. (2005) · Knowledge and Information Systems · doi:10.1007/s10115-004-0172-7 · Cited by 464
Self-citationTime series shapelets: a novel technique that allows accurate, interpretable and fast classification
Ye, L., & Keogh, E. (2011) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0179-5 · Cited by 406
Self-citationMaking Time-series Classification More Accurate Using Learned Constraints
Ratanamahatana, C. A., & Keogh, E. (2004) · Proceedings of the 2004 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972740.2 · Cited by 407
Self-citationProcess data analytics in the era of big data
Qin, S. J. (2014) · AIChE Journal · doi:10.1002/aic.14523 · Cited by 383
A review on distance based time series classification
Abanda, A., Mori, U., & Lozano, J. A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-018-0596-4 · Cited by 296
Rotation-invariant similarity in time series using bag-of-patterns representation
Lin, J., Khade, R., & Li, Y. (2012) · Journal of Intelligent Information Systems · doi:10.1007/s10844-012-0196-5 · Cited by 295
Semi-supervised time series classification
Wei, L., & Keogh, E. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150498 · Cited by 282
Self-citationHierarchical Clustering of Time-Series Data Streams
Rodrigues, P. P., Gama, J., & Pedroso, J. P. (2008) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.190727 · Cited by 260
Discrete Wavelet transform-based time series analysis and mining
Chaovalit, P., Gangopadhyay, A., Karabatis, G., & Chen, Z. (2011) · ACM Computing Surveys · doi:10.1145/1883612.1883613 · Cited by 218
Metadata
- Authors
- Keogh, E., & Kasetty, S.
- Venue
- Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2002-07-23
- Pages
- 102-111
- OpenAlex
- W2006783944
- Cited by
- 442
- References
- 42
- FWCI
- 22.01
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.