The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Towards a Rigorous Evaluation of Time-Series Anomaly Detection
Open accessAbstract
In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets, giving the impression of clear improvements in TAD. However, most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; even a random anomaly score can easily turn into a state-of-the-art TAD method. Therefore, the comparison of TAD methods after applying the PA protocol can lead to misguided rankings. Furthermore, we question the potential of existing TAD methods by showing that an untrained model obtains comparable detection performance to the existing methods even when PA is forbidden. Based on our findings, we propose a new baseline and an evaluation protocol. We expect that our study will help a rigorous evaluation of TAD and lead to further improvement in future researches.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network
Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330672 · Cited by 1,804
Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
Hundman, K., Constantinou, V., Laporte, C., Colwell, I., & Soderstrom, T. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219845 · Cited by 1,771
Graph Neural Network-Based Anomaly Detection in Multivariate Time Series
Deng, A., & Hooi, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16523 · Cited by 1,413
Deep Learning for Anomaly Detection: A Survey
Chalapathy, R., & Chawla, S. (2019) · arXiv · doi:10.48550/arxiv.1901.03407 · Cited by 1,240
USAD: UnSupervised Anomaly Detection on Multivariate Time Series
Audibert, J., Michiardi, P., Guyard, F., Marti, S., & Zuluaga, M. A. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403392 · Cited by 1,094 · cited 4×
“Figure 4 plots F1 PA and F1 for SWaT and WADI, as re-ported by the original studies on USAD, DAGMM, LSTM-VAE, OmniAnomaly, and GDN.”
BackgroundMethodInfluentialLSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection
Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016) · arXiv · doi:10.48550/arxiv.1607.00148 · Cited by 819
A Dataset to Support Research in the Design of Secure Water Treatment Systems
Goh, J., Adepu, S., Junejo, K. N., & Mathur, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-71368-7_8 · Cited by 729 · cited 3×
“In this section, we introduce a list of the five most widely used TAD benchmark datasets as follows: Secure water treatment (SWaT) (Goh et al. 2016): the SWaT dataset was collected over 11 days from a scaled-down water treatment testbed comprising 51 sensors (Mathur and Tippenhauer 2016).”
MethodPatch SVDD: Patch-Level SVDD for Anomaly Detection and Segmentation
Yi, J., & Yoon, S. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-69544-6_23 · Cited by 551
Self-citation
Cited by
Deep Learning for Time Series Anomaly Detection: A Survey
Zamanzadeh Darban, Z., Webb, G. I., Pan, S., Aggarwal, C., & Salehi, M. (2025) · ACM Computing Surveys · doi:10.1145/3691338 · Cited by 613
Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series
Sørbø, S., & Ruocco, M. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00988-8 · Cited by 58
“Changes in labels will necessarily affect the evaluation scores, especially if an event is included or excluded, as there are usually very few anomalies.”
InfluentialICS anomaly detection based on sensor patterns and actuator rules in spatiotemporal dependency
Cai, J., Wei, Z., & Luo, J. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3393528 · Cited by 38
Metadata
- Authors
- Kim, S., Choi, K., Choi, H.-S., Lee, B., & Yoon, S.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2022-06-28
- Volume
- 36
- Issue
- 7
- Pages
- 7194-7201
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3199473923
- Cited by
- 194
- References
- 10
- FWCI
- 21.12
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards a Rigorous Evaluation of Time-Series Anomaly Detection. https://app.cochat.ai/papers/10.1609/aaai.v36i7.20680
- The paper
- Kim, S., Choi, K., Choi, H.-S., Lee, B., & Yoon, S. (2022). Towards a Rigorous Evaluation of Time-Series Anomaly Detection. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v36i7.20680
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.