Effectively Detecting Operational Anomalies In Large-Scale IoT Data Infrastructures By Using A GAN-Based Predictive Model
The DOI matches its Crossref record; 44 of the 47 references it cites were found in a registry or an index.
Abstract
Quality of data services is crucial for operational large-scale internet-of-things (IoT) research data infrastructure, in particular when serving large amounts of distributed users. Effectively detecting runtime anomalies and diagnosing their root cause helps to defend against adversarial attacks, thereby essentially boosting system security and robustness of the IoT infrastructure services. However, conventional anomaly detection methods are inadequate when facing the dynamic complexities of these systems. In contrast, supervised machine learning methods are unable to exploit large amounts of data due to the unavailability of labeled data. This paper leverages popular GAN-based generative models and end-to-end one-class classification to improve unsupervised anomaly detection. A novel heterogeneous BiGAN-based anomaly detection model Heterogeneous Temporal Anomaly-reconstruction GAN (HTA-GAN) is proposed to make better use of a one-class classifier and a novel anomaly scoring function. The Generator-Encoder-Discriminator BiGAN structure can lead to practical anomaly score computation and temporal feature capturing. We empirically compare the proposed approach with several state-of-the-art anomaly detection methods on real-world datasets, anomaly benchmarks and synthetic datasets. The results show that HTA-GAN outperforms its competitors and demonstrates better robustness.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A significant body of work on automated diagnosis and prognosis using machine learning has lately acquired popularity in the medical sector [58,59].”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [59]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1993.341010 · Cited by 14,265
Wold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
Deep Learning for Anomaly Detection: A Review
Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2022) · ACM Computing Surveys · doi:10.1145/3439950 · Cited by 2,837
Deep Learning for Anomaly Detection: A Review
Pang, G., Shen, C., Cao, L., et al. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · doi:10.48550/arxiv.2007.02500 · Cited by 1,718
f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks
Schlegl, T., Seeböck, P., Waldstein, S. M., Langs, G., & Schmidt-Erfurth, U. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.010 · Cited by 1,545
Distance-based outliers: algorithms and applications
Knorr, E. M., Ng, R. T., & Tucakov, V. (2000) · The VLDB Journal The International Journal on Very Large Data Bases · doi:10.1007/s007780050006 · Cited by 1,195
High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning
Erfani, S. M., Rajasegarar, S., Karunasekera, S., & Leckie, C. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.03.028 · Cited by 1,162
Outlier Detection for Temporal Data: A Survey
Gupta, M., Gao, J., Aggarwal, C. C., & Han, J. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.184 · Cited by 1,001
From Model, Signal to Knowledge: A Data-Driven Perspective of Fault Detection and Diagnosis
Dai, X., & Gao, Z. (2013) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2013.2243743 · Cited by 728
2008 Eighth IEEE International Conference on Data Mining
(2008) · doi:10.1109/icdm14818.2008 · Cited by 689
Donahue, J., Krähenbühl, P., & Darrell, T. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09782 · Cited by 616
Fiore, U., De Santis, A., Perla, F., Zanetti, P., & Palmieri, F. (2019) · Information Sciences · doi:10.1016/j.ins.2017.12.030 · Cited by 602
2018 IEEE International Conference on Big Data (Big Data)
(2018) · doi:10.1109/bigdata44402.2018 · Cited by 471
Efficient GAN-Based Anomaly Detection
Zenati, H., Foo, C.-S., Lecouat, B., Manek, G., & Chandrasekhar, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06222 · Cited by 460
A Game-Theoretical Approach for User Allocation in Edge Computing Environment
He, Q., Cui, G., Zhang, X., Chen, F., Deng, S., Jin, H., Li, Y., & Yang, Y. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2019.2938944 · Cited by 398
Unrolled Generative Adversarial Networks
Metz, L., Poole, B., Pfau, D., & Sohl‐Dickstein, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02163 · Cited by 382
2016 IEEE 16th International Conference on Data Mining (ICDM)
(2016) · doi:10.1109/icdm38702.2016 · Cited by 347
Generative Adversarial Active Learning for Unsupervised Outlier Detection
Liu, Y., Li, Z., Zhou, C., Jiang, Y., Sun, J., Wang, M., & He, X. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2905606 · Cited by 335
Gao, H., Qiu, B., Barroso, R. J. D., Hussain, W., Xu, Y., & Wang, X. (2023) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3163144 · Cited by 239
LSTM-Based VAE-GAN for Time-Series Anomaly Detection
Niu, Z., Yu, K., & Wu, X. (2020) · Sensors · doi:10.3390/s20133738 · Cited by 237
Good Semi-supervised Learning that Requires a Bad GAN
Dai, Z., Yang, Z., Yang, F., Cohen, W. W., & Salakhutdinov, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09783 · Cited by 209
Proceedings of the 2015 SIAM International Conference on Data Mining
(2015) · doi:10.1137/1.9781611974010 · Cited by 201
GAN augmentation to deal with imbalance in imaging-based intrusion detection
Andresini, G., Appice, A., Rose, L., & Malerba, D. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.04.017 · Cited by 162
Using LSTM encoder-decoder algorithm for detecting anomalous ADS-B messages
Habler, E., & Shabtai, A. (2018) · Computers & Security · doi:10.1016/j.cose.2018.07.004 · Cited by 148
A Classification Framework for Anomaly Detection
SteinwartIngo, HushDon, & ScovelClint (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1058109 · Cited by 142
Stochastic Modeling and Quality Evaluation of Infrastructure-as-a-Service Clouds
Xia, Y., Zhou, M., Luo, X., Zhu, Q., Li, J., & Huang, Y. (2013) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2276477 · Cited by 133
Self-citationRapid Distance-Based Outlier Detection via Sampling
Sugiyama, M., & Borgwardt, K. M. (2013) · Cited by 126
2015 IEEE International Conference on Data Mining Workshop (ICDMW)
(2015) · doi:10.1109/icdmw37719.2015 · Cited by 118
Learning to Decompose and Disentangle Representations for Video Prediction
Hsieh, J.-T., Liu, B., Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04166 · Cited by 108
Halpern, J. Y., & Lagoze, C. (1999) · Proceedings of the fourth ACM conference on Digital libraries · doi:10.1145/313238.313240 · Cited by 101
Gao, H., Xu, K., Cao, M., Xiao, J., Xu, Q., & Yin, Y. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3102591 · Cited by 91
Xu, R., Cheng, Y., Liu, Z., Xie, Y., & Yang, Y. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.05.035 · Cited by 78
Personalized APIs Recommendation With Cognitive Knowledge Mining for Industrial Systems
Yin, Y., Huang, Q. Z., Gao, H., & Xu, Y. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3039500 · Cited by 64
Energy-efficient VM scheduling based on deep reinforcement learning
Wang, B., Liu, F., & Lin, W. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.07.023 · Cited by 64
Feature Grouping-Based Outlier Detection Upon Streaming Trajectories
Mao, J., Wang, T., Jin, C., & Zhou, A. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2744619 · Cited by 63
Decentralized Learning of Generative Adversarial Networks from Non-iid DataPreprint
Yonetani, R., Takahashi, T., Hashimoto, A., & Ushiku, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09684 · Cited by 48
IEEE International Conference of Multimedia and Expo (ICME) 2016
(2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2483041 · Cited by 24
Chen, P., Xia, Y., & Yu, C. (2021) · International Journal of Web Services Research · doi:10.4018/ijwsr.2021010102 · Cited by 8
Self-citation
Research works printed but not linked: 15
- Aggarwal, An introduction to outlier analysis. Outlier analysis, pp. 1–34
- Angiulli, European conference on principles of data mining and knowledge discovery, 15
- Zong, Deep autoencoding gaussian mixture model for unsupervised anomaly detection, International conference on learning representations
- Zhang, Proceedings of the AAAI conference on artificial intelligence, 1409
- Luo, 2017 IEEE International Conference on Multimedia and Expo (ICME), 439
- Knorr, Vldb, 211
- Ramaswamy, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 427
- Zhang, Pacific-Asia Conference on Knowledge Discovery and Data Mining, 813
- Liao, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 537
- Gao, IEEE Transactions on Neural Networks and Learning Systems (TNNLS)
- Schlegl, International conference on information processing in medical imaging, 146
- Akcay, Asian conference on computer vision, 622
- Arjovsky, International conference on machine learning, 214
- Sabokrou, Proceedings of the IEEE conference on computer vision and pattern recognition, 3379
- Zheng, Proceedings of the AAAI Conference on Artificial Intelligence, 1286
Cited by
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“A significant body of work on automated diagnosis and prognosis using machine learning has lately acquired popularity in the medical sector [58, 59].”
BackgroundAmiri, Z., Heidari, A., Navimipour, N. J., Unal, M., & Mousavi, A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16382-x · Cited by 189
Xin, R., Chen, P., & Zhao, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111724 · Cited by 99
“A microservice application can be observed by monitoring tools, which help operators to detect performance anomalies [7], [8].”
BackgroundSelf-citationEarly Detection of Earthquakes Using IoT and Cloud Infrastructure: A Survey
Abdalzaher, M. S., Krichen, M., Yiltas-Kaplan, D., Dhaou, I. B., & Adoni, W. Y. H. (2023) · Sustainability · doi:10.3390/su151511713 · Cited by 116
“This can lead to faster and more accurate decision-making, improving the effectiveness of these systems [264,265].”
BackgroundSong, Y., Xin, R., Chen, P., Zhang, R., Chen, J., & Zhao, Z. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.03.020 · Cited by 100
Self-citation
Metadata
- Authors
- Chen, P., Liu, H., Xin, R., Carval, T., Zhao, J., Xia, Y., & Zhao, Z.
- Affiliations
- Xihua University (CN)
- University of Amsterdam (NL)
- Euro-Argo ERIC (FR)
- Chongqing University (CN)
- Corresponding author
- Xia, Yunni
- Zhao, Zhiming
- Venue
- The Computer Journal
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-10-01
- Volume
- 65
- Issue
- 11
- Pages
- 2909-2925
- ISSN
- 0010-4620 (ISSN-L)
- 1460-2067
- OpenAlex
- W4293586385
- Cited by
- 128
- References
- 47
- Citations by year
- 2026: 11 · 2025: 21 · 2024: 29 · 2023: 60 · 2022: 7
- FWCI
- 9.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- The Computer Journal (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Keywords
- unsupervised anomaly detection, generative adversarial networks, one-class classification, temporal feature extraction, root cause diagnosis, adversarial attacks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.