LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the vector of words from word2vec can not be used directly, which needs to be transformed into the vector of log events and further transformed into the vector of log sequences. To reduce computational cost and avoid multiple transformations, in this paper, we propose an offline feature extraction model, named LogEvent2vec, which takes the log event as input of word2vec to extract the relevance between log events and vectorize log events directly. LogEvent2vec can work with any coordinate transformation methods and anomaly detection models. After getting the log event vector, we transform log event vector to log sequence vector by bary or tf-idf and three kinds of supervised models (Random Forests, Naive Bayes, and Neural Networks) are trained to detect the anomalies. We have conducted extensive experiments on a real public log dataset from BlueGene/L (BGL). The experimental results demonstrate that LogEvent2vec can significantly reduce computational time by 30 times and improve accuracy, comparing with word2vec. LogEvent2vec with bary and Random Forest can achieve the best F1-score and LogEvent2vec with tf-idf and Naive Bayes needs the least computational time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Drain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943 · cited 5×
“After Drain [27] log parsing, we obtain 376 log events.”
MethodInfluentialInterpreting TF-IDF term weights as making relevance decisions
Wu, H. C., Luk, R. W. P., Wong, K. F., & Kwok, K. L. (2008) · ACM Transactions on Information Systems · doi:10.1145/1361684.1361686 · Cited by 821
LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs
Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., Sun, P., et al. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/658 · Cited by 708
Tools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574 · cited 1×
“[37] provide the tools and benchmarks for automated log parsing.”
MethodSelf-citationTowards Automated Log Parsing for Large-Scale Log Data Analysis
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2762673 · Cited by 230 · cited 1×
“He et al. [26] design and implement a parallel log parser (namely POP) on top of Spark, a large-scale data processing platform.”
Self-citation
Cited by
IoT anomaly detection methods and applications: A survey
Chatterjee, A., & Ahmed, B. S. (2022) · Internet of Things · doi:10.1016/j.iot.2022.100568 · Cited by 323
“…health [55, 5] Robotics and manufacturing [56, 57, 58] Detection applications for IoT sensors [59, 60] Other industrial or manufacturing [61, 62, 63, 64] Surveillance and video [65, 66, 67] General-purpose frameworks [68, 69, 70] Network and communication frameworks [71, 72, 73, 74] User…”
BackgroundMethodZhang, C., Wang, X., Zhang, H., Zhang, H., & Han, P. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3125967 · Cited by 49
SXAD: Shapely eXplainable AI-Based Anomaly Detection Using Log Data
Alam, K., Kifayat, K., Sampedro, G. A., Karovič, V., & Naeem, T. (2024) · IEEE Access · doi:10.1109/access.2024.3425472 · Cited by 32
“Anomaly detection analyses log data of heterogeneous systems such as system logs [4], [18], [19], [20], [21], event logs [22] and various other logs [23], [24], [25], [26], [27].”
BackgroundMethodLogRESP-Agent: A Recursive AI Framework for Context-Aware Log Anomaly Detection and TTP Analysis
Lee, J., Jeong, Y., Han, T., & Lee, T. (2025) · doi:10.20944/preprints202506.0235.v1 · Cited by 9
Metadata
- Authors
- Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P. K., Alfarraj, O., & Tolba, A.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2020-04-26
- Volume
- 20
- Issue
- 9
- Pages
- 2451
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W3018898752
- Cited by
- 107
- References
- 6
- FWCI
- 7.90
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things. https://app.cochat.ai/papers/10.3390/s20092451
- The paper
- Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P. K., Alfarraj, O., & Tolba, A. (2020). LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things. Sensors. https://doi.org/10.3390/s20092451
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.