The DOI matches its Crossref record; 58 of the 64 references it cites were found in a registry or an index; 12 of the 36 sources its claims cite were checked for support.
Log-based Anomaly Detection Without Log Parsing
Open accessAbstract
Software systems often record important runtime information in system logs for troubleshooting purposes. There have been many studies that use log data to construct machine learning models for detecting system anomalies. Through our empirical study, we find that existing log-based anomaly detection approaches are significantly affected by log parsing errors that are introduced by 1) OOV (out-of-vocabulary) words, and 2) semantic misunderstandings. The log parsing errors could cause the loss of important information for anomaly detection. To address the limitations of existing methods, we propose NeuralLog, a novel log-based anomaly detection approach that does not require log parsing. NeuralLog extracts the semantic meaning of raw log messages and represents them as semantic vectors. These representation vectors are then used to detect anomalies through a Transformer-based classification model, which can capture the contextual information from log sequences. Our experimental results show that the proposed approach can effectively understand the semantic meaning of log messages and achieve accurate anomaly detection results. Overall, NeuralLog achieves F1-scores greater than 0.95 on four public datasets, outperforming the existing approaches.
Overview
CoChat Summary
TL;DR
NeuralLog is a log-based anomaly detection approach that skips log parsing by encoding raw log messages with BERT/WordPiece and classifying log sequences with a Transformer. It reports F1-scores above 0.95 on four public datasets and outperforms SVM, LR, IM, LogRobust, and Log2Vec on BGL, Thunderbird, and Spirit.
Key findings
- An empirical study on BGL and Thunderbird found that existing log parsers suffer from OOV words and semantic misunderstanding; on a 60/40 BGL split, 94.12% of unique testing words were unseen in training, and these OOV words produced 1,304 unseen log events, accounting for 86.59% of testing log events.
- NeuralLog achieved F1-scores greater than 0.95 on all four public datasets: HDFS 0.98, BGL 0.98, Thunderbird 0.96, and Spirit 0.97; it was best on BGL, Thunderbird, and Spirit and comparable on HDFS.
- Parsing errors degraded baselines: SVM with Drain achieved F1 0.46 on BGL and 0.50 on Thunderbird, while LogRobust achieved F1 0.75 on BGL and 0.68 on Thunderbird; manually fixing parsing errors improved LogRobust by 25% and SVM by 29% on average on BGL.
- Ablations showed NeuralLog’s raw-message encoding mattered: NeuralLog-Template reached F1 0.90 on BGL and 0.90 on Thunderbird versus NeuralLog’s 0.98 and 0.96, and NeuralLog-Word2Vec reached F1 0.80 on Thunderbird versus NeuralLog’s 0.96.
Methods
- The paper first empirically studies log parsing errors on BGL and Thunderbird (14,672,653 log messages total, 353,394 labeled anomalies), evaluating Drain, AEL, IPLoM, and Spell and their impact on SVM and LogRobust.
- NeuralLog preprocesses raw log messages, applies WordPiece tokenization, encodes messages with BERT base (12 transformer layers, 768 hidden units), adds positional encoding, and uses a Transformer encoder-based classifier (12 attention heads, feed-forward size 2048) for anomaly detection.
- Evaluation uses four public datasets: HDFS, BGL, Thunderbird, and Spirit. HDFS sequences are built by block ID with an 80/20 train-test split; BGL, Thunderbird, and Spirit use the first 80% of time-sorted messages for training and the last 20% for testing, with sliding windows of length 20 and step size 1; metrics are Precision, Recall, and F1-score.
Limitations
- Stated: the subject datasets come from a distributed system and supercomputers, so the number of systems is limited and does not cover all domains.
- Stated: tool comparison relies on implementations and default settings from prior work, and manual labeling of the public datasets may introduce noise.
- Stated: preprocessing removes numeric and special-character tokens, which may discard useful information such as node ID, task ID, IP address, or exit code. No additional limitations are evident from the text beyond those stated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 8 Not valid · 18 Not checked · 105 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Moreover, DeepLog [5], using the indexes of log templates to predict the next log event, considers all new log events as anomalies because they cannot be predicted by the model.”
What the sources say
- Could not verifycites [5] sourceExperimental — accuracy not yet measured
The Introduction (p. 1) and Related Work (p. 2) discuss text classification and model compression, not DeepLog. Searches for DeepLog and for unseen log templates and anomaly detection found no passage addressing whether new log events are treated as anomalies. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The main reason is that LogRobust utilizes the Drain log parser [24] to obtain log templates. As aforementioned in Section III-C, the Drain parser could inaccurately parse a noticeable number of log messages on BGL and Thunderbird datasets.”
What the sources say
- Could not verifycites [24] sourceExperimental — accuracy not yet measured
The citation is attached to the identification of Drain as the log parser LogRobust uses. Our copy’s Results and Conclusion discuss subword word embeddings, not Drain or LogRobust; searches for those names and for log-template parsing found no relevant passage. It also provides no evidence about parsing errors on BGL or Thunderbird. (LLM, conf=0.76)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Therefore, Log2Vec could not effectively handle some domain-specific words, such as technical terms or entity names [50], [51], [52].”
What the sources say
- Could not verifycites [50] sourceExperimental — accuracy not yet measured
The Introduction (page not available) discusses the general difficulty word-embedding methods face with out-of-vocabulary terms. Searches for Log2Vec, domain-specific words, technical terms, and entity names found no passage assessing whether Log2Vec handles those words effectively. (LLM, conf=0.78)
- Could not verifycites [51] sourceExperimental — accuracy not yet measured
The References (p. 11) list Log2vec, but the paper does not describe its ability to handle domain-specific words. Searches for Log2vec alongside technical terms, entity names, and vocabulary limitations found no passage addressing the claimed limitation. (LLM, conf=0.76)
- Could not verifycites [52] sourceExperimental — accuracy not yet measured
Section 4.2 (p. 6) examines weight decay in SGD and Adam, while Section 4.5 (p. 8) discusses applications of AdamW. Searches for Log2Vec, domain-specific words, technical terms, and entity names found no passage addressing whether Log2Vec handles those words effectively. (LLM, conf=0.76)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, it is hard to extract contextual information from characters to form meaningful words [50], [51], [52].”
What the sources say
- Partly supportedcites [50] sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says character-based word models struggle in some settings, including where the mapping from form to meaning is complex. It does not say that extracting contextual information from characters is generally hard; the claim extends a specific limitation to a broader one. (LLM, conf=0.78)
- Could not verifycites [51] sourceExperimental — accuracy not yet measured
The Framework (pp. 3–4) describes extracting log keys from log messages and learning contextual relations among those keys. The Introduction (p. 1) discusses a different difficulty: RNNs do not encode context from both sides of a log entry. Searches for character-level context and forming words found no passage addressing the claimed difficulty. (LLM, conf=0.76)
- Could not verifycites [52] sourceExperimental — accuracy not yet measured
Section 4.3 (p. 7) discusses AdamW’s generalization, and Section 4.5 (p. 8) mentions its use in natural-language-understanding models. Neither addresses the difficulty of extracting context from characters to form words. Searches for the claim’s wording and for character-level word formation found no supporting passage. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
What the sources say
- Could not verifycites [49] sourceExperimental — accuracy not yet measured
The Workflow (p. 4) and Anomaly Detection (p. 8) describe feature vectors and PCA-based log anomaly detection, not Log2Vec. Searches for MIMICK, OOV words, and character-level word embeddings found no passage addressing the claimed adoption or its effect on performance. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
79 more claims
Create a free account to see all 132 claims and every source verdict.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 18
- The quote only partly states this claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Logs are a crucial source of information in software systems, capturing system runtime behavior and aiding in software engineering tasks like program comprehension [1], [2], anomaly detection [3]- [6], and failure diagnosis [7], [8].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [3]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“adopt Word2vec model [13] to obtain embedding vectors of log events, then applied an LSTM model to detect anomalies.”
MethodDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 3×
“LogRobust [8] incorporates a pre-trained Word2vec model, namely FastText [31], and combines with TF-IDF weight [32] for learning the representation vectors of log templates, which are generated by Drain [24].”
MethodUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · cited 4×
“For GPT2 and Roberta encoders, we use their base model with 12 layers, 12 attention heads, and 768 hidden units.”
MethodInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 4×
“In our work, we adopt the WordPiece tokenization [36], [37], which is widely used in many recent language modeling studies [38], [39], [40].”
MethodInfluentialDistributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934 · cited 5×
“Conventionally, words of log content are further transformed into vectors by using Word2Vec [42], then the representation vector of each sentence would be calculated based on the word vectors.”
MethodInfluential
Research works printed but not linked: 2
- radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1, 9
- 2021, Bert pretrained models
Cited by
Log-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257
“[26] found that log parsing errors can lead to many incorrect log events, thus downgrading anomaly detection performance.”
BackgroundMethodInfluentialSelf-citationEadro: An End-to-End Troubleshooting Framework for Microservices on Multi-source Data
Lee, C., Yang, T., Chen, Z., Su, Y., & Lyu, M. R. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00150 · Cited by 149
“Previous anomaly detection approaches are usually based on system logs [58]–[62] or KPIs [63]–[67], or both [68], targeting traditional distributed systems without complex invocation relationships.”
Log Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129
“Recent studies [16], [33] demonstrate that these pretrained models can be applied to understand the semantic meanings of log messages, thus favouring many downstream log analytics tasks.”
BackgroundMethodInfluentialSelf-citationLogFiT: Log Anomaly Detection Using Fine-Tuned Language Models
Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3358730 · Cited by 111
“Many of the approaches [8], [9], [11], [12] require log parsing, while recent studies [11], [16] suggested that log parsing can reduce accuracy.”
BackgroundMethodInfluentialBrain: Log Parsing With Bidirectional Parallel Tree
Yu, S., He, P., Chen, N., & Wu, Y. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3270566 · Cited by 110
“, the accuracy of log-based anomaly detection can decrease by more than 20% with just a few parsing errors [15].”
Background
Metadata
- Authors
- Le, V.-H., & Zhang, H.
- Affiliations
- University of Newcastle Australia (AU)
- Venue
- 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)
- Publisher
- IEEE
- Published
- 2021-11
- Pages
- 492-504
- OpenAlex
- W4205965165
- Semantic Scholar
- 7c93c88260829ea01429a6944f4525440817896c
- Cited by
- 361
- References
- 64
- Citations by year
- 2026: 85 · 2025: 107 · 2024: 78 · 2023: 66 · 2022: 23 · 2021: 2
- FWCI
- 32.57
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- NOVA (University of Newcastle Australia) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Australian Research Council
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- log anomaly detection, out-of-vocabulary words, system logs, anomaly detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Log-based Anomaly Detection Without Log Parsing. https://app.cochat.ai/papers/10.1109/ase51524.2021.9678773
- The paper
- Le, V.-H., & Zhang, H. (2021). Log-based Anomaly Detection Without Log Parsing. 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). https://doi.org/10.1109/ase51524.2021.9678773
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.