The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index; 12 of the 32 sources its claims cite were checked for support.
Practitioners’ Expectations on Log Anomaly Detection
Open accessAbstract
Log anomaly detection has become a common practice for software engineers to analyze software system behavior. Despite significant research efforts in log anomaly detection over the past decade, it remains unclear what are practitioners’ expectations on log anomaly detection and whether current research meets their needs. To fill this gap, we conduct an empirical study, surveying 312 practitioners from 36 countries about their expectations on log anomaly detection. In particular, we investigate various factors influencing practitioners’ willingness to adopt log anomaly detection tools. We then perform a literature review on log anomaly detection, focusing on publications in premier venues from 2015 to 2025, to compare practitioners’ needs with the current state of research. Based on this comparison, we highlight the directions for researchers to focus on to develop log anomaly detection techniques that better meet practitioners’ expectations.
Overview
CoChat Summary
TL;DR
This empirical study surveys 312 practitioners from 36 countries and interviews 15 professionals to identify expectations for log anomaly detection, then compares those expectations with 36 premier-venue papers from 2014–2024. It finds strong practitioner demand for automated tools but major gaps in interpretability, generalizability, customization, privacy, and use of metrics/traces.
Key findings
- 95.5% of surveyed practitioners consider automated log anomaly detection tools essential or worthwhile, yet only 55.8% currently use log monitoring tools for anomaly detection; one-third of those tools cannot automatically identify anomalies.
- Practitioners prioritize correct detection: over 70% expect recall and precision above 60%, and 70.5% prefer log sequence-level detection, with 29.5% preferring log event-level detection.
- Data availability exceeds research use: 83.7% and 74.9% of practitioners report metrics and traces as always or sometimes available, but only 4 of 36 reviewed papers incorporate metrics or traces.
- The literature review finds only 6 papers explicitly perform log event-level detection, over half do not mention detection time, and few or no papers address interpretability, generalizability, customization, or privacy protection.
Methods
- Mixed-methods design: semi-structured interviews with 15 professionals from eight IT companies (average 8.07 years experience), followed by an online survey of 312 practitioners from 36 countries.
- Survey analysis used percentages for single/multiple-choice questions and Likert-scale charts; open-ended responses were qualitatively analyzed, with “I don’t know” ratings excluded (<1% of valid responses).
- Literature review searched IEEE, ACM, DBLP, and Google Scholar for 2014–2024 premier-venue publications, applied inclusion criteria and forward snowballing, and identified 36 papers; two authors analyzed capabilities across nine factors.
Limitations
- The paper states that some practitioners may not fully comprehend the questions, mitigated by instructions, a flow chart, and excluding “I don’t know” responses.
- The paper states the sample does not encompass all software engineers, is limited to company practitioners and GitHub contributors, and excludes practitioners not proficient in English or Chinese; additional adoption factors may exist.
- Evident from the text: recruitment through social/professional circles and GitHub contributors may introduce selection bias, and the literature review is restricted to premier venues and specific inclusion criteria, so it may not represent all log anomaly detection research.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
3 Valid · 9 Not valid · 9 Not checked · 163 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“BGL logs primarily contain information related to the RAS kernel”
§ Practitioners' Expectations on Log Anomaly Detection > Generalizability.
What the sources say
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
Our copy of Wohlin (2014) is entirely about snowballing as a search approach for systematic literature studies in software engineering, with a replication of MacDonell et al.'s cross-company effort-estimation review (Sections 2-4.10). No passage mentions BGL logs, log analysis, or an RAS kernel; searches for 'BGL logs RAS kernel' and 'log data analysis system reliability' returned only snowballing/reliability passages (2. RELATED WORK, 4.1 Introduction, 4.3.1, 7. ACKNOWLEDGMENTS, General Terms). The claim is therefore not addressed by this source at all, and our copy names no alternative work for it. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“industrial datasets are often access-restricted due to security and privacy concerns, as highlighted by Lee et al. [5].”
§ Practitioners' Expectations on Log Anomaly Detection > Generalizability.
What the sources say
- Supportedcites Lee et al. [5] source
Section III, Data Collection, explicitly states that most industrial datasets are access-restricted and links the limited availability of public data to security and privacy concerns. This supports the citing sentence's paraphrase. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
§ Practitioners' Expectations on Log Anomaly Detection > Generalizability.
What the sources say
- Could not verifycites [19] sourceExperimental — accuracy not yet measured
The Data sources section counts 66 papers used to derive an IoT development framework; it does not identify four papers evaluating techniques across domains. The Testing section discusses IoT hardware and software testing, but not training on a source domain and testing on a target domain. Searches for that phrase and for cross-domain evaluation found no supporting passage in our copy. (LLM, conf=0.76)
- Could not verifycites [20] sourceExperimental — accuracy not yet measured
Section 3.2 describes a survey of mobile-application performance optimization publications, but does not count papers evaluated across source and target domains. Searches for “source domain,” “target domain,” and cross-domain evaluation found no passage in our copy addressing the claimed four-paper count or training-and-testing setup. (LLM, conf=0.75)
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
Sections 4.1 and 4.2 describe a replication of a literature review about cross-company and within-company effort-estimation models. Section 4.10 compares literature-review methods, not techniques trained on one domain and tested on another. Our copy does not establish that this paper is one of four papers conducting the cross-domain evaluation claimed. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While 2 papers [19], [36] achieve satisfactory results on public datasets like HDFS”
§ Practitioners' Expectations on Log Anomaly Detection > Generalizability.
What the sources say
- Could not verifycites [19] sourceExperimental — accuracy not yet measured
The Step 2 passage mentions Hadoop Distributed File System only as a technology discussed in an IoT development study, not as a public dataset. The Data sources and Developers’ perception sections do not report two papers achieving satisfactory results on HDFS; searches for HDFS datasets and evaluation results found no supporting passage. Page numbers were unavailable in our copy. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [36] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and anomaly detection time to be under 5 seconds when an anomaly appears.”
§ Practitioners' Expectations on Log Anomaly Detection > Scalability.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
123 more claims
Create a free account to see all 184 claims and to verify the 4 references still open.
- Probably not a substantial claim. 29
- It may state two separate claims as one. 33
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Determines operational usability[41],[146]”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [146]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Online evaluation becomes necessary once an agent proposes actions, because action quality depends on the behaviour of the system being changed [105], [41], [146].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [146]
Experimental — accuracy not yet measured
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“Starting from these 35 articles, we perform one level of forward snowballing [22] and gather 71 additional publications.”
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Regarding interpretability , integrating post-hoc eXplainable Artificial Intelligence (XAI) methods, such as Local Interpretable Model-agnostic Explanations (LIME) [74], SHapley Additive exPla-nations (SHAP) [75], and Anchors [76], with DL models can enhance the understanding of model behavior.”
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835 · cited 2×
“L OGS are a crucial source of information in software systems, capturing system runtime behavior and aiding in software engineering tasks like program comprehension [1], [2], anomaly detection [3], [4], [5], [6], and failure diagnosis [7], [8].”
Qualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493 · cited 1×
“We generate our interview questions from two sources: first principles and standard open-ended interviews [13].”
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 2×
“Most of the papers evaluate their techniques on two public datasets, Hadoop Distributed File System (HDFS) [27] and Blue Gene/L supercomputer (BGL) [28], typically grouping logs into sequences using sessions and window sizes.”
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761 · cited 1×
“Zhang et al. [50] integrated attention mechanisms with a Bi-LSTM model to capture comprehensive log sequences bidirectionally.”
LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs
Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., Sun, P., et al. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/658 · Cited by 708 · cited 2×
“L OGS are a crucial source of information in software systems, capturing system runtime behavior and aiding in software engineering tasks like program comprehension [1], [2], anomaly detection [3], [4], [5], [6], and failure diagnosis [7], [8].”
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · cited 1×
“…of LLMs to aid in the development of log anomaly detection techniques, given their increasing popularity for software engineering tasks [68], [69], [70], particularly in log analysis-related activities, such as employing zero-shot learning for log parsing [71] and integrating inter-pretable domain…”
Research works printed but not linked: 2
- Leavy, 2022, Research Design: Quantitative, Qualitative, Mixed Methods, Arts-Based, and Community-Based Participatory Research Approaches
- Spencer, 2009, Card Sorting: Designing Usable Categories
Cited by
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
“Logs remain the most common operational signal, and also one of the least forgiving [62], [63], [127], [128].”
InfluentialLightweight Techniques for Federated Anomaly Detection in Log Data
Pustozerova, A., Gómez, A. G., Landauer, M., Wurzenberger, M., Skopik, F., Weippl, E. R., Mayer, R. L., & Ekelhart, A. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3677674 · Cited by 4
“According to [40], anomaly detection systems must work quickly and with minimal labeled data, as such data are often scarce.”
InfluentialLLM4Log: A Systematic Review of Large Language Model-based Log Analysis
Ma, Z., Yang, J., & Chen, T.-H. P. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3842759 · Cited by 6
Wang, L., Zhou, X., Zhao, X., & He, Y. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133840 · Cited by 1
Program Anomalies Detection by Fuzzy Conceptual Learning From Correct Traces in a Bounded Domain
Islam, F., Alban, A. Q., Malluhi, Q. M., & Jaoua, A. (2025) · IEEE Access · doi:10.1109/access.2025.3640722
“Several methods for detecting anomalies build a training dataset from different sources, such as the states of a network, system [7], or software [8].”
Metadata
- Authors
- Ma, X., Li, Y., Keung, J., Yu, X., Zou, H., Yang, Z., Sarro, F., & Barr, E. T.
- Affiliations
- Hong Kong Metropolitan University (HK)
- City University of Hong Kong (HK)
- Zhejiang University (CN)
- Johns Hopkins University (US)
- Shandong University (CN)
- University College London (GB)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2025-09
- Volume
- 51
- Issue
- 9
- Pages
- 2455-2471
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4413478212
- Cited by
- 21
- References
- 84
- Citations by year
- 2026: 16 · 2025: 5
- FWCI
- 6.57
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Research Ethics Committee of the Hong Kong Metropolitan University (HE-SF2025/19)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- log anomaly detection, practitioner expectations, technology adoption, software logging, software monitoring, research-practice gap
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Practitioners’ Expectations on Log Anomaly Detection. https://app.cochat.ai/papers/10.1109/tse.2025.3586700
- The paper
- Ma, X., Li, Y., Keung, J., Yu, X., Zou, H., Yang, Z., Sarro, F., & Barr, E. T. (2025). Practitioners’ Expectations on Log Anomaly Detection. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2025.3586700
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.