Mining Causality of Network Events in Log Data
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Network log messages (e.g., syslog) are expected to be valuable and useful information to detect unexpected or anomalous behavior in large scale networks. However, because of the huge amount of system log data collected in daily operation, it is not easy to extract pinpoint system failures or to identify their causes. In this paper, we propose a method for extracting the pinpoint failures and identifying their causes from network syslog data. The methodology proposed in this paper relies on causal inference that reconstructs causality of network events from a set of time series of events. Causal inference can filter out accidentally correlated events, thus it outputs more plausible causal events than traditional cross-correlation-based approaches can. We apply our method to 15 months' worth of network syslog data obtained from a nationwide academic network in Japan. The proposed method significantly reduces the number of pseudo correlated events compared with the traditional methods. Also, through three case studies and comparison with trouble ticket data, we demonstrate the effectiveness of the proposed method for practical network operation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Tickets and change records capture intent, approval, and accountability [55], [65].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [65]
In the citing paper
“They also help define operational ground truth, including whether mitigation was accepted, whether a change was rolled back, and whether a similar incident later recurred [65], [66].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [65]
In the citing paper
“NetOps practitioners assess risk using criteria that align with autonomy decisions, such as the number of affected devices, novelty of the change, the adequacy of verification procedures, and the history of similar changes causing incidents [73], [70], [65].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [65]
Citations
Cites
Collective dynamics of ‘small-world’ networks
Watts, D. J., & Strogatz, S. H. (1998) · Nature · doi:10.1038/30918 · Cited by 43,060
(2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_100245 · Cited by 1,424
An Algorithm for Fast Recovery of Sparse Causal Graphs
Spirtes, P., & Glymour, C. (1991) · Social Science Computer Review · doi:10.1177/089443939100900106 · Cited by 953
Estimating high-dimensional directed acyclic graphs with the PC-algorithm
Kalisch, M., & Bühlmann, P. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.math/0510436 · Cited by 591
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J.-G., Wang, Y., & Li, J. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.60 · Cited by 542
Ramoni, M. F., & Sebastiani, P. (2005) · Encyclopedia of Data Warehousing and Mining · doi:10.4018/978-1-59140-557-3.ch128 · Cited by 541
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764) · doi:10.1109/ipom.2003.1251233 · Cited by 453
Order-independent constraint-based causal structure learning
Colombo, D., & Maathuis, M. H. (2014) · arXiv (Cornell University) · doi:10.5555/2627435.2750365 · Cited by 439
Clustering event logs using iterative partitioning
Makanju, A. A. O., Zincir-Heywood, A. N., & Milios, E. E. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557154 · Cited by 338
Estimating High-Dimensional Directed Acyclic Graphs with the PC-Algorithm
KalischMarkus, & BühlmannPeter (2007) · Journal of Machine Learning Research · doi:10.5555/1248659.1248681 · Cited by 330
Structured comparative analysis of systems logs to diagnose performance problems
Nagaraj, K., Killian, C. E., & Neville, J. (2012) · Purdue e-Pubs (Purdue University System) · Cited by 289
An Algorithm for Deciding if a Set of Observed Independencies Has a Causal Explanation
Verma, T., & Pearl, J. (1992) · Uncertainty in Artificial Intelligence · doi:10.1016/b978-1-4832-8287-9.50049-9 · Cited by 231
Evolve or Die: High-Availability Design Principles Drawn from Googles Network Infrastructure
Govindan, R., Minei, I., Kallahalla, M., Koley, B., & Vahdat, A. (2016) · Proceedings of the 2016 ACM SIGCOMM Conference · doi:10.1145/2934872.2934891 · Cited by 215
Dynamic syslog mining for network failure monitoring
Yamanishi, K., & Maruyama, Y. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081927 · Cited by 213
SherLog: error diagnosis by connecting clues from run-time logs
Yuan, D., Mai, H., Xiong, W., Tan, L., Zhou, Y., & Pasupathy, S. (2010) · ACM SIGARCH Computer Architecture News · doi:10.1145/1735970.1736038 · Cited by 212
Log-based predictive maintenance
Sipos, R., Fradkin, D., Moerchen, F., & Wang, Z. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623340 · Cited by 199
Gerhards, R. (2009) · doi:10.17487/rfc5424 · Cited by 192
Inferring models of concurrent systems from logs of their behavior with CSight
Beschastnikh, I., Brun, Y., Ernst, M. D., & Krishnamurthy, A. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568246 · Cited by 181
Failure prediction based on log files using Random Indexing and Support Vector Machines
Fronza, I., Sillitti, A., Succi, G., Terho, M., & Vlasenko, J. (2012) · Journal of Systems and Software · doi:10.1016/j.jss.2012.06.025 · Cited by 149
Towards automated performance diagnosis in a large IPTV network
Mahimkar, A. A., Ge, Z., Shaikh, A., Wang, J., Yates, J., Zhang, Y., & Zhao, Q. (2009) · Proceedings of the ACM SIGCOMM 2009 conference on Data communication · doi:10.1145/1592568.1592596 · Cited by 147
Chen, P., Qi, Y., Zheng, P., & Hou, D. (2014) · IEEE INFOCOM 2014 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2014.6848128 · Cited by 140
Incremental Mining of System Log Format
Mizutani, M. (2013) · 2013 IEEE International Conference on Services Computing · doi:10.1109/scc.2013.73 · Cited by 139
A Fast PC Algorithm for High Dimensional Causal Discovery with Multi-Core PCs
Le, T. D., Hoang, T., Li, J., Liu, L., Liu, H., & Hu, S. (2019) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2016.2591526 · Cited by 135
Using Hidden Semi-Markov Models for Effective Online Failure Prediction
Salfner, F., & Malek, M. (2007) · 2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007) · doi:10.1109/srds.2007.35 · Cited by 133
Troubleshooting blackbox SDN control software with minimal causal sequences
Scott, C., Wundsam, A., Raghavan, B., Panda, A., Or, A., Lai, J., Huang, E., Liu, Z., El-Hassany, A., Whitlock, S., et al. (2014) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2619239.2626304 · Cited by 126
PC algorithm for nonparanormal graphical models
Harris, N., & Drton, M. (2013) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · doi:10.5555/2567709.2567770 · Cited by 101
Mining dependency in distributed systems through unstructured logs analysis
Lou, J.-G., Fu, Q., Wang, Y., & Li, J. (2010) · ACM SIGOPS Operating Systems Review · doi:10.1145/1740390.1740411 · Cited by 98
Spatio-temporal factorization of log data for understanding network events
Kimura, T., Ishibashi, K., Mori, T., Sawada, H., Toyono, T., Nishimatsu, K., Watanabe, A., Shimoda, A., & Shiomoto, K. (2014) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2014.6847986 · Cited by 87
Proactive failure detection learning generation patterns of large-scale network logs
Kimura, T., Watanabe, A., Toyono, T., & Ishibashi, K. (2015) · International Conference on Network and Service Management (CNSM) · doi:10.1109/cnsm.2015.7367332 · Cited by 76
Insider Threat Detection Using Log Analysis and Event Correlation
Ambre, A., & Shekokar, N. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.03.175 · Cited by 69
LOGAN: Problem Diagnosis in the Cloud Using Log-Based Reference Models
Tak, B. C., Tao, S., Yang, L., Zhu, C., & Ruan, Y. (2016) · 2016 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2016.12 · Cited by 54
Diagnosing the root-causes of failures from cluster log files
Chuah, E., Kuo, S.-H., Hiew, P., Tjhi, W.-C., Lee, G., Hammond, J., Michalewicz, M. T., Hung, T., & Browne, J. C. (2010) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc.2010.5713159 · Cited by 53
Towards an NLP-based log template generation algorithm for system log analysis
Kobayashi, S., Fukuda, K., & Esaki, H. (2014) · Proceedings of The Ninth International Conference on Future Internet Technologies · doi:10.1145/2619287.2619290 · Cited by 38
Self-citationSome Variations on the PC Algorithm.
Abellán, J., Gómez‐Olmedo, M., & Moral, S. (2006) · Cited by 37
3-Dimensional root cause diagnosis via co-analysis
Zheng, Z., Yu, L., Lan, Z., et al. (2012) · International Conference on Autonomic Computing · doi:10.1145/2371536.2371571 · Cited by 33
Minimizing faulty executions of distributed systems
Scott, C., Panda, A., Brajkovic, V., et al. (2016) · Networked Systems Design and Implementation · Cited by 31
Mining causes of network events in log data with causal inference
Kobayashi, S., Fukuda, K., & Esaki, H. (2017) · IFIP/IEEE International Symposium on Integrated Network Management (IM) · doi:10.23919/inm.2017.7987263 · Cited by 25
Self-citationSINET5: A low-latency and high-bandwidth backbone network for SDN/NFV Era
Kurimoto, T., Urushidani, S., Yamada, H., Yamanaka, K., Nakamura, M., Abe, S., Fukuda, K., Koibuchi, M., Takakura, H., Yamada, S., et al. (2017) · International Conference on Communications · doi:10.1109/icc.2017.7996843 · Cited by 24
Self-citationHighly available network design and resource management of SINET4
Urushidani, S., Aoki, M., Fukuda, K., Abe, S., Nakamura, M., Koibuchi, M., Ji, Y., & Yamada, S. (2013) · Telecommunication Systems · doi:10.1007/s11235-013-9817-8 · Cited by 23
Self-citationImpact of Tohoku earthquake on R&E network in Japan
Fukuda, K., Aoki, M., Abe, S., et al. (2011) · doi:10.1145/2079360.2079361 · Cited by 19
Self-citationOperational Security Log Analytics for Enterprise Breach Detection
Li, Z., & Oprea, A. (2016) · IEEE Cybersecurity Development (SecDev) · doi:10.1109/secdev.2016.015 · Cited by 17
Research works printed but not linked: 1
- heckerman, 2000, Dependency networks for inference, collaborative filtering, and data visualization, J Mach Learn Res, 1, 49
Cited by
Causal analysis of network logs with layered protocols and topology knowledge
Kobayashi, S., Otomo, K., & Fukuda, K. (2019) · 2019 15th International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm46954.2019.9012718 · Cited by 13
Self-citationLarge language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
“A causal diagnosis method aims to recover this propagation structure, so that the output is not only a ranked root cause but also an explanation path linking symptoms to candidate causes [49], [50].”
Influential
Metadata
- Authors
- Kobayashi, S., Otomo, K., Fukuda, K., & Esaki, H.
- Affiliations
- The University of Tokyo (JP)
- National Institute of Informatics (JP)
- Venue
- IEEE Transactions on Network and Service Management
- Published
- 2018-03
- Volume
- 15
- Issue
- 1
- Pages
- 53-67
- ISSN
- 1932-4537 (ISSN-L)
- OpenAlex
- W2771013155
- Cited by
- 63
- References
- 41
- Citations by year
- 2026: 7 · 2025: 5 · 2024: 5 · 2023: 7 · 2022: 10 · 2021: 11 · 2020: 8 · 2019: 7 · 2018: 3
- FWCI
- 3.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Network and Service Management (publishedVersion)
- Funders
- NTT
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Computer Networks and Communications › Mobile Agent-Based Network Management
- Keywords
- syslog, causal inference, anomaly detection, cross correlation function, academic networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.