CoChat
Sign Up
PreprintCited by 1Digital and Cyber ForensicsNetwork Security and Intrusion DetectionInformation and Cyber Security

LLM-driven Provenance Forensics for Threat Investigation and Detection

Open access
Mukherjee, Kunal, Kantarcioglu, Murat·2025·arXiv
DOI 10.48550/arxiv.2508.21323arXiv 2508.21323
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce PROVSEEK, an LLM-powered agentic framework for automated provenance-driven forensic analysis and threat intelligence extraction. PROVSEEK employs specialized toolchains to dynamically retrieve relevant context by generating precise, context-aware queries that fuse knowledge from threat reports with evidence from system provenance data. The framework resolves provenance queries, orchestrates multiple role-specific agents, and synthesizes structured, ground-truth verifiable forensic summaries. By combining agent orchestration with Retrieval-Augmented Generation (RAG) and chain-of-thought (CoT) reasoning, data-guided filtration using a behavioral model, PROVSEEK enables adaptive multi-step analysis that iteratively refines hypotheses, verifies supporting evidence, and produces scalable, interpretable forensic explanations of attack behaviors. PROVSEEK is designed for automated threat investigation without task-specific training data, enabling forensic-style investigation even when no prior knowledge of the environment. We conduct a comprehensive evaluation on publicly available DARPA datasets, demonstrating that PROVSEEK outperforms retrieval-based methods for the intelligence extraction task, achieving a 34% improvement in contextual precision/recall; and for threat detection task, PROVSEEK achieves 22%/29% higher precision/recall compared to both a baseline agent approach and State-Of-The-Art (SOTA) Provenance-based Intrusion Detection System (PIDS). In our scalability study, we show that PROVSEEK increases token usage by 1.42x and latency by 1.63x as the database size increases 50x, making it optimal for large-scale deployment. We also conducted an ablation and error analysis study to show how different components of PROVSEEK affect the detection performance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2508.21323

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The latest research positions LLMs as reverse causal reasoning engines or reasoning hubs, realizing automated Root Cause Analysis (RCA) by scheduling external tools.

    “the latest research positions them as reverse causal reasoning engines or reasoning hubs, realizing automated Root Cause Analysis (RCA) by scheduling external tools.”

    From From Correlation to Intervention: A Survey on Log-Based Causal Analysis for Security and AIOps (2026) · cites this paper as [42]

    The Background (p. 3) calls the LLM a reasoning hub that issues queries and retrieves provenance evidence, and the Problem Statement (p. 3) describes an LLM planner that sequences tool calls for forensic investigation. The Introduction (p. 1) says provenance data enables reconstruction of attack chains, but the paper does not characterize the LLM as a reverse causal reasoning engine or establish automated Root Cause Analysis as a distinct capability. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • PROVSEEK positions the LLM as a scheduling hub, coordinating specialized tools like digital forensics and database queries through Chain of Thought (CoT) to simulate the iterative investigative process of human experts.

    “PROVSEEK[42] positions the LLM as a scheduling hub, coordinating specialized tools like digital forensics and database queries through Chain of Thought (CoT) to simulate the iterative investigative process of human experts.”

    From From Correlation to Intervention: A Survey on Log-Based Causal Analysis for Security and AIOps (2026) · cites this paper as [42]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 40

  1. [3][3] M. A. Inam et al., 'Sok: History is a vast early warning system: Auditing the provenance of system intrusions,' in IEEE Symposium on Security and Privacy (SP) , 2023. -
  2. [4][4] T. Bilot et al., 'Sometimes simpler is better: A comprehensive analysis of state-of-the-art provenancebased intrusion detection systems,' in 34th USENIX Security Symposium (SEC) , 2025. -
  3. [6][6] X. Han et al., 'Sigl: Securing software installations through deep graph learning,' in 30th USENIX Security Symposium (SEC) , 2021. -
  4. [7][7] Q. Wang et al., 'You are what you do: Hunting stealthy malware via data provenance analysis,' in Network and Distributed System Security Symposium (NDSS) , Feb. 2020. -
  5. [8][8] K. Mukherjee et al., 'Evading provenance-based ml detectors with adversarial system actions,' in 32nd USENIX Security Symposium (SEC) , 2023. -
  6. [9][9] M. U. Rehman, H. Ahmadi, and W. U. Hassan, 'Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,' in IEEE Symposium on Security and Privacy (SP) , 2024. -
  7. [10][10] Z. Jia, Y. Xiong, Y. Nan, Y. Zhang, J. Zhao, and M. Wen, ' { Magic } : Detecting advanced persistent threats via masked graph representation learning,' in 32nd USENIX Security Symposium (SEC) , 2024. -
  8. [11][11] B. Jiang et al., 'ORTHRUS: Achieving High Quality of Attribution in Provenance-based Intrusion Detection Systems,' in 34th USENIX Security Symposium (SEC) , 2025. -
  9. [12][12] L. Wang et al., 'Incorporating gradients to rules: Towards lightweight, adaptive provenance-based intrusion detection,' in Network and Distributed System Security Symposium (NDSS) , 2025. -
  10. [13][13] K. Mukherjee et al., 'Proviot: Detecting stealthy attacks in iot through federated edge-cloud security,' in International Conference on Applied Cryptography and Network Security , Springer, 2024. -
  11. [14][14] Z. Cheng et al., 'Kairos: Practical intrusion detection and investigation using whole-system provenance,' in IEEE Symposium on Security and Privacy (SP) , 2024. -
  12. [17][17] J. Zengy et al., 'Shadewatcher: Recommendationguided cyber threat analysis using system audit records,' in IEEE Symposium on Security and Privacy (SP) , 2022. -
  13. [18][18] J. Zeng, Z. L. Chua, Y. Chen, K. Ji, Z. Liang, and J. Mao, 'Watson: Abstracting behaviors from audit logs via aggregation of contextual semantics,' in Network and Distributed System Security Symposium (NDSS) , Feb. 2021. -
  14. [19][19] A. Goyal, X. Han, G. Wang, and A. Bates, 'Sometimes, you aren't what you do: Mimicry attacks against provenance graph host intrusion detection systems,' in Network and Distributed System Security Symposium (NDSS) , Feb. 2023. -
  15. [21][21] F. Yang, J. Xu, C. Xiong, Z. Li, and K. Zhang, ' { Prographer } : An anomaly detection system based on provenance graph embedding,' in 32nd USENIX Security Symposium (SEC) , 2023. -
  16. [22][22] A. Goyal, G. Wang, and A. Bates, 'R-caid: Embedding root cause analysis within provenance-based intrusion detection,' in IEEE Symposium on Security and Privacy (SP) , 2024. -
  17. [23][23] K. Mukherjee, J. Yu, P. De, and D. M. Divakaran, 'Provdp: Differential privacy for system provenance dataset,' in International Conference on Applied Cryptography and Network Security , Springer, 2025, pp. 189-219. -
  18. [24][24] T. Wang et al., 'Provcreator: Synthesizing complex heterogenous graphs with node and edge attributes,' arXiv preprint arXiv:2507.20967 , 2025. - · arXiv
  19. [25][25] OpenAI, 'Gpt-4 technical report,' arXiv preprint arXiv:2303.08774 , 2023. [Online]. Available: https: //arxiv.org/abs/2303.08774 - · arXiv
  20. [26][26] Y. Gao et al., 'Retrieval-augmented generation for large language models: A survey,' arXiv preprint arXiv:2312.10997 , 2023. [Online]. Available: https: //arxiv.org/abs/2312.10997 - · arXiv
  21. [27][27] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, 'Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,' ACM Computing Surveys , vol. 55, no. 9, pp. 1-35, 2023. DOI: 10 . 1145 / 3560815 [Online]. Available: https://arxiv.org/abs/2107.13586 - · arXiv
  22. [28][28] J. Wei et al., 'Chain-of-thought prompting elicits reasoning in large language models,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 35, 2022, pp. 24 824-24 837. [Online]. Available: https://arxiv.org/abs/2201.11903 - · arXiv
  23. [29][29] P. Lewis et al., 'Retrieval-augmented generation for knowledge-intensive NLP tasks,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 33, 2020, pp. 9459-9474. [Online]. Available: https://arxiv.org/abs/2005.11401 - · arXiv
  24. [30][30] Y. Cheng, O. Bajaber, S. A. Tsegai, D. Song, and P. Gao, 'CTINEXUS: Leveraging optimized LLM in-context learning for constructing cybersecurity knowledge graphs under data scarcity,' arXiv preprint arXiv:2410.21060 , 2024. - · arXiv
  25. [36][36] X. Han, T. Pasquier, A. Bates, J. Mickens, and M. Seltzer, 'Unicorn: Runtime provenance-based detector for advanced persistent threats,' in Network and Distributed System Security Symposium (NDSS) , Feb. 2020. -
  26. [37][37] T. Pasquier et al., 'Runtime analysis of whole-system provenance,' in ACM Conference on Computer and Communications Security (CCS) , 2018. -
  27. [38][38] Z. Xu, P. Fang, C. Liu, X. Xiao, Y. Wen, and D. Meng, 'Depcomm: Graph summarization on system audit logs for attack investigation,' in IEEE Symposium on Security and Privacy (SP) , 2022. -
  28. [39][39] P. Fang et al., 'Back-propagating system dependency impact for attack investigation,' in USENIX Security Symposium , 2022. -
  29. [40][40] W. U. Hassan et al., 'NoDoze: Combatting threat alert fatigue with automated provenance triage,' in Network and Distributed System Security Symposium (NDSS) , 2019. -
  30. [41][41] W. U. Hassan, M. A. Noureddine, P. Datta, and A. Bates, 'OmegaLog: High-fidelity attack investigation via transparent multi-layer log analysis,' in Network and Distributed System Security Symposium (NDSS) , 2020. -
  31. [42][42] P. Gao et al., 'SAQL: A stream-based query system for real-time abnormal system behavior detection,' in USENIX Security Symposium , 2018. -
  32. [43][43] J. Lin and D. Mohaisen, 'From large to mammoth: A comparative evaluation of large language models in vulnerability detection,' in Network and Distributed System Security (NDSS) Symposium , 2025. -
  33. [44][44] S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini, 'Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,' in 2024 IEEE symposium on security and privacy (SP) , IEEE, 2024, pp. 862-880. -
  34. [45][45] Y. Oliinyk, M. Scott, R. Tsang, C. Fang, H. Homayoun, et al., 'Fuzzing busybox: Leveraging LLM and crash reuse for embedded bug unearthing,' in 33rd USENIX Security Symposium (USENIX Security 24) , 2024, pp. 883-900. -
  35. [46][46] H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, 'Examining zero-shot vulnerability repair with large language models,' in 2023 IEEE Symposium on Security and Privacy (SP) , IEEE, 2023, pp. 2339-2356. -
  36. [47][47] Y. Li et al., 'KnowPhish: Large language models meet multimodal knowledge graphs for enhancing reference-based phishing detection,' in 33rd USENIX Security Symposium (USENIX Security 24) , 2024, pp. 793-810. -
  37. [48][48] G. Deng et al., 'PentestGPT: Evaluating and harnessing large language models for automated penetration testing,' in USENIX Security Symposium , 2024, pp. 847-864. -
  38. [49][49] P. Hu, R. Liang, and K. Chen, 'DeGPT: Optimizing decompiler output with LLM,' in Proceedings 2024 Network and Distributed System Security Symposium , 2024. -
  39. [50][50] T. Kohno, Y. Acar, and W. Loh, 'Ethical frameworks and computer security trolley problems: Foundations for conversations,' in 32nd USENIX Security Symposium (USENIX Security 23) , 2023, pp. 5145-5162. -
  40. [58][58] S. T. King and P. M. Chen, 'Backtracking intrusions,' in USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2003. Figure 10: PROVSEEK chat templates used for the intro message. <!-- image -->

Cited by

Sorted by citations

Related

Authors
Mukherjee, K., & Kantarcioglu, M.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2508.21323
arXiv
2508.21323
OpenAlex
W4415989282
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Digital and Cyber Forensics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
Keywords
digital forensics, threat detection, LLM agents, multi-agent orchestration, retrieval-augmented generation, chain-of-thought reasoning, intrusion detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review