CoChat

Can Large Language Models Be Trusted for AI-Based Intrusion Detection in Cybersecurity? An Evidence-Grounded Log Analysis

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 3 of the 30 sources its claims cite were checked for support.

Abstract

Security Operations Centers (SOCs) rely heavily on system logs for intrusion detection, yet the growing volume and complexity of modern infrastructure logs make manual analysis increasingly impractical. While traditional intrusion detection systems provide automated alerts, they often lack contextual reasoning and interpretable explanations. Recent advances in large language models (LLMs) offer new opportunities for semantic log analysis and natural-language reasoning, motivating their use as automated SOC assistants. However, existing studies largely evaluate LLMs based on detection performance or qualitative usefulness, without rigorously assessing whether their explanations are grounded in actual log evidence. In this paper, we propose a systematic evaluation framework for applying closed-weight, web-accessible LLMs to intrusion detection from raw Linux system logs. Using standardized prompts and fixed log windows, we treat each LLM as a black-box SOC analyst that produces structured intrusion assessments, suspicion scores, and cited evidence. Beyond traditional labels, we introduce evidence faithfulness as a core evaluation metric, measuring whether the evidence referenced by a model is explicitly present in the input logs. We evaluate multiple widely used closed weight online LLMs, including ChatGPT, Gemini, Grok, NotebookLM, Perplexity, and Claude. Our results reveal significant differences in suspicion calibration, alert generation, and evidence grounding across models. While several LLMs consistently flag suspicious activity, unfaithful or partially hallucinated evidence remains common. These findings highlight that reliable explanation grounding is essential for trustworthy LLM-assisted intrusion detection and motivate future research on evaluation-driven prompt and model design.

Overview

CoChat Summary

TL;DR

The paper proposes an evaluation framework that treats six closed-weight, web-accessible LLMs as black-box SOC analysts over raw Linux system logs, adding a strict evidence-faithfulness metric that checks whether cited log lines appear verbatim. It finds large differences in suspicion calibration and grounding, with NotebookLM at 0% evidence faithfulness despite high suspicion scores, while ChatGPT, Gemini, and Claude show high grounding.

Key findings

  • Across Table I, mean suspicion scores ranged from 49.50 (ChatGPT) to 93.00 (NotebookLM); Claude had 83.83 and a 66.67% attack rate, while all other models had 0% attack rate.
  • Evidence faithfulness was 100% for ChatGPT and Claude, 90% for Gemini, 80% for Grok and Perplexity, and 0% for NotebookLM due to systematic paraphrasing despite confident judgments.
  • Average alerts per window were 3.00 for Claude and NotebookLM, 2.80 for Gemini, 2.50 for ChatGPT, and 2.00 for Grok and Perplexity.
  • Suspicion scores were model-specific and not directly comparable; strong reasoning did not guarantee forensic traceability, and evidence faithfulness was independent of detection accuracy.

Methods

  • Linux system logs from `/var/log/messages` on a server environment with simulated intrusion and anomalous activities were partitioned into non-overlapping, chronologically ordered windows of exactly 200 consecutive log lines.
  • Six black-box LLMs were evaluated: ChatGPT (GPT-5.2), Grok 4.1, Gemini 3, NotebookLM (Gemini 3), Claude (Opus 4.5; subset of windows due to context limitations), and Perplexity (Sonar), using a standardized SOC-analyst prompt and strict JSON output schema without fine-tuning or retrieval augmentation.
  • Evidence faithfulness was a binary window-level metric requiring all cited evidence lines to appear verbatim in the input log window; other metrics included mean and standard deviation of suspicion scores, alerts per window, intrusion label distribution, and evidence faithfulness rate, with no ground-truth attack labels assumed.

Limitations

  • The paper states limitations including the absence of external ground truth, black-box model treatment, and logs from a single environment; Claude was evaluated on a subset of windows due to context-length limitations and excluded from aggregate comparisons where appropriate.
  • Evident from the text: the small window counts (6 for Claude, 10 for other models) and single-environment logs limit generalizability, and the lack of ground-truth labels means actual detection accuracy cannot be assessed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/cai68641.2026.11536239

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 1 Not valid · 13 Not checked · 18 Own results and interpretations

4 more claims

Create a free account to see all 34 claims and to verify the 7 references still open.

See all claims — free account
  • It may state two separate claims as one. 15
  • Probably not a substantial claim. 14

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 24 references

Research works printed but not linked: 1

  1. Li, IDS-Agent: An LLM agent for explainable intrusion detection in IoT networks

Metadata

Authors
Beyzavi, H., Shiraskar, S., Albluwi, A., Albalawi, M., Rizk, D., Rizk, F., Rizk, P., Nehmetallah, G., Rizk, R., & Santosh, K.
Affiliations
  • IBM (United States) (US)
  • Catholic University of America (US)
  • King Abdulaziz University (SA)
  • The University of Texas at Tyler (US)
  • Arctus Inc. (Canada) (CA)
  • University of South Dakota (US)
Venue
2026 IEEE Conference on Artificial Intelligence (CAI)
Publisher
IEEE
Published
2026-05-08
Pages
1805-1810
OpenAlex
W7163066643
Cited by
0
References
14
FWCI
0.00
Type
conference paper
Language
en
Open access
closed
Locations
IEEE Conference on Artificial Intelligence (CAI) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
large language models, intrusion detection, security operations center, system log analysis, hallucinations, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Can Large Language Models Be Trusted for AI-Based Intrusion Detection in Cybersecurity? An Evidence-Grounded Log Analysis. https://app.cochat.ai/papers/10.1109/cai68641.2026.11536239
The paper
Beyzavi, H., Shiraskar, S., Albluwi, A., Albalawi, M., Rizk, D., Rizk, F., Rizk, P., Nehmetallah, G., Rizk, R., & Santosh, K. (2026). Can Large Language Models Be Trusted for AI-Based Intrusion Detection in Cybersecurity? An Evidence-Grounded Log Analysis. 2026 IEEE Conference on Artificial Intelligence (CAI). https://doi.org/10.1109/cai68641.2026.11536239

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.