The DOI matches its Crossref record; its reference list is not available.
Honeypot's Best Friend? Investigating ChatGPT's Ability to Evaluate Honeypot Logs
Open accessAbstract
Honeypots can gather substantial data from intruders, but many honeypots lack the necessary features to analyse and explain the nature of these potential attacks. Typically, honeypot analysis reports only highlight the attacking IP addresses and the malicious requests. As such, analysts might miss out on the more useful insights that can be derived from the honeypot data, such as the attackers’ plan or emerging threats. Meanwhile, recent advances in large language models (LLM) – such as ChatGPT – have opened up the possibility of using artificial intelligence (AI) to comprehend honeypot data better, for instance, to perform an automated and intelligent log analysis that can explain consequences, provide labels, and deal with obfuscation. In this study, we probed ChatGPT’s proficiency in understanding and explaining honeypot logs from actual recorded attacks on our honeypots. Our data encompassed 627 requests to Elasticsearch honeypots and 73 attacks detected by SSH honeypots, collected over a two-week period. Our analysis was focused on evaluating ChatGPT’s explanation ability regarding the potential consequences of each attack, in alignment with the MITRE ATT&CK Framework, and whether ChatGPT can identify any obfuscation techniques that might be used by attackers. We found that ChatGPT achieved a 96.65% accuracy in correctly explaining the consequences of the attack targeting Elasticsearch servers. Furthermore, ChatGPT achieved a 72.46% accuracy in matching a given attack to…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs are known to sometimes produce convincing yet ungrounded statements or hallucinated details [2], [17].”
From Can Large Language Models Be Trusted for AI-Based Intrusion Detection in Cybersecurity? An Evidence-Grounded Log Analysis (2026) · cites this paper as [2]
In the citing paper
“Recent NLP studies have emphasized measuring how well model outputs stay supported by provided data (i.e., faithfulness) [2], [5], but this has not been systematically applied to intrusion detection scenarios.”
From Can Large Language Models Be Trusted for AI-Based Intrusion Detection in Cybersecurity? An Evidence-Grounded Log Analysis (2026) · cites this paper as [2]
In the citing paper
“However, existing studies largely evaluate LLMs based on detection performance or qualitative usefulness, without rigorously assessing whether their explanations are grounded in actual log evidence.”
From Can Large Language Models Be Trusted for AI-Based Intrusion Detection in Cybersecurity? An Evidence-Grounded Log Analysis (2026) · cites this paper as [2]
Citations
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 8 · Cited by 8
Research works printed but not linked: 19
- 10.5120/ijca2023922624 · DOI
- 10.1145/3588195.3595943 · DOI
- Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. 2023. From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy. IEEE Access (2023).
- Yekta Kocaogullar, Orcun Cetin, Budi Arief, Calvin Brierley, Jamie Pont, and Julio C Hernandez-Castro. 2022. Hunting High or Low: Evaluating the Effectiveness of High-Interaction and Low-Interaction Honeypots. In 12th Int’l Workshop on Socio-Technical Aspects in Security (STAST 2022).
- Van-Hoang Le and Hongyu Zhang. 2023. An Evaluation of Log Parsing with ChatGPT. arXiv preprint arXiv:2306.01590 (2023). · arXiv
- 10.1109/ASE56229.2023.00206 · DOI
- Jinyang Liu, Junjie Huang, Yintong Huo, Zhihan Jiang, Jiazhen Gu, Zhuangbin Chen, Cong Feng, Minzhi Yan, and Michael R Lyu. 2023. Scalable and Adaptive Log-based Anomaly Detection with Expert in the Loop. arXiv preprint arXiv:2306.05032 (2023). · arXiv
- Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yanqing Zhao, Yuhang Chen, Hao Yang, Yanfei Jiang, and Xun Chen. 2023. LogPrompt: Prompt Engineering Towards Zero-Shot and Interpretable Log Analysis. arXiv preprint arXiv:2308.07610 (2023). · arXiv
- Forrest McKee and David Noever. 2023. Chatbots in a Honeypot World. arXiv preprint arXiv:2301.03771 (2023). · arXiv
- 10.1145/1233341.1233399 · DOI
- 10.1145/3590777.3590780 · DOI
- Nenad Petrović. 2023. Machine Learning-Based Run-Time DevSecOps: ChatGPT Against Traditional Approach. In 10th Int’l Conf. on Electrical, Electronic and Computing Engineering (IcETRAN). IEEE, 1–5.
- Attia Qammar, Hongmei Wang, Jianguo Ding, Abdenacer Naouri, Mahmoud Daneshmand, and Huansheng Ning. 2023. Chatbots to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabilities, Attacks, Challenges, and Future Recommendations. arXiv preprint arXiv:2306.09255 (2023). · arXiv
- Jiaxing Qi, Shaohan Huang, Zhongzhi Luan, Carol Fung, Hailong Yang, and Depei Qian. 2023. LogGPT: Exploring ChatGPT for Log-Based Anomaly Detection. arXiv preprint arXiv:2309.01189 (2023). · arXiv
- 10.1007/s42979-021-00557-0 · DOI
- Mark Scanlon, Frank Breitinger, Christopher Hargreaves, Jan-Niclas Hilgert, and John Sheppard. 2023. ChatGPT for Digital Forensic Investigation: The Good, The Bad, and The Unknown. arXiv preprint arXiv:2307.10195 (2023). · arXiv
- Mark Scanlon, Bruce Nikkel, and Zeno Geradts. 2023. Digital forensic investigation in the age of ChatGPT. Forensic Sci. Int’l: Digital Investigation 44 (2023).
- 10.1145/3487351.3492723 · DOI
- Pawankumar Sharma and Bibhu Dash. 2023. Impact of Big Data Analytics and ChatGPT on Cybersecurity. In 2023 4th Int’l Conf. on Computing and Communication Systems (I3CS). IEEE, 1–6.
Metadata
- Authors
- Ozkok, M. B., Birinci, B., Cetin, O., Arief, B., & Hernandez-Castro, J.
- Affiliations
- Sabancı Üniversitesi (TR)
- Sabanci University, Turkey
- University of Kent (GB)
- Universidad Politécnica de Madrid (ES)
- Venue
- European Interdisciplinary Cybersecurity Conference
- Publisher
- ACM
- Published
- 2024-06-05
- Pages
- 128-135
- OpenAlex
- W4399337393
- Cited by
- 8
- References
- 8
- Citations by year
- 2026: 7 · 2025: 1
- FWCI
- 3.50
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- European Interdisciplinary Cybersecurity Conference (EICC) (publishedVersion, cc-by)
- Kent Academic Repository (University of Kent) (acceptedVersion)
- Sabanci University (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- ChatGPT, large language models, automated log analysis, MITRE ATT&CK, obfuscation detection, deobfuscation, SSH honeypot, false positives, attack attribution, threat intelligence, artificial intelligence in cybersecurity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Honeypot's Best Friend? Investigating ChatGPT's Ability to Evaluate Honeypot Logs. https://app.cochat.ai/papers/10.1145/3655693.3655716
- The paper
- Ozkok, M. B., Birinci, B., Cetin, O., Arief, B., & Hernandez-Castro, J. (2024). Honeypot's Best Friend? Investigating ChatGPT's Ability to Evaluate Honeypot Logs. European Interdisciplinary Cybersecurity Conference. https://doi.org/10.1145/3655693.3655716
