CoChat

LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks

Happe, Andreas, Kaplan, Aaron, Cito, Jürgen · 2026 · Empirical Software Engineering
Cited by 22

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

Penetration-testing is crucial for identifying and mitigating system vulnerabilities, with privilege-escalation being a critical subtask involving gaining elevated access to protected resources. The emergence of Large Language Models (LLMs) presents new avenues for automating these security practices by emulating human behavior. However, a comprehensive understanding of LLMs’ efficacy and limitations in performing autonomous Linux privilege-escalation attacks remains underexplored. To address this gap, we introduce hackingBuddyGPT , a fully automated LLM-driven prototype designed for evaluating autonomous Linux privilege-escalation. We curated a novel, publicly available Linux privilege-escalation benchmark comprising distinct, single-vulnerability virtual machines, enabling controlled and reproducible evaluation. Our empirical analysis assesses the quantitative success rates and qualitative operational behaviors of various LLMs— GPT-3.5-Turbo , GPT-4-Turbo , and Llama3 —against baselines of human professional penetration-testers and traditional automated tools. We investigate the impact of context management strategies, different context sizes, and various high-level guidance mechanisms on LLM performance. Results show that GPT-4-Turbo demonstrates high efficacy, successfully exploiting 33–83% of vulnerabilities, a performance comparable to human penetration testers (75%). In contrast, local models like Llama3 exhibited limited success (0–33%), and GPT-3.5-Turbo achieved moderate rates (16–50%). High-level guidance significantly boosts LLM success rates , for instance when using GPT-4-Turbo from 33% to 66% (without guidance) or from 66% to 83%, while state management through LLM-driven reflection doubled unaided GPT-4-Turbo success rates (from 33% to 66%). Qualitative analysis reveals both LLMs’ strengths and weaknesses in generating valid commands and highlights challenges in common-sense reasoning, error handling, and multi-step exploitation, particularly with temporal dependencies. Cost analysis indicates that GPT-4-Turbo can achieve human-comparable performance at competitive costs per exploited vulnerability, especially with optimized context management. Our work provides a baseline for evaluating LLM capabilities in autonomous privilege escalation, guiding future research toward more effective and reliable LLM-guided penetration-testing.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10664-025-10758-3

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 36

  1. Bishop M (2007) About penetration testing. IEEE Secur Priv 5(6):84–87. https://doi.org/10.1109/MSP.2007.159 · DOI
  2. Boukhlif M, Kharmoum N, Hanine M (2024) Llms for intelligent software testing: a comparative study. In: Proceedings of the 7th international conference on networking, intelligent systems and security, association for computing machinery, New York, NY, USA, NISS ’24. https://doi.org/10.1145/3659677.3659749 · DOI
  3. Dagan G, Keller F, Lascarides A (2023) Dynamic planning with a llm. arXiv:2308.06391 · arXiv
  4. Ding Y, Zhang LL, Zhang C, Xu Y, Shang N, Xu J, Yang F, Yang M (2024) Longrope: extending llm context window beyond 2 million tokens. arXiv:2402.13753 · arXiv
  5. Dube R (2024) Large language models in information security research: a january 2024 survey. ResearchGate preprint RG 2(20107.26404)
  6. Fang R, Bindu R, Gupta A, Zhan Q, Kang D (2024c) Teams of llm agents can exploit zero-day vulnerabilities. https://arxiv.org/abs/2406.01637, arXiv:2406.01637 · arXiv
  7. Geer D, Harthorne J (2002) Penetration testing: a duet. In: 18th annual computer security applications conference, 2002. Proceedings, pp 185–195. https://doi.org/10.1109/CSAC.2002.1176290 · DOI
  8. Gupta M, Akiri C, Aryal K, Parker E, Praharaj L (2023) From chatgpt to threatgpt: impact of generative ai in cybersecurity and privacy. IEEE Access · DOI · Paper page
  9. Halawi D, Wei A, Wallace E, Wang TT, Haghtalab N, Steinhardt J (2024) Covert malicious finetuning: challenges in safeguarding llm adaptation. https://arxiv.org/abs/2406.20053, arXiv:2406.20053 · arXiv
  10. Happe A, Cito J (2023b) Understanding hackers’ work: An empirical study of offensive security practitioners. In: Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering. Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2023 https://doi.org/10.1145/3611643.3613900 · DOI
  11. Happe A, Cito J (2025) Benchmarking practices in llm-driven offensive security: testbeds, metrics, and experiment design. https://arxiv.org/abs/2504.10112, arXiv:2504.10112 · arXiv
  12. Harang R, Ducau FN (2018) Measuring the speed of the red queen’s race. Las Vegas, NV, USA, BlackHat
  13. Hassanin M, Moustafa N (2024) A comprehensive overview of large language models (llms) for cyber defences: opportunities and directions. https://arxiv.org/abs/2405.14487, arXiv:2405.14487 · arXiv
  14. He X, Yang D, Feng W, Fu TJ, Akula A, Jampani V, Narayana P, Basu S, Wang WY, Wang XE (2022) Cpl: counterfactual prompt learning for vision and language models. arXiv:2210.10362 · DOI
  15. Huang Y, Gupta S, Xia M, Li K, Chen D (2023) Catastrophic jailbreak of open-source llms via exploiting generation. arXiv:2310.06987 · arXiv
  16. Huang W, Ma X, Qin H, Zheng X, Lv C, Chen H, Luo J, Qi X, Liu X, Magno M (2024) How good are low-bit quantized llama3 models? an empirical study. arXiv:2404.14047 · arXiv
  17. Huang J, Zhu Q (2024) Penheal: a two-stage llm framework for automated pentesting and optimal remediation. arXiv:2407.17788 · arXiv · doi:10.2139/ssrn.4941478
  18. Jin H, Huang L, Cai H, Yan J, Li B, Chen H (2024) From llms to llm-based agents for software engineering: a survey of current, challenges and future. https://arxiv.org/abs/2408.02479, arXiv:2408.02479 · arXiv
  19. Jin Y, Jang E, Cui J, Chung JW, Lee Y, Shin S (2023) Darkbert: a language model for the dark side of the internet. arXiv:2305.08596 · DOI
  20. Kong A, Zhao S, Chen H, Li Q, Qin Y, Sun R, Zhou X, Wang E, Dong X (2023) Better zero-shot reasoning with role-play prompting. arXiv:2308.07702 · DOI · Paper page
  21. Kosinski M (2023) Theory of mind might have spontaneously emerged in large language models. arXiv:2302.02083 · arXiv
  22. Kowira EM, Suki NN, Nathan Y (2024) Automated privilege escalation enumeration and execution script for linux. In: AIP Conference Proceedings. AIP Publishing, vol 2802 · doi:10.1063/5.0183425
  23. Lee J, Chen A, Dai Z, Dua D, Sachan DS, Boratko M, Luan Y, Arnold SMR, Perot V, Dalmia S, Hu H, Lin X, Pasupat P, Amini A, Cole JR, Riedel S, Naim I, Chang MW, Guu K (2024) Can long-context language models subsume retrieval, rag, sql, and more? https://arxiv.org/abs/2406.13121, arXiv:2406.13121 · arXiv
  24. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Wt Y, Rocktäschel T et al (2020b) Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Syst 33:9459–9474
  25. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih Wt, Rocktäschel T, Riedel S, Kiela D (2020a) Retrieval-augmented generation for knowledge-intensive nlp tasks. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Advances in neural information processing systems. Curran Associates, Inc., vol 33, pp 9459–9474, https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
  26. Li X, Cao Y, Ma Y, Sun A (2024) Long context vs. rag for llms: an evaluation and revisits. https://arxiv.org/abs/2501.01880, arXiv:2501.01880 · arXiv
  27. Liu Y, Deng G, Li Y, Wang K, Zhang T, Liu Y, Wang H, Zheng Y, Liu Y (2023) Prompt injection attack against llm-integrated applications. arXiv:2306.05499 · arXiv
  28. Mavikumbure HS, Cobilean V, Wickramasinghe CS, Drake D, Manic M (2024) Generative ai in cyber security of cyber physical systems: benefits and threats. In: 2024 16th International Conference on Human System Interaction (HSI), pp 1–8. https://doi.org/10.1109/HSI61632.2024.10613562 · DOI
  29. Shah S, Mehtre BM (2015) An overview of vulnerability assessment and penetration testing techniques. J Comput Virol Hacking Tech 11:27–49 · DOI
  30. Shebli HMZA, Beheshti BD (2018) A study on penetration testing process and tools. In: 2018 IEEE Long Island Systems, Applications and Technology Conference (LISAT), pp 1–7. https://doi.org/10.1109/LISAT.2018.8378035 · doi:10.1109/lisat.2018.8378035
  31. Sommer R, Paxson V (2010) Outside the closed world: on using machine learning for network intrusion detection. In: 2010 IEEE symposium on security and privacy. IEEE, pp 305–316 · DOI · Paper page
  32. Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S et al (2023) Llama 2: open foundation and fine-tuned chat models. arXiv:2307.09288 · arXiv
  33. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30
  34. Weidman G (2014) Penetration testing: a hands-on introduction to hacking. No starch press
  35. Xu J, Stokes JW, McDonald G, Bai X, Marshall D, Wang S, Swaminathan A, Li Z (2024b) Autoattacker: a large language model guided system to implement automatic cyber-attacks. arXiv:2403.01038 · arXiv
  36. Zhang J, Bu H, Wen H, Chen Y, Li L, Zhu H (2024) When llms meet cybersecurity: a systematic literature review. https://arxiv.org/abs/2405.03644, arXiv:2405.03644 · arXiv

Cited by

See all 22

Metadata

Authors
Happe, A., Kaplan, A., & Cito, J.
Venue
Empirical Software Engineering
Published
2026-02-10
Volume
31
Issue
3
ISSN
  • 1382-3256
  • 1573-7616
Cited by
22
References
25
Language
en
Licence
Funders
TU Wien

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks. https://app.cochat.ai/papers/10.1007/s10664-025-10758-3
The paper
Happe, A., Kaplan, A., & Cito, J. (2026). LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks. Empirical Software Engineering. https://doi.org/10.1007/s10664-025-10758-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.