CoChat
Sign Up
PreprintCited by 2Adversarial Robustness in Machine LearningSpam and Phishing DetectionInformation and Cyber Security

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Is Generative AI the Next Tactical Cyber Weapon For Threat Actors? Unforeseen Implications of AI Generated Cyber Attacks

Open access
Usman, Yusuf, Upadhyay, Aadesh, Gyawali, Prashnna, Chataut, Robin·2024·arXiv
DOI 10.48550/arxiv.2408.12806arXiv 2408.12806
Open this paper in CoChat — free accountOpen PDF

Abstract

In an era where digital threats are increasingly sophisticated, the intersection of Artificial Intelligence and cybersecurity presents both promising defenses and potent dangers. This paper delves into the escalating threat posed by the misuse of AI, specifically through the use of Large Language Models (LLMs). This study details various techniques like the switch method and character play method, which can be exploited by cybercriminals to generate and automate cyber attacks. Through a series of controlled experiments, the paper demonstrates how these models can be manipulated to bypass ethical and privacy safeguards to effectively generate cyber attacks such as social engineering, malicious code, payload generation, and spyware. By testing these AI generated attacks on live systems, the study assesses their effectiveness and the vulnerabilities they exploit, offering a practical perspective on the risks AI poses to critical infrastructure. We also introduce Occupy AI, a customized, finetuned LLM specifically engineered to automate and execute cyberattacks. This specialized AI driven tool is adept at crafting steps and generating executable code for a variety of cyber threats, including phishing, malware injection, and system exploitation. The results underscore the urgency for ethical AI practices, robust cybersecurity measures, and regulatory oversight to mitigate AI related threats. This paper aims to elevate awareness within the cybersecurity community about the evolving digital threat landscape, advocating for proactive defense strategies and responsible AI development to protect against emerging cyber threats.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.12806

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Regulating ChatGPT and other Large Generative AI Models

    Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · cited 1×

    “Regarding the specific case of ChatGPT, researchers [19] [20] have outlined five key priorities for its role in research: focusing on human verification, developing rules for accountability, investing in truly open LLMs, embracing the benefits of AI, and widening the debate on LLMs. Open questions…”
    Background

Show all 1 references

Research works printed but not linked: 35

  1. M. Gupta, C. Akiri, K. Aryal, E. Parker and L. Praharaj, "From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy," in IEEE Access, vol. 11, pp. 80218-80245, 2023, doi: 10.1109/ACCESS.2023.3300381. · DOI
  2. K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image recognition", Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 770-778, Jun. 2016.
  3. H. Liu, D. Xu, Y. Tian, C. Peng, Z. Wu, and Z. Wang, "Wasserstein Generative Adversarial Networks Based Differential Privacy Metaverse Data Sharing," in IEEE Journal of Biomedical and Health Informatics, doi: 10.1109/JBHI.2023.3287092. · DOI
  4. X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, "Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models," presented at the ACM SIGSAC Conference on Computer and Communications Security (CCS), 2024.
  5. L. Weidinger, J. Uesato, M. Rauh, C. Griffin, P.-S. Huang, J. Mellor, A. Glaese, M. Cheng, B. Balle, A. Kasirzadeh, et al., "Taxonomy of Risks Posed by Language Models," in 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 214-229.
  6. B. D. Lund and T. Wang, "Chatting about ChatGPT: How may AI and GPT impact academia and libraries?," Library Hi Tech News, vol. 40, no. 3, pp. 26-29, 2023, doi: 10.1108/LHTN-01-2023-0009. · DOI · Paper page
  7. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention Is All You Need," in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998-6008.
  8. M. Tamkin, J. Brundage, J. Clark, and D. Ganguli, "Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models," arXiv preprint arXiv:2102.02503, 2021. · arXiv
  9. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., "On the Opportunities and Risks of Foundation Models," arXiv preprint arXiv:2108.07258, 2021. · arXiv
  10. J. Deng, H. Sun, Z. Zhang, J. Cheng, and M. Huang, "Recent Advances Towards Safe, Responsible, and Moral Dialogue Systems: A Survey," arXiv preprint arXiv:2302.09270, 2023. · arXiv
  11. H. Devinney, J. Björklund, and H. Björklund, "Theories of "Gender" in NLP Bias Research," in 2022 ACM Conference on Fairness, Account- ability, and Transparency, 2022, pp. 2083-2102.
  12. W. I. Cho, J. Kim, J. Yang, and N. S. Kim, "Towards Cross-Lingual Generalization of Translation Gender Bias," in Proceedings of the 2021 ACM Conference on Fairness, Accountability and Transparency, 2021, pp. 449-457.
  13. H. R. Kirk, Y. Jun, F. Volpin, H. Iqbal, E. Benussi, F. Dreyer, A. Shtedritski, and Y. Asano, "Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models," Advances in Neural Information Processing Systems, vol. 34, pp. 2611-2624, 2021.
  14. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein, "Detoxifying Language Models Risks Marginalizing Minority Voices," in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2390-2397.
  15. S. Lin, J. Hilton, and O. Evans, "TruthfulQA: Measuring How Models Mimic Human Falsehoods," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 3214-3252
  16. W. Cai, R. Encarnacion, B. Chern, S. Corbett-Davies, M. Bogen, S. Bergman, and S. Goel, "Adaptive Sampling Strategies to Construct Equitable Training Datasets," in 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 1467-1478.
  17. H. Ngo, C. Raterink, J. G. Araújo, I. Zhang, C. Chen, A. Morisot, and N. Frosst, "Mitigating Harm in Language Models with ConditionalLike- lihood Filtration," arXiv preprint arXiv:2108.07790, 2021. · arXiv
  18. E. A. van Dis, J. Bollen, W. Zuidema, R. van Rooij, and C. L. Bockting, "ChatGPT: Five Priorities for Research," Nature, vol. 614, no. 7947, pp. 224-226, 2023.
  19. T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing, "Exploring AI Ethics of ChatGPT: A Diagnostic Analysis," arXiv preprint arXiv:2301.12867, 2023. · arXiv
  20. J. Rudolph, S. Tan, and S. Tan, "ChatGPT: Bullshit Spewer or the End of Traditional Assessments in Higher Education?" Journal of Applied Learning and Teaching, vol. 6, no. 1, 2023.
  21. S. Hisamoto, M. Post, and K. Duh, "Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data in Your Machine Translation System?" Trans. Assoc. Comput. Linguist., vol. 8, pp. 49-63, 2020.
  22. H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang, "Membership Inference Attacks on Machine Learning: A Survey," ACM Computing Surveys (CSUR), vol. 54, no. 11s, pp. 1-37, 2022.
  23. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, "Extracting Training Data from Large Language Models," in 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, Aug 2021, pp. 2633-2650.
  24. X. Pan, M. Zhang, S. Ji, and M. Yang, "Privacy Risks of GeneralPurpose Language Models," in 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 2020, pp. 1314-1331.
  25. H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr, "What Does it Mean for a Language Model to Preserve Privacy?" in 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 2280-2292.
  26. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., "Evaluating Large Language Models Trained on Code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  27. H. Hajipour, T. Holz, L. Schönherr, and M. Fritz, "Systematically Find- ing Security Vulnerabilities in Black-Box Code Generation Models," arXiv preprint arXiv:2302.04012, 2023. · arXiv
  28. R. Karanjai, "Targeted Phishing Campaigns using Large Scale Language Models," arXiv pre 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) Published: 2020
  29. V. P. S., "How can we manage biases in artificial intelligence systems -A systematic literature review," Int. J. Inf. Manag. Data Insights, vol. 3, pp. 1-9, 2023, doi: 10.1016/j.jjimei.2023.100165. · DOI · Paper page
  30. M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, "A Survey of Machine Learning for Big Code and Naturalness," ACM Comput. Surv., vol. 51, no. 4, Art. 81, 37 pages, Jul. 2019, doi: 10.1145/3212695. · DOI · Paper page
  31. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, and others, "Language models are few-shot learners," in Advances in Neural Information Processing Systems, vol. 33, 2020.
  32. F. Okeke, "An Assessment of the Use of Generative AI in Cyberse- curity: Challenges and Opportunities," ResearchGate, Dec. 2023. doi: 10.13140/RG.2.2.20613.12001. · DOI
  33. P. Dhoni and R. Kumar, "Synergizing Generative AI and Cyberse- curity: Roles of Generative AI Entities, Companies, Agencies, and Government in Enhancing Cybersecurity," TechRxiv, 2023. [Online]. Available: http://dx.doi.org/10.36227/techrxiv.23968809.v1. [Accessed: Oct. 11, 2023]. · DOI
  34. R. Chataut, P. K. Gyawali and Y. Usman, "Can AI Keep You Safe? A Study of Large Language Models for Phishing Detection," 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, 2024, pp. 0548-0554, doi: 10.1109/CCWC60891.2024.10427626. · DOI
  35. B. Guembe, A. Azeta, S. Misra, V. Osamor, L. Fernandez-Sanz, and V. Pospelova, "The Emerging Threat of AI-driven Cyber Attacks: A Review," Applied Artificial Intelligence, vol. 36, no. 1, pp. 1-34, 2022.

Cited by

Sorted by citations

See all 2

Metadata

Authors
Usman, Y., Upadhyay, A., Gyawali, P., & Chataut, R.
Affiliations
  • Quinnipiac University (US)
  • University of North Texas (US)
  • West Virginia University (US)
  • Texas Christian University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2408.12806
arXiv
2408.12806
OpenAlex
W4402699012
Cited by
2
References
1
Citations by year
2026: 1 · 2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
Keywords
generative AI, large language models, social engineering, phishing, malware generation, spyware, critical infrastructure security, AI ethics, AI regulation, LLM fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Is Generative AI the Next Tactical Cyber Weapon For Threat Actors? Unforeseen Implications of AI Generated Cyber Attacks. https://app.cochat.ai/papers/10.48550/arxiv.2408.12806
The paper
Usman, Y., Upadhyay, A., Gyawali, P., & Chataut, R. (2024). Is Generative AI the Next Tactical Cyber Weapon For Threat Actors? Unforeseen Implications of AI Generated Cyber Attacks. arXiv. https://doi.org/10.48550/arxiv.2408.12806