Spear Phishing With Large Language Models
Open accessAbstract
Recent progress in artificial intelligence (AI), particularly in the domain of large language models (LLMs), has resulted in powerful and versatile dual-use systems. This intelligence can be put towards a wide variety of beneficial tasks, yet it can also be used to cause harm. This study explores one such harm by examining how LLMs can be used for spear phishing, a form of cybercrime that involves manipulating targets into divulging sensitive information. I first explore LLMs' ability to assist with the reconnaissance and message generation stages of a spear phishing attack, where I find that LLMs are capable of assisting with the email generation phase of a spear phishing attack. To explore how LLMs could potentially be harnessed to scale spear phishing campaigns, I then create unique spear phishing messages for over 600 British Members of Parliament using OpenAI's GPT-3.5 and GPT-4 models. My findings provide some evidence that these messages are not only realistic but also cost-effective, with each email costing only a fraction of a cent to generate. Next, I demonstrate how basic prompt engineering can circumvent safeguards installed in LLMs, highlighting the need for further research into robust interventions that can help prevent models from being misused. To further address these evolving risks, I explore two potential solutions: structured access schemes, such as application programming interfaces, and LLM-based defensive systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For spear-phishing scams (Hazell, 2023), the analyzed study does not examine how open FMs could bypass defenses built into operating systems or email services.
“There is no analysis of how open FMs can be used to circumvent existing defenses built into operating systems or email services.”
From On the Societal Impact of Open Foundation Models (2024) · cites this paper as Hazell, 2023
LLMs can be leveraged to scale up personalized attacks, such as spear phishing content.
“LLMs can also be leveraged to scale up personalized attacks, such as spear phishing content (Hazell, 2023).”
From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Hazell, 2023
Users can acquire knowledge on API queries and LLM prompting directly from the models themselves.
“Users can also acquire knowledge on API queries and LLM prompting directly from the models themselves (Hazell, 2023).”
From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Hazell, 2023
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“In recent years, researchers have trained progressively larger models on greater volumes of data with significantly more computational resources, a process known as "scaling" [26].”
Background
Flagged first, then by citations
Cited by
Sorted by citations
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243 · see citing works
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243 · see citing works
Devising and Detecting Phishing Emails Using Large Language Models
Heiding, F., Schneier, B., Vishwanath, A., Bernstein, J., & Park, P. S. (2024) · IEEE Access · doi:10.1109/access.2024.3375882 · Cited by 123 · see citing works
Large Language Models in Cybersecurity: State-of-the-Art
Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024) · arXiv · doi:10.48550/arxiv.2402.00891 · Cited by 92 · see citing works
“According to Hazell et al. [32], LLMs are able to assist during the reconnaissance stage of a spear phishing attack, a process when attackers get sensitive information about their targets in order to develop compelling messages.”
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84 · see citing works
Anatomy of an AI-powered malicious social botnet
Yang, K., & Menczer, F. (2024) · Journal of Quantitative Description: Digital Media · doi:10.51685/jqd.2024.icwsm.7 · Cited by 71 · see citing works
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.674 · Cited by 48 · see citing works
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
“Future research should focus on refining existing authorship attri-bution methods to detect and prevent malicious activities such as generating misinformation, plagiarism, and propaganda (Goldstein et al., 2023, Hazell, 2023, Lund et al., 2023, Spitale et al., 2023).”
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31 · see citing works
Self-citationMalla: Demystifying Real-world Large Language Model Integrated Malicious Services
Lin, Z., Cui, J., Liao, X., & Wang, X. (2024) · arXiv · doi:10.48550/arxiv.2401.03315 · Cited by 13 · see citing works
“Past research showcased how LLMs can be weaponized across diverse domains, such as misinformation propagation [91,102,125], deepfake user profile creation [109], spear phishing campaigns [97, 98], attack and malware generation [97] and the generation of hateful memes [119].”
Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023) · arXiv · doi:10.48550/arxiv.2311.04378 · Cited by 12 · see citing works
On the Societal Impact of Open Foundation Models
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07918 · Cited by 11 · see citing works
A Lifecycle-Oriented Survey of Emerging Threats and Vulnerabilities in Large Language Models
de Maio, C., Di Gisi, M., Fenza, G., Gallo, M., & Loia, V. (2025) · IEEE Access · doi:10.1109/access.2025.3619764 · Cited by 10 · see citing works
Jones, C. R., & Bergen, B. K. (2024) · arXiv · doi:10.48550/arxiv.2412.17128 · Cited by 7 · see citing works
- Authors
- Hazell, J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Hazell, Julian
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.06972
- OpenAlex
- W4376312605
- Cited by
- 28
- References
- 1
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 21
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Cybercrime and Law Enforcement Studies
- Keywords
- spear phishing, large language models, cybercrime, prompt engineering, LLM misuse, GPT-3.5, GPT-4, Members of Parliament
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
