Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Open accessAbstract
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capabilities amplify the dual-use risks for malicious purposes of these models. Dual-use is difficult to prevent as instruction-following capabilities now enable standard attacks from computer security. The capabilities of these instruction-following LLMs provide strong economic incentives for dual-use by malicious actors. In particular, we show that instruction-following LLMs can produce targeted malicious content, including hate speech and scams, bypassing in-the-wild defenses implemented by LLM API vendors. Our analysis shows that this content can be generated economically and at cost of $125-500 \times$ cheaper than human effort alone. Together, our findings suggest that LLMs will increasingly attract more sophisticated adversaries and attacks, and addressing these attacks may require new approaches to mitigations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Potential consequences of a successful prompt injection attack include exfiltrating user data, executing arbitrary code, and more.
“Potential consequences include exfiltrating user data, executing arbitrary code, and more [18, 23, 33, 42].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [23]
Sections 3.3.2 and 3.4 show attack prompts soliciting credit-card or Social Security information, supporting a potential risk of obtaining sensitive data. They do not show a prompt injection exfiltrating user data. Section 3.3.3 discusses arbitrary execution in a separate iPhone hack, not as a demonstrated consequence of the paper’s LLM attacks. (LLM, conf=0.78)
Experimental — accuracy not yet measuredEven without actively specifying any social engineering techniques, the model's output could mirror some standard scam persuasion techniques.
“Even without actively specifying any social engineering techniques, the model's output could mirror some standard scam persuasion techniques [20].”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [20]
Section 5.1 reports that LLMs generated convincing scam emails, and Appendix C gives an example using limited availability and reassurance—recognizable scam persuasion techniques. The copy does not establish that these techniques appeared without being requested; a prompt shown in Section 3.4 explicitly asks for urgency. (LLM, conf=0.82)
Experimental — accuracy not yet measuredLLMs also excel at phishing attacks.
“However, LLMs also excel at phishing attacks 29 .”
From Human innovation and the creative agency of the world in the age of generative AI (2024) · cites this paper as [29]
When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target.
“When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target [21,43] (that was either given in the prompt or, importantly, the model acquired during the chat session).”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [21]
Previous work has shown that LLMs can produce convincing scams such as phishing emails.
“Previous work has shown that LLMs can produce convincing scams such as phishing emails [21].”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [21]
Different social engineering and persuasion tricks can be automated with LLMs.
“Different social engineering and persuasion tricks can be automated with LLMs [20] (e.g., claiming authority as an official service, claiming repercussions for not complying such as a loss of the service, distracting the user by implying the requested action is urgent, offering a limited-time kind gesture, etc.).”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [20]
It remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding.
“Even if applied, it remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding [21], which future models might further enable.”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [21]
A successful prompt injection attack can allow an external attacker to take actions (and call tools) on behalf of the user.
“A successful attack can allow an external attacker to take actions (and call tools) on behalf of the user.”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [23]
Sections 3.3.2 and 4 discuss prompts that bypass content filters and cause LLMs to generate malicious text. I searched for prompt injection through external content, tool calls, and actions taken on a user's behalf, but found no passage addressing those capabilities in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“Our work furthers the study of harms and risks of LLMs (Bender et al., 2021; Bommasani et al., 2021; Liang et al., 2022; Abid et al., 2021; Gehman et al., 2020; Ganguli et al., 2022; Weidinger et al., 2021; 2022), focusing on demonstrating their potential of being actively misused.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Malware Obfuscation Techniques: A Brief Survey
You, I., & Yim, K. (2010) · 2010 International Conference on Broadband, Wireless Computing, Communication and Applications · doi:10.1109/bwcca.2010.85 · Cited by 607 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569 · see citing works
A Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500 · see citing works
Roemer, R., Buchanan, E., Shacham, H., & Savage, S. (2012) · ACM Transactions on Information and System Security · doi:10.1145/2133375.2133377 · Cited by 499 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 2× · see citing works
“We chose OpenAI as their models are accessible via an API, have state-of-the-art performance (Liang et al., 2022), and are claimed to have state-of-the-art defenses against malicious use cases (Markov et al., 2022).”
BackgroundMethodEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · see citing works
Persistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1× · see citing works
“This improvement is driven in large part by scale and the ability to be instruction following (Mishra et al., 2021; Sanh et al., 2021; Wei et al., 2021; Ouyang et al., 2022; Glaese et al., 2022; Bai et al., 2022; Wang et al., 2022, inter alia ).”
BackgroundInfluentialA Survey of Botnet and Botnet Detection
Feily, M., Shahrestani, A., & Ramadass, S. (2009) · International Conference on Emerging Security Information, Systems and Technologies · doi:10.1109/securware.2009.48 · Cited by 384 · see citing works
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · see citing works
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · see citing works
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2021) · arXiv · doi:10.48550/arxiv.2104.08773 · Cited by 269 · see citing works
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Self-citationCode obfuscation techniques for metamorphic viruses
Borello, J.-M., & Mé, L. (2008) · Journal of Computer Virology and Hacking Techniques · doi:10.1007/s11416-008-0084-2 · Cited by 213 · see citing works
Comparative analysis of various ransomware virii
Gazet, A. (2008) · Journal of Computer Virology and Hacking Techniques · doi:10.1007/s11416-008-0092-2 · Cited by 211 · see citing works
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · cited 1× · see citing works
“Our work furthers the study of harms and risks of LLMs (Bender et al., 2021; Bommasani et al., 2021; Liang et al., 2022; Abid et al., 2021; Gehman et al., 2020; Ganguli et al., 2022; Weidinger et al., 2021; 2022), focusing on demonstrating their potential of being actively misused.”
BackgroundThe Global Call Center Report: International Perspectives on Management and Employment
Holman, D. J., Batt, R., & Holtgrewe, U. (2007) · eCommons (Cornell University) · Cited by 180 · see citing works
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T. T. H., Adler, S., Jiang, A., & Weng, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26752 · Cited by 153 · see citing works
Toxicity Detection: Does Context Really Matter?
Pavlopoulos, J., Sorensen, J. S., Dixon, L., Thain, N., & Androutsopoulos, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.396 · Cited by 145 · see citing works
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · see citing works
Improving alignment of dialogue agents via targeted human judgements
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14375 · Cited by 134 · see citing works
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A., & Weng, L. (2022) · arXiv · doi:10.48550/arxiv.2208.03274 · Cited by 129 · see citing works
AVOIDIT: A Cyber Attack Taxonomy
Simmons, C., Ellis, C. D., Shiva, S. G., Dasgupta, D., & Wu, Q. (2009) · Cited by 124 · see citing works
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · see citing works
Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTPreprint
Sun, L., Hashimoto, K., Yin, W., Asai, A., Li, J., Yu, P. S., & Xiong, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04985 · Cited by 92 · see citing works
Robust Encodings: A Framework for Combating Adversarial Typos
Jones, E., Jia, R., Raghunathan, A., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.245 · Cited by 89 · see citing works
The Radicalization Risks of GPT-3 and Advanced Neural Language Models
McGuffie, K., & Newhouse, A. (2020) · arXiv · doi:10.48550/arxiv.2009.06807 · Cited by 86 · see citing works
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.12017 · Cited by 81 · see citing works
Self-citationTruth, Lies, and Automation: How Language Models Could Change Disinformation
Center for Security and Emerging Technology, Buchanan, B., Lohn, A., Musser, M., & Sedova, K. (2021) · doi:10.51593/2021ca003 · Cited by 74 · see citing works
A First Look at the Crypto-Mining Malware Ecosystem
Pastrana, S., & Suárez‐Tangil, G. (2019) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3355369.3355576 · Cited by 68 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksPreprint
Wang, Y., Mishra, S. R., Alipoormolabashi, P., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07705 · Cited by 46 · see citing works
Applied Comparative Evaluation of the Metasploit Evasion Module
Casey, P., Topor, M., Hennessy, E., Alrabaee, S., Aloqaily, M., & Boukerche, A. (2019) · 2019 IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc47284.2019.8969663 · Cited by 16 · see citing works
LIME: Learning Inductive Bias for Primitives of Mathematical ReasoningPreprint
Wu, Y., Rabe, M. N., Li, W., Ba, J. L., Grosse, R., & Szegedy, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06223 · Cited by 14 · see citing works
TempLM: Distilling Language Models into Template-Based Generators
Zhang, T., Lee, M., Li, X. L., Shen, E., & Hashimoto, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.124 · Cited by 5 · see citing works
Self-citationKoret, J., & Bachaalany, E. (2015) · doi:10.1002/9781119183525.ch6 · Cited by 1 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462
Robust Encodings: A Framework for Combating Adversarial Typos
Jones, E., Jia, R., Raghunathan, A., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.01229
Economic Analysis of Ransomware
Hernandez-Castro, J., Cartwright, E., & Stepanova, A. (2017) · arXiv · doi:10.48550/arxiv.1703.06660
Toxicity Detection: Does Context Really Matter?
Pavlopoulos, J., Sorensen, J., Dixon, L., Thain, N., & Androutsopoulos, I. (2020) · arXiv · doi:10.48550/arxiv.2006.00998
TempLM: Distilling Language Models into Template-Based Generators
Zhang, T., Lee, M., Li, L., Shen, E., & Hashimoto, T. B. (2022) · arXiv · doi:10.48550/arxiv.2205.11055
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667 · see citing works
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445 · see citing works
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441 · see citing works
Multi-step Jailbreaking Privacy Attacks on ChatGPT
Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., & Song, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.272 · Cited by 269 · see citing works
Huang, X., Ruan, W., Huang, W., Jin, G., Dong, Y., Wu, C., Bensalem, S., Mu, R., Yi, Q., Zhao, X., et al. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10824-0 · Cited by 181 · see citing works
Adversarial machine learning :
Vassilev, A., Oprea, A., Fordyce, A. J., & Anderson, H. S. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 194 · see citing works
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.40 · Cited by 130 · see citing works
Safeguarding large language models: a survey
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11389-2 · Cited by 161 · see citing works
The Security of Using Large Language Models: A Survey with Emphasis on ChatGPT
Zhou, W., Zhu, X., Han, Q., Li, L., Chen, X., Wen, S., & Xiang, Y. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124983 · Cited by 101 · see citing works
Mustapha, K. B. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103066 · Cited by 96 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · see citing works
Hate Speech Detection Using Large Language Models: A Comprehensive Review
Albladi, A., Islam, M., Das, A., Bigonah, M., Zhang, Z., Jamshidi, F., Rahgouy, M., Raychawdhary, N., Marghitu, D., & Seals, C. (2025) · IEEE Access · doi:10.1109/access.2025.3532397 · Cited by 72 · see citing works
(2024) · doi:10.1109/sp54263.2024.00182 · Cited by 62 · see citing works
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57 · see citing works
Self-citationLarge Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2023) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 74 · see citing works
Vulnerability Detection and Monitoring Using LLM
Akuthota, V., Kasula, R., Sumona, S. T., Mohiuddin, M., Reza, M. T., & Rahman, M. M. (2023) · IEEE International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE) · doi:10.1109/wiecon-ece60392.2023.10456393 · Cited by 46 · see citing works
A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models
Derner, E., Batistič, K., Zahálka, J., & Babuška, R. (2024) · IEEE Access · doi:10.1109/access.2024.3450388 · Cited by 58 · see citing works
Next-Generation Phishing: How LLM Agents Empower Cyber Attackers
Afane, K., Wei, W., Mao, Y., Farooq, J., & Chen, J. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825018 · Cited by 44 · see citing works
Attack Prompt Generation for Red Teaming and Defending Large Language Models
Deng, B., Wang, W., Feng, F., Deng, Y., Wang, Q., & He, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.143 · Cited by 50 · see citing works
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33 · see citing works
Embedding Democratic Values into Social Media AIs via Societal Objective Functions
Jia, C., Lam, M. S., Mai, M. C., Hancock, J. T., & Bernstein, M. S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641002 · Cited by 31 · see citing works
Mustapha, K. B., Mustapha, K. B., Yap, E. H., & Abakr, Y. A. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-10-2023-0198 · Cited by 28 · see citing works
Esmradi, A., Yip, D. W., & Chan, C. F. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-97-1274-8_6 · Cited by 24 · see citing works
Prompt-to-SQL Injections in LLM-Integrated Web Applications: Risks and Defenses
Pedro, R., Coimbra, M. E., Castro, D., Carreira, P., & Santos, N. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00007 · Cited by 56 · see citing works
Cyberattacks Using ChatGPT: Exploring Malicious Content Generation Through Prompt Engineering
Alotaibi, L., Seher, S., & Mohammad, N. (2024) · ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS) · doi:10.1109/icetsis61505.2024.10459698 · Cited by 24 · see citing works
Esmradi, A., Yip, D. W., & Chan, C. F. (2023) · arXiv · doi:10.48550/arxiv.2312.10982 · Cited by 24 · see citing works
Human innovation and the creative agency of the world in the age of generative AI
Peschl, M. F. (2024) · Possibility Studies & Society · doi:10.1177/27538699241238049 · Cited by 23 · see citing works
ChatGPT Knows Your Attacks: Synthesizing Attack Trees Using LLMs
Gadyatskaya, O., & Papuc, D. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7969-1_18 · Cited by 22 · see citing works
Artificial intelligence security and privacy: a survey
He, X., Xu, G., Han, X., Wang, Q., Zhao, L., Shen, C., Lin, C., Zhao, Z., Li, Q., Yang, L., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4388-5 · Cited by 23 · see citing works
Mapping the individual, social and biospheric impacts of Foundation Models
Hernández, A. D., Krishna, S., Perini, A., Katell, M. A., Bennett, S., Borda, A., Hashem, Y., Hadjiloizou, S., Mahomed, S., Jayadeva, S., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658939 · Cited by 20 · see citing works
Recommender Systems Meet Large Language Model Agents: A Survey
Zhu, X., Wang, Y., Gao, H., Xu, W., Wang, C., Liu, Z., Wang, K., Jin, M., Pang, L. X., Weng, Q., et al. (2025) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000050 · Cited by 21 · see citing works
Yip, D. W., Esmradi, A., & Chan, C. F. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487667 · Cited by 19 · see citing works
On Large Language Models’ Resilience to Coercive Interrogation
Zhang, Z., Shen, G., Tao, G., Cheng, S., & Zhang, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00208 · Cited by 14 · see citing works
Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks
Xing, W., Li, M., Li, M., & Han, M. (2026) · ACM Computing Surveys · doi:10.1145/3806048 · Cited by 31 · see citing works
Survey on AI Ethics: A Socio‐Technical Perspective
Mbiazi, D., Bhange, M., Babaei, M., Sheth, I., Kenfack, P. J., & Kahou, S. E. (2025) · Computational Intelligence · doi:10.1111/coin.70149 · Cited by 31 · see citing works
Enhancing Privacy and Security in Large-Language Models: A Zero-Knowledge Proof Approach
Singh, S. R. (2024) · International Conference on Cyber Warfare and Security · doi:10.34190/iccws.19.1.2096 · Cited by 12 · see citing works
Designing an artificial intelligence-enabled large language model for financial decisions
Saxena, A., & Rishi, B. (2025) · Management Decision · doi:10.1108/md-02-2024-0305 · Cited by 11 · see citing works
JailbreakLens : Visual Analysis of Jailbreak Attacks Against Large Language Models
Feng, Y., Chen, Z., Kang, Z., Wang, S., Tian, H., Zhang, W., Zhu, M., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3575694 · Cited by 10 · see citing works
Decomposition, Synthesis, and Attack: A Multi-Instruction Fusion Method for Jailbreaking LLMs
Jiang, S., Chen, X., Xu, K., Chen, L., Ren, H., & Tang, R. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3525741 · Cited by 9 · see citing works
Contextual Spelling Correction with Large Language Models
Song, G. H., Wu, Z., Pundak, G., Chandorkar, A., Joshi, K., Velez, X., Caseiro, D., Haynor, B., Wang, W., Siddhartha, N., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389637 · Cited by 8 · see citing works
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.13352 · Cited by 7 · see citing works
“Potential consequences include exfiltrating user data, executing arbitrary code, and more [16, 21, 31, 39].”
BackgroundDetection and Defense Against Prominent Attacks on Preconditioned LLM-Integrated Virtual Assistants
Chan, C. F., Yip, D. W., & Esmradi, A. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487759 · Cited by 8 · see citing works
Dual Intention Escape: Penetrating and Toxic Jailbreak Attack against Large Language Models
Xue, Y., Wang, J., Yin, Z., Ma, Y., Qin, H., Tao, R., & Liu, X. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714654 · Cited by 7 · see citing works
S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models
Yuan, X., Li, J., Wang, D., Chen, Y., Mao, X., Huang, L., Chen, J., Xue, H., Liu, X., Wang, W., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728971 · Cited by 27 · see citing works
Exploring the Dual Role of LLMs in Cybersecurity: Threats and Defenses
Bryce, C., Kalousis, A., Leroux, I., Madinier, H., Pasche, T., & Ruch, P. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_26 · Cited by 6 · see citing works
AI-powered smart toys: interactive friends or surveillance devices?
Pavliv, V., Akbari, N., & Wagner, I. (2024) · International Conference on the Internet of Things (IoT) · doi:10.1145/3703790.3703841 · Cited by 6 · see citing works
A Literature Review of Personalized Large Language Models for Email Generation and Automation
Novelo, R., Silva, R. R., & Bernardino, J. (2025) · Future Internet · doi:10.3390/fi17120536 · Cited by 6 · see citing works
Security and Privacy of Large Language Models: Threat Taxonomy, Ethical Implications, and Governance
Pribisalić, M., & Martinčić-Ipšić, S. (2026) · AI · doi:10.3390/ai7050152 · Cited by 9 · see citing works
LLM-driven Instruction Following: Progresses and Concerns
Yin, W., Ye, Q., Liu, P., Ren, X., & Schütze, H. (2023) · doi:10.18653/v1/2023.emnlp-tutorial.4 · Cited by 7 · see citing works
The Dangerous Effects of a Frustratingly Easy LLMs Jailbreak Attack
Bombieri, M., Ponzetto, S. P., & Rospocher, M. (2025) · IEEE Access · doi:10.1109/access.2025.3589112 · Cited by 5 · see citing works
Rashid, S., Bollis, E., Pellicer, L. F. A. O., Rabbani, D. S. R., Palacios, R., Gupta, A., & Gupta, A. D. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7030077 · Cited by 6 · see citing works
Jin, Z., Liu, S., Li, H., Zhao, X., & Qu, H. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3557568 · Cited by 6 · see citing works
IntentObfuscator: A Jailbreaking Method via Confusing LLM with Prompts
Shang, S., Yao, Z., Yao, Y., Su, L., Fan, Z., Zhang, X., & Jiang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70903-6_8 · Cited by 4 · see citing works
Design Patterns for Securing LLM Agents against Prompt Injections
Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.08837 · Cited by 2 · see citing works
Effectiveness of Privacy-preserving Algorithms in LLMs: A Benchmark and Empirical Analysis
Sun, J., Suleiman, B., Ullah, I., & Razzak, I. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714531 · Cited by 3 · see citing works
Liu, Y., Cong, T., Zhao, Z., Michael Backes, A., Shen, Y., & Zhang, Y. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3672386 · Cited by 3 · see citing works
RogueGPT: transforming ChatGPT-4 into a rogue AI with dis-ethical tuning
Buscemi, A., & Proverbio, D. (2025) · AI and Ethics · doi:10.1007/s43681-025-00750-4 · Cited by 3 · see citing works
A2CEM: A contrastive embedding framework for adversarial robustness in large language models
Ergün, A. E., & Onan, A. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114056 · Cited by 3 · see citing works
StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Models
Yoosuf, S., Ali, T., Lekssays, A., AlSabah, M., & Khalil, I. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07884-1_25 · Cited by 3 · see citing works
Ahmed, S. S., & Jothi, J. A. A. (2024) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas61960.2024.10732418 · Cited by 3 · see citing works
Exploring the Use and Misuse of Large Language Models
Valdez, H. P. D., Abri, F., Webb, J. B., & Austin, T. H. (2025) · Information · doi:10.3390/info16090758 · Cited by 5 · see citing works
Exploring Advanced Methodologies in Security Evaluation for Large Language Models
Huang, J., Zhang, J., Wang, Q., Han, W., & Zhang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-4519-7_10 · Cited by 2 · see citing works
Harmful Prompt Classification for Large Language Models
Gupta, O., Lozano, M., Busalim, A. H., Jaiswal, R. R., & Quille, K. (2024) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3701268.3701271 · Cited by 4 · see citing works
A Review of the Application of Prompt Engineering in the Safety of Large Language Models
Guo, H., Zhang, L., Feng, X., & Zheng, Q. (2024) · International Conference on Information Education and Artificial Intelligence (ICIEAI) · doi:10.1145/3724504.3724574 · Cited by 3 · see citing works
Yi, X., Li, Y., Shi, D., Wang, L., Wang, X., & He, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129101 · Cited by 4 · see citing works
Xu, Q., Tian, Z., Wu, H., Huang, Z., Liang, D., Song, Y., Pan, Z., Liu, F., & Li, D. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3626228 · Cited by 2 · see citing works
Hijacking Large Language Models via Adversarial In-Context Learning
Zhou, X., Yao, Q., Zade, S. Z., Khanduri, P., & Zhu, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37673-2_13 · Cited by 19 · see citing works
SwordEcho: A LLM Jailbreaking Optimization Strategy Driven by Reinforcement Learning
Tang, X., Xiao, W., Yao, Z., & Han, J. (2024) · doi:10.1145/3709026.3709115 · Cited by 1 · see citing works
Spear or Shield: The Role of Generative AI in Intelligent Network Services
Du, H., Niyato, D. T., Kang, J., Xiong, Z., Lam, K., Fang, Y. M., & Li, Y. (2025) · IEEE Network · doi:10.1109/mnet.2025.3594769 · Cited by 1 · see citing works
Gu, L., Ji, X., Zhang, Z., Ma, J., Jia, X., & Jiang, W. T. (2025) · Electronics · doi:10.3390/electronics14193967 · Cited by 1 · see citing works
Esugo, M., Alao, O., & Mahmoud, H. (2025) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon58223.2025.11221566 · Cited by 1 · see citing works
SCALA: Toward Imperceptible and Efficient Black-Box Textual Adversarial Perturbations
Sun, S., Brucker, A. D., Hu, J. W., Huang, X., & Ruan, W. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3629604 · Cited by 1 · see citing works
Shan, W., Fu, M. C., Yang, R., & Tantithamthavorn, C. K. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00029 · Cited by 1 · see citing works
Кушнерьов, О. С., Shevchuk, R., Yevseiev, S., & Karpiński, M. (2026) · Information · doi:10.3390/info17020155 · Cited by 1 · see citing works
Clouseau: A Hierarchical Multi-Agent Approach for Autonomous Attack Investigation
Aldaihan, A., Alotaibi, F., & Maffeis, S. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00051 · Cited by 1 · see citing works
Alshammari, A. A., & Alsaleh, O. I. (2026) · Electronics · doi:10.3390/electronics15112242 · Cited by 2 · see citing works
Li, J., Wang, J., Zhao, Z., & Wei, Z. (2026) · Computer Standards & Interfaces · doi:10.1016/j.csi.2026.104181 · Cited by 1 · see citing works
Collaborative Filtering of Malicious Information from the MULTimedia Data Using s
Parasuraman, M., Abirami, G., Arockia Raj, A., Batri, K., Munnangi, A. K., & Kumaresan, M. (2023) · International Conference on Computer Science and Emerging Technologies (CSET) · doi:10.1109/cset58993.2023.10346953 · Cited by 1 · see citing works
GalaxyGPT: A Hybrid Framework for Large Language Model Safety
Zhou, H., Zheng, J., & Zhang, L. (2024) · IEEE Access · doi:10.1109/access.2024.3425662 · Cited by 2 · see citing works
Frustratingly Easy Jailbreak of Large Language Models via Output Prefix AttacksPreprint
Wang, Y., Chen, M., Peng, N., & Chang, K.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-4385503/v1 · Cited by 1 · see citing works
Yi, X., Li, Y., Wang, L., Wang, X., & He, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5169050 · Cited by 1 · see citing works
АТАКИ З ПРОМПТ-ІН’ЄКЦІЯМИ ТА МЕТОДИ ЇХ ЗАПОБІГАННЯ
Пелипад, М. (2024) · Международная научно-практическая конференция «Актуальные проблемы науки и образования в условиях современных вызовов» · doi:10.62731/mcnd-31.05.2024.008
Filter for confidential information
Bezymiannyi, O., & Shapoval, N. (2023) · Electronics and Control Systems · doi:10.18372/1990-5548.78.18256
Italiano, G. F., Martino, A., & Piccardo, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-81404-4_1
Yan, L., Lyu, C., Xing, R., Li, W., Samih, Y., Ji, T., Wang, L., & Jiang, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_12
Dynamic Adversarial Training for Jailbreak Robustness: A Security Enhancement Scheme for LLMs
Ma, M., Chu, Q., Yuan, H., Gong, T., & Yu, N. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_43
Personalized Prompt Attack Strategies Against Large Language Models
Zhang, Z., Gao, Y., Su, Z., & Wang, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_13
Can’t Say Cant? Measuring and Reasoning of Dark Jargons in Large Language Models
Ji, X., Zhang, J., Zhou, Z., Zhao, Z., Qiao, Q., Han, K., Hossen, M. I., & Hei, X. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-94458-1_7
Securing social network user data in large language model deployments: challenges and best practices
Jalali, N. A., & Hongsong, C. (2025) · Cluster Computing · doi:10.1007/s10586-025-05482-y
JailBreakLLM: An Effective LLaMa Model Designed Specifically to Jailbreak OpenAI GPT
Antor, R. A., Newaz, K. T., Mahmud, S., Mahi, M. T., & Mansoor, N. (2025) · IEEE International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN) · doi:10.1109/qpain66474.2025.11171650
Working Towards Toxic datasets for LLM Safeguarding
Guo, L., Wang, Z., Wang, Z., & He, T. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227669
StreamGuard: A Streaming-Based Defense Against Jailbreaking Attacks in Large Language Models
Li, L., Gao, X., Wang, X., Zhao, J., & Ye, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4109-6_21
G-STAR: A Threat Modeling Framework for General-Purpose AI Systems
Xiong, P., Lohrasbi, S., Kotian, P., & Buffett, S. (2025) · Conference on Privacy, Security and Trust · doi:10.1109/pst65910.2025.11268824
Boosting Jailbreak Transferability for Large Language Models
Zheng, J., Liu, H., Zhou, L., Yan, H., & Liu, P. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5761-5_31
A world without violet: peculiar consequences of granting moral status to Artificial Intelligences
Topan, S. (2026) · AI & Society · doi:10.1007/s00146-025-02834-5
AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Software
Yang, R., Fu, M. C., Tantithamthavorn, C. K., Arora, C., Gulmammadova, G., & Chua, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00279 · Cited by 1 · see citing works
HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
Krylov, A. S., Vagizov, I. I., Korzh, D., Douiba, M., Guezzaz, A., Kokh, V. N., Erokhin, S. D., Тутубалина, Е. В., & Rogov, O. Y. (2025) · Doklady Mathematics · doi:10.1134/s1064562425700425 · Cited by 1 · see citing works
- Authors
- Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Stanford University (US)
- Berkeley
- Venue
- 2024 IEEE Security and Privacy Workshops (SPW)
- Publisher
- IEEE
- Published
- 2024-05-23
- Pages
- 132-143
- OpenAlex
- W4400315206
- Cited by
- 114
- References
- 45
- Citations by year
- 2026: 29 · 2025: 41 · 2024: 30 · 2023: 14
- FWCI
- 29.51
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Security and Privacy Workshops (SPW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- large language models, dual-use risks, hate speech, scams, instruction following, security attacks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
