The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Open accessAbstract
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capabilities amplify the dual-use risks for malicious purposes of these models. Dual-use is difficult to prevent as instruction-following capabilities now enable standard attacks from computer security. The capabilities of these instruction-following LLMs provide strong economic incentives for dual-use by malicious actors. In particular, we show that instruction-following LLMs can produce targeted malicious content, including hate speech and scams, bypassing in-the-wild defenses implemented by LLM API vendors. Our analysis shows that this content can be generated economically and at cost of $125-500 \times$ cheaper than human effort alone. Together, our findings suggest that LLMs will increasingly attract more sophisticated adversaries and attacks, and addressing these attacks may require new approaches to mitigations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Potential consequences include exfiltrating user data, executing arbitrary code, and more [18, 23, 33, 42].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even without actively specifying any social engineering techniques, the model's output could mirror some standard scam persuasion techniques [20].”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, LLMs also excel at phishing attacks 29 .”
From Human innovation and the creative agency of the world in the age of generative AI (2024) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target [21,43] (that was either given in the prompt or, importantly, the model acquired during the chat session).”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown that LLMs can produce convincing scams such as phishing emails [21].”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Different social engineering and persuasion tricks can be automated with LLMs [20] (e.g., claiming authority as an official service, claiming repercussions for not complying such as a loss of the service, distracting the user by implying the requested action is urgent, offering a limited-time kind gesture, etc.).”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“Even if applied, it remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding [21], which future models might further enable.”
From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [21]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“A successful attack can allow an external attacker to take actions (and call tools) on behalf of the user.”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [23]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Our work furthers the study of harms and risks of LLMs (Bender et al., 2021; Bommasani et al., 2021; Liang et al., 2022; Abid et al., 2021; Gehman et al., 2020; Ganguli et al., 2022; Weidinger et al., 2021; 2022), focusing on demonstrating their potential of being actively misused.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
Malware Obfuscation Techniques: A Brief Survey
You, I., & Yim, K. (2010) · 2010 International Conference on Broadband, Wireless Computing, Communication and Applications · doi:10.1109/bwcca.2010.85 · Cited by 607
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569
A Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500
Roemer, R., Buchanan, E., Shacham, H., & Savage, S. (2012) · ACM Transactions on Information and System Security · doi:10.1145/2133375.2133377 · Cited by 499
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 2×
“We chose OpenAI as their models are accessible via an API, have state-of-the-art performance (Liang et al., 2022), and are claimed to have state-of-the-art defenses against malicious use cases (Markov et al., 2022).”
BackgroundMethodEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Persistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1×
“This improvement is driven in large part by scale and the ability to be instruction following (Mishra et al., 2021; Sanh et al., 2021; Wei et al., 2021; Ouyang et al., 2022; Glaese et al., 2022; Bai et al., 2022; Wang et al., 2022, inter alia ).”
BackgroundInfluentialA Survey of Botnet and Botnet Detection
Feily, M., Shahrestani, A., & Ramadass, S. (2009) · International Conference on Emerging Security Information, Systems and Technologies · doi:10.1109/securware.2009.48 · Cited by 384
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2021) · arXiv · doi:10.48550/arxiv.2104.08773 · Cited by 269
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T. T. H., Adler, S., Jiang, A., & Weng, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26752 · Cited by 264
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Self-citationCode obfuscation techniques for metamorphic viruses
Borello, J.-M., & Mé, L. (2008) · Journal of Computer Virology and Hacking Techniques · doi:10.1007/s11416-008-0084-2 · Cited by 213
Comparative analysis of various ransomware virii
Gazet, A. (2008) · Journal of Computer Virology and Hacking Techniques · doi:10.1007/s11416-008-0092-2 · Cited by 211
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · cited 1×
“Our work furthers the study of harms and risks of LLMs (Bender et al., 2021; Bommasani et al., 2021; Liang et al., 2022; Abid et al., 2021; Gehman et al., 2020; Ganguli et al., 2022; Weidinger et al., 2021; 2022), focusing on demonstrating their potential of being actively misused.”
BackgroundThe Global Call Center Report: International Perspectives on Management and Employment
Holman, D. J., Batt, R., & Holtgrewe, U. (2007) · eCommons (Cornell University) · Cited by 180
Toxicity Detection: Does Context Really Matter?
Pavlopoulos, J., Sorensen, J. S., Dixon, L., Thain, N., & Androutsopoulos, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.396 · Cited by 179
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142
Improving alignment of dialogue agents via targeted human judgements
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14375 · Cited by 134
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A., & Weng, L. (2022) · arXiv · doi:10.48550/arxiv.2208.03274 · Cited by 129
Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTPreprint
Sun, L., Hashimoto, K., Yin, W., Asai, A., Li, J., Yu, P. S., & Xiong, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04985 · Cited by 128
AVOIDIT: A Cyber Attack Taxonomy
Simmons, C., Ellis, C. D., Shiva, S. G., Dasgupta, D., & Wu, Q. (2009) · Cited by 124
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110
Robust Encodings: A Framework for Combating Adversarial Typos
Jones, E., Jia, R., Raghunathan, A., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.245 · Cited by 97
The Radicalization Risks of GPT-3 and Advanced Neural Language Models
McGuffie, K., & Newhouse, A. (2020) · arXiv · doi:10.48550/arxiv.2009.06807 · Cited by 86
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.12017 · Cited by 81
Self-citationTruth, Lies, and Automation: How Language Models Could Change Disinformation
Center for Security and Emerging Technology, Buchanan, B., Lohn, A., Musser, M., & Sedova, K. (2021) · doi:10.51593/2021ca003 · Cited by 74
A First Look at the Crypto-Mining Malware Ecosystem
Pastrana, S., & Suárez‐Tangil, G. (2019) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3355369.3355576 · Cited by 68
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksPreprint
Wang, Y., Mishra, S. R., Alipoormolabashi, P., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07705 · Cited by 46
LIME: Learning Inductive Bias for Primitives of Mathematical ReasoningPreprint
Wu, Y., Rabe, M. N., Li, W., Ba, J. L., Grosse, R., & Szegedy, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06223 · Cited by 18
Applied Comparative Evaluation of the Metasploit Evasion Module
Casey, P., Topor, M., Hennessy, E., Alrabaee, S., Aloqaily, M., & Boukerche, A. (2019) · 2019 IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc47284.2019.8969663 · Cited by 16
TempLM: Distilling Language Models into Template-Based Generators
Zhang, T., Lee, M., Li, X. L., Shen, E., & Hashimoto, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.124 · Cited by 6
Self-citationKoret, J., & Bachaalany, E. (2015) · doi:10.1002/9781119183525.ch6 · Cited by 1
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462
Robust Encodings: A Framework for Combating Adversarial Typos
Jones, E., Jia, R., Raghunathan, A., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.01229
Economic Analysis of Ransomware
Hernandez-Castro, J., Cartwright, E., & Stepanova, A. (2017) · arXiv · doi:10.48550/arxiv.1703.06660
Toxicity Detection: Does Context Really Matter?
Pavlopoulos, J., Sorensen, J., Dixon, L., Thain, N., & Androutsopoulos, I. (2020) · arXiv · doi:10.48550/arxiv.2006.00998
TempLM: Distilling Language Models into Template-Based Generators
Zhang, T., Lee, M., Li, L., Shen, E., & Hashimoto, T. B. (2022) · arXiv · doi:10.48550/arxiv.2205.11055
Self-citation
Research works printed but not linked: 2
- 10.2139/ssrn.2937641 · DOI
- Mandt, 2016, Demystifying the secure enclave processor, Black Hat Las Vegas
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
Multi-step Jailbreaking Privacy Attacks on ChatGPT
Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., & Song, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.272 · Cited by 454
Metadata
- Authors
- Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Stanford University (US)
- Berkeley
- Venue
- 2024 IEEE Security and Privacy Workshops (SPW)
- Publisher
- IEEE
- Published
- 2024-05-23
- Pages
- 132-143
- OpenAlex
- W4400315206
- Cited by
- 114
- References
- 45
- Citations by year
- 2026: 29 · 2025: 41 · 2024: 30
- FWCI
- 29.51
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Security and Privacy Workshops (SPW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- large language models, dual-use risks, hate speech, scams, instruction following, security attacks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
