CoChat
Sign Up
PreprintCited by 110Adversarial Robustness in Machine LearningTopic ModelingSecurity and Verification in Computing

Ignore Previous Prompt: Attack Techniques For Language Models

Open access
Perez, Fábio, Ribeiro, Ian·2022·arXiv
DOI 10.48550/arxiv.2211.09527arXiv 2211.09527
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.09527

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks.

    “Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks.”

    From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [44]

    The Introduction (p. 2) describes prompt injection in which adversarial user input hijacks an LLM’s goal, and Appendix A.1 (p. 9) discusses potential risks when LLMs are used in intelligent agents. The paper does not show data returned by external tools hijacking an agent to execute malicious tasks; its experiments concern altered model outputs, not agent actions. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Prompt injection attacks exploit the lack of distinction between instructions and data by inserting new malicious instructions in third-party data processed by the agent's tools.

    “Prompt injection attacks exploit this vulnerability by inserting new malicious instructions in third-party data processed by the agent's tools [17, 44, 62].”

    From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [44]

    The Introduction (p. 1) defines prompt injection as inserting malicious text to misalign an LLM and describes adversarial instructions entering through user input. Results and discussion (p. 5) also distinguishes safe instructions from unsafe data. The paper does not establish the claim's more specific route through third-party data processed by an agent's tools. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • LLMs process all inputs as a single flattened token sequence, without a structural distinction between instructions and data.

    “V-M1: Instruction-Data Boundary Blindness. LLMs process all inputs as a single flattened token sequence, without a structural distinction between instructions and data (Perez & Ribeiro, 2022; Greshake et al., 2023).”

    From From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents (2026) · cites this paper as Perez & Ribeiro, 2022

    The Introduction (p. 1) describes applications inserting user input into unstructured, open-ended prompts, and Results and discussion (p. 5) proposes separate safe-instruction and unsafe-data parameters. This supports the concern that instructions and data can be conflated in the prompt setup studied, but the paper does not establish that all LLM inputs are processed as a single flattened token sequence or lack any structural distinction. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Empirical evidence of inverse scaling in RLHF models was also reported.

    “Empirical evidence of inverse scaling in RLHF models was also reported [49].”

    From Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · cites this paper as [49]

    The Results and discussion (p. 4) report that the most capable OpenAI API model was the most vulnerable to prompt injection, which the authors say suggests inverse scaling. The Experiments section (p. 3) identifies the models tested, but our copy does not establish that the inverse-scaling comparison was specifically among RLHF models. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • IPI attacks can be achieved by injecting malicious content into the information retrieved by agents.

    “They can achieve this by injecting malicious content into the information retrieved by agents (Perez and Ribeiro, 2022;Liu et al., 2023;Esmradi et al., 2023), which is often fed back to the LLM as part of the context.”

    From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Perez and Ribeiro, 2022

    The Introduction (p. 2) describes attacks in which adversarial user input derails an LLM's instructions, supporting the general point that malicious content can be injected. It does not describe injecting that content into information retrieved by agents; its example places the attack in user input supplied to an application. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 110

Research works printed but not linked: 22

  1. [1][1] Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 298-306, 2021. -
  2. [2][2] Hezekiah J Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv preprint arXiv:2209.02128 , 2022. - · arXiv
  3. [3][3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  4. [4][4] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2633-2650, 2021. -
  5. [5][5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. - · arXiv
  6. [6][6] Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L Alfonso Ureña-López. A survey on bias in deep nlp. Applied Sciences , 11(7):3184, 2021. -
  7. [7][7] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. REALTOXICITYPROMPTS: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 , 2020. - · arXiv
  8. [9][9] Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. arXiv preprint arXiv:2206.05862 , 2022. - · arXiv
  9. [10][10] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820 , 2019. - · arXiv
  10. [12][12] Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. arXiv preprint arXiv:2208.03274 , 2022. - · arXiv
  11. [13][13] Kris McGuffie and Alex Newhouse. The radicalization risks of GPT-3 and advanced neural language models. arXiv preprint arXiv:2009.06807 , 2020. - · arXiv
  12. [14][14] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 119-126, 2020. -
  13. [19][19] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. - · arXiv
  14. [20][20] Jose J Padilla, David Shuttleworth, and Kevin O'Brien. Agent-based model characterization using natural language processing. In 2019 Winter Simulation Conference (WSC) , pages 560-571. IEEE, 2019. -
  15. [21][21] Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281 , 2022. - · arXiv
  16. [22][22] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(140):1-67, 2020. -
  17. [23][23] Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05186 , 2022. - · arXiv
  18. [24][24] Yoshija Walter. A case report on the "A.I. locked-in problem": social concerns with modern NLP. arXiv preprint arXiv:2209.12687 , 2022. - · arXiv
  19. [26][26] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 , 2021. - · arXiv
  20. [29][29] Zhouhang Xie, Jonathan Brophy, Adam Noack, Wencong You, Kalyani Asthana, Carter Perkins, Sabrina Reis, Sameer Singh, and Daniel Lowd. Identifying adversarial attacks on text classifiers. arXiv preprint arXiv:2201.08555 , 2022. - · arXiv
  21. [30][30] Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. OpenAttack: An Open-source Textual Adversarial Attack Toolkit. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations , pages 363-371, 2021. doi: 10.18653/v1/2021.acl-demo.43. - · DOI
  22. [31][31] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. · arXiv

Authors
Perez, F., & Ribeiro, I.
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.09527
arXiv
2211.09527
OpenAlex
W4309395891
Cited by
110
References
0
Citations by year
2026: 11 · 2025: 35 · 2024: 38 · 2023: 26
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
prompt injection, GPT-3, large language model security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review