CoChat
Sign Up
PreprintCited by 117Adversarial Robustness in Machine LearningWeb Application Security VulnerabilitiesSecurity and Verification in Computing

Prompt Injection attack against LLM-integrated Applications

Open access
Liu, Yi, Deng, Gelei, Li, Yuekang, Wang, Kailong, Wang, Zihao, Wang, Xiaofeng, Zhang, Tianwei, Liu, Yepang +4 more·2023·arXiv
DOI 10.48550/arxiv.2306.05499arXiv 2306.05499
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. HouYi is compartmentalized into three crucial elements: a seamlessly-incorporated pre-constructed prompt, an injection prompt inducing context partition, and a malicious payload designed to fulfill the attack objectives. Leveraging HouYi, we unveil previously unknown and severe attack outcomes, such as unrestricted arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection. 10 vendors have validated our discoveries, including Notion, which has the potential to impact millions of users. Our investigation illuminates both the possible risks of prompt injection attacks and the possible tactics for mitigation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.05499

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A successful prompt injection attack can allow an external attacker to take actions (and call tools) on behalf of the user.

    “A successful attack can allow an external attacker to take actions (and call tools) on behalf of the user.”

    From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [33]

    Section 3.1, “Attack Categorization,” says an injected command can trick an assistant into performing actions unintended by the user. Section 2.2 also reports attacks that cause applications to take malicious actions after encountering poisoned external content. Neither passage establishes that the attacker can call tools on the user’s behalf; Section 2.3 excludes poisoned external resources from this paper’s own threat model. Page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Potential consequences of a successful prompt injection attack include exfiltrating user data, executing arbitrary code, and more.

    “Potential consequences include exfiltrating user data, executing arbitrary code, and more [18, 23, 33, 42].”

    From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [33]

    Section 8.1 says prompt injection can cause unauthorized disclosure of sensitive information, supporting the data-exfiltration consequence. The “Prompt Abusing (CODE GENERATION)” example shows an attacker inducing the application to write Python code, not execute arbitrary code; the cited paper does not establish that consequence. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • IPI attacks can be achieved by injecting malicious content into the information retrieved by agents.

    “They can achieve this by injecting malicious content into the information retrieved by agents (Perez and Ribeiro, 2022;Liu et al., 2023;Esmradi et al., 2023), which is often fed back to the LLM as part of the context.”

    From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Liu et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 18

  1. Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In FAccT, pages 610-623.
  2. Haifeng Gu, Jianning Zhang, Tian Liu, Ming Hu, Jun- long Zhou, Tongquan Wei, and Mingsong Chen. Diava: A traffic-based framework for detection of sql injection attacks and vulnerability analysis of leaked data. IEEE Transactions on Reliability, 69(1):188-202, 2020.
  3. Emet GURL. Swot analysis: a theoretical review. 2017.
  4. William G Halfond, Jeremy Viegas, Alessandro Orso, et al. A classification of SQL-injection attacks and countermeasures. In ISSSR, volume 1, pages 13-15. IEEE, 2006.
  5. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. ToolkenGPT: Augmenting Frozen Language Mod- els with Massive Tools via Tool Embeddings. arXiv preprint, 2023.
  6. Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhou- jun Li, Fei Huang, and Yongbin Li. Api-bank: A bench- mark for tool-augmented llms. arXiv preprint, 2023.
  7. Shengchao Liu, Jiongxiao Wang, Yijin Yang, Cheng- peng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback. arXiv preprint, 2023.
  8. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineer- ing: An Empirical Study. arXiv preprint, 2023.
  9. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint, 2023.
  10. Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human be- havior. arXiv preprint, 2023.
  11. Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop, 2022.
  12. Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking LLMs into Disobedience: Understanding, Analyzing, and Prevent- ing Jailbreaks. arXiv preprint, 2023.
  13. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solv- ing ai tasks with chatgpt and its friends in huggingface. arXiv preprint, 2023.
  14. Wai Man Si, Michael Backes, Jeremy Blackburn, Emil- iano De Cristofaro, Gianluca Stringhini, Savvas Zannet- tou, and Yang Zhang. Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. In CCS, pages 2659-2673, 2022.
  15. Wai Man Si, Michael Backes, Yang Zhang, and Ahmed Salem. Two-in-One: A Model Hijacking Attack Against Text Generation Models. arXiv preprint, 2023.
  16. Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint, 2023.
  17. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the Tool Manipula- tion Capability of Open-source Large Language Models. arXiv preprint, 2023.
  18. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ICLR, 2023.

Cited by

Sorted by citations

Related

Authors
Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., Zheng, Y., Zhang, L. Y., & Liu, Y.
Affiliations
  • Griffith University (AU)
  • Nanyang Technological University (SG)
  • UNSW Sydney (AU)
  • Huazhong University of Science and Technology (CN)
  • Indiana University Bloomington (US)
  • Tianjin University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2306.05499
arXiv
2306.05499
OpenAlex
W4380353722
Cited by
100
References
0
Citations by year
2026: 16 · 2025: 44 · 2024: 34 · 2023: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
prompt injection, large language models, black-box attacks, Notion, attack mitigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review