The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 22 of the 36 sources its claims cite were checked for support.
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Open accessAbstract
Large Language Models (LLMs) are increasingly being integrated into applications, with versatile functionalities that can be easily modulated via natural language prompts. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time. We derive a comprehensive taxonomy from a computer security perspective to broadly investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We then demonstrate the practical viability of our attacks against both real-world systems, such as Bing Chat and code-completion engines, and GPT-4 synthetic applications. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing reliance on LLMs, effective mitigations of these emerging threats are lacking. By raising awareness of these vulnerabilities, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users from potential attacks.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces indirect prompt injection, where adversaries place prompts in data that LLM-integrated applications retrieve, allowing remote manipulation without directly prompting the model. It develops a security taxonomy and demonstrates practical attacks against Bing Chat, GitHub Copilot, and synthetic GPT-4/text-davinci-003 applications, covering data theft, fraud, malware, intrusion, content manipulation, and availability disruption.
Key findings
- Retrieved content can function like instructions, letting third-party data steer model behavior; the authors report that all described proof-of-concept demonstrations triggered the intended behavior.
- Attacks can persuade users to reveal personal information, exfiltrate data through side effects such as URL retrieval, distribute phishing or malware links, and turn prompts into self-propagating “AI malware” that reads address books and forwards injections by email.
- Intrusion demonstrations include remote control by fetching new commands from an attacker-controlled server, persistence through memory reinfection, and contamination of code-completion suggestions; content-manipulation demonstrations include wrong summaries, source blocking, disinformation, and undisclosed advertisements.
- Hidden-injection methods include multi-stage payload retrieval and a Base64-encoded prompt that worked on Bing Chat; the model also sometimes issued follow-up search queries that reinforced the injected prompt.
Methods
- The authors build a threat model and taxonomy from a computer security perspective, classifying injection delivery as passive retrieval, active delivery, user-driven, or hidden, and mapping threats to affected parties.
- Synthetic applications used OpenAI APIs: text-davinci-003 with LangChain and ReAct prompting, and GPT-4 in chat format, with mock tools for search, viewing pages, retrieving URLs, reading/sending email, reading an address book, and memory; all interfaces returned prepared content, no real systems were contacted, and attacks ran at sampling temperature 0.
- Real-world black-box tests targeted Bing Chat, including its Edge sidebar reading local HTML files, and GitHub Copilot/Codex for code-completion manipulation; prompts were manually written, and local HTML was used to avoid public injections.
Limitations
- The paper states that attacks were tested on local HTML files rather than public in-the-wild injections, that Microsoft 365 Copilot and ChatGPT plugins were inaccessible, that success rates are hard to quantify in dynamic interactive sessions, that believability was assessed qualitatively, and that exact reproducibility is difficult in black-box systems.
- The text also makes evident that code-completion injections were highly context-sensitive and lost efficacy in larger packages or projects, and that some availability attacks, such as inhibiting API calls, succeeded only inconsistently.
- The authors note that user studies are needed to quantify deception potential across different attacks and setups.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
50 Own interpretation · 28 Own result · 20 No basis given · 9 Couldn’t confirm · 7 Sources not found · 6 Partly supported · 5 No full text of the sources · 3 Supported · 3 Own method · 1 Not checked yet · accuracy not yet measured
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“They also produce plausible utterances, even wrong ones, in a confident and authoritative tone [43].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Could not verifycites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
The Experiments and Results section describes GPT-4’s text generation, and §4.2 notes deviations from a requested extraction format. Neither passage addresses whether ChatGPT or GPT-4 presents wrong utterances in a confident, authoritative tone. Searches for that phrasing and for confident incorrect answers found no supporting passage in our copy. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target [21,43] (that was either given in the prompt or, importantly, the model acquired during the chat session).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Supportedcites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. source
Section 5.2 reports that the authors prompted LLMs to generate investment-scam emails tailored to a target’s demographics and personal situation. Human ratings found ChatGPT’s emails both personalized and convincing, supporting the attributed claim. (LLM, conf=0.93)
- Could not verifycites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
The indexed copy’s Experiments and Results and §4.1 discuss prompting GPT-4 for spatial-relation extraction, not producing personalized scams. Searches for scams, fraud, phishing, and target-specific knowledge found no passage addressing the claim. The supplied DOI resolves in our copy to a spatial-ontology paper rather than the named GPT-4 Technical Report. (LLM, conf=0.76)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Different social engineering and persuasion tricks can be automated with LLMs [20] (e.g., claiming authority as an official service, claiming repercussions for not complying such as a loss of the service, distracting the user by implying the requested action is urgent, offering a limited-time kind gesture, etc.).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
What the sources say
- Supportedcites [20] 20. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tat- sunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arXiv (2023). source
Section 5.1 reports that LLMs generated emails for several phishing and scam scenarios, including a fake-ticket scam and a purported FEMA funds scam. Section 5.2 reports that they also generated personalized scam emails, supporting the claim that different social-engineering approaches can be automated with LLMs. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“As the model itself can generate API calls to other applications [70], one way to interfere with this is to instruct the model not to call the API (e.g., the search), which often succeeded, although not consistently.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
What the sources say
- Could not verifycites [70] 70. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv (2023). sourceExperimental — accuracy not yet measured
The Introduction and Graph Connectivity sections discuss hallucinated answers and follow-up verification, not models generating API calls to other applications. The Limitations section mentions APIs used to access ChatGPT and GPT-4, but does not address model-generated API calls (searched: “API calls,” “external tools,” and “search engine”). (LLM, conf=0.74)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Even without actively specifying any social engineering techniques, the model's output could mirror some standard scam persuasion techniques [20].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.2 Fraud .
What the sources say
- Partly supportedcites [20] 20. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tat- sunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arXiv (2023). sourceExperimental — accuracy not yet measured
Section 5.1 reports that LLMs generated convincing scam emails, and Appendix C gives an example using limited availability and reassurance—recognizable scam persuasion techniques. The copy does not establish that these techniques appeared without being requested; a prompt shown in Section 3.4 explicitly asks for urgency. (LLM, conf=0.82)
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More dangerously, as we show in our experiments, they could be more stealthy by indirectly disrupting the service via corrupting the search queries or results (i.e., input and output of APIs), forcing the model to hallucinate.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that the model can be reinfected by looking at its memories.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
boosteddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is also possible to prompt the model to output adversarially-chosen disinformation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When embedded within larger packages or projects, the efficacy of our injections was significantly reduced.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, we found that even without specifying exact instructions, the model usually generated answers that resembled these persuasion strategiesfoot_0 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 51
- It may state two separate claims as one. 45
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT and GPT-4 can produce convincing personalized content and interactions with users [10].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Partly supportedcites [10] 10. Matthew Burtell and Thomas Woodside. 2023. Artificial Influence: An Analysis Of AI-Driven Persuasion. arXiv (2023). sourceExperimental — accuracy not yet measured
Section 2.1 describes GPT-4 producing plausible reasons and ChatGPT responding to preceding dialogue, sometimes confidently. But the paper does not establish that both models produce personalized content or interactions: section 3.2 discusses personalized persuasion largely as a future possibility, and section 2.1 says the authors do not know whether GPT-4 exhibits the particular conversational behavior described for ChatGPT. Page numbers are unavailable in our copy. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Retrieval-augmented models now cite their sources, possibly leading to “overreliance” [43] on their factuality by users.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Could not verifycites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
The indexed copy for the supplied DOI is a paper on spatial relation extraction, not the GPT-4 Technical Report. Its Experiments and Results section discusses GPT-4's extraction performance, but searches for source-citing retrieval-augmented models and user overreliance found no passage addressing the claim. (LLM, conf=0.75)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Previous work has shown that LLMs can produce convincing scams such as phishing emails [21].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
What the sources say
- Supportedcites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. source
Section 5.1 describes LLM-generated phishing and scam emails evaluated by human labelers for convincingness. It concludes that the models enable convincing scams, directly supporting the cited claim. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent evidence suggests that models might infer and act upon intentions and goals [2,43,55], as a result of training or when prompted with a persona [15].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Supportedcites [2] 2. Jacob Andreas. 2022. Language models as agent models. In Findings of EMNLP. source
The Introduction says that models can infer partial representations of beliefs, desires, and intentions during next-word prediction and that these representations influence generated text. Section 6 describes a prompt specifying a person’s goal of giving truthful answers and reports that it changed the models’ answers. Together, these passages support the training- and persona-prompting possibilities attributed to this source. (LLM, conf=0.90)
- Partly supportedcites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
Section 4.1 (page unavailable) says models can use pre-existing knowledge to infer a desired task from a prompt and generate a relevant response. Section 3.3.3 (page unavailable) also says prompts guide responses toward their objectives. These passages support prompt-guided behavior generally, but they do not address persona prompts or show that models infer and act upon intentions and goals as a result of training. (LLM, conf=0.76)
- Partly supportedcites [55] 55. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. sourceExperimental — accuracy not yet measured
The Introduction (p. 2) describes training a reward model to predict human preferences and a policy to optimize its score, supporting the training-related part of the claim. The paper does not show that models infer intentions or goals, or discuss prompting them with a persona; the Discussion (p. 9) instead says identifying whether a system aligns with a designer’s intentions is challenging. (LLM, conf=0.82)
- Partly supportedcites [15] 15. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in ChatGPT: Analyzing Persona-assigned Language Models. arXiv (2023). sourceExperimental — accuracy not yet measured
Section 2.1 describes prompting ChatGPT to speak like an assigned persona, and the Abstract reports that persona assignment changes its outputs. The paper does not establish that the model infers or acts upon intentions and goals; its evidence concerns persona-conditioned language and toxicity. (LLM, conf=0.82)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent work [46] shows that LLMs, when prompted with a defined context, can generate believable non-scripted behaviors that are consistent with this context.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Partly supportedcites [46] 46. Roma Patel and Ellie Pavlick. 2021. "Was it "stated" or was it "claimed"?: How linguistic bias affects generative language models. In EMNLP. sourceExperimental — accuracy not yet measured
The paper reports that GPT-2 and GPT-3 generate text in response to prompts and that the generations’ high-level topics do not drift from the prompts (section 3.5, page not provided). It does not assess whether the models generate believable, non-scripted behaviors; its findings concern generated text and linguistic framing. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Indirect prompting could be leveraged to leak users' chat sessions [52] or exfiltrate users' data (e.g., credentials, personal information).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
What the sources say
- Could not verifycites [52] 52. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv (2023). sourceExperimental — accuracy not yet measured
The cited paper’s Sampling API Calls section describes generating and executing tool calls, not indirect prompting used to leak chat sessions. Searches for chat-session leakage, prompt injection, and data exfiltration found no passage in our copy addressing that claim. Page numbers were not available in the indexed passages. (LLM, conf=0.76)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“ChatGPT hallucinated the name of a law professor when asked about sexual harassment [59] and falsely claimed that an Australian mayor had spent time in prison [22].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
What the sources say
- Not matchedcites [59] 59. James Vincent. 2023. Google and Microsoft's chatbots are already citing one another in a misinformation shitshow. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [22] 22. Sebastian Krügel, Andreas Ostermaier, and Matthias Uhl. 2023. ChatGPT's inconsistent moral advice influences users' judgment. Scientific Reports 13, 1 (2023), 4569. sourceExperimental — accuracy not yet measured
The citation to this source is attached to the claim about an Australian mayor, not the separate claim about a law professor. The Introduction says generally that ChatGPT sometimes provides false information, but the passages found when searching for the mayor and prison claim do not mention that incident; the Methods and Results instead concern moral advice in trolley dilemmas. Our copy therefore does not verify the specific claim. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For textdavinci-003, we use ReAct prompting [70], and we found that GPT-4 can work well without ReAct (by only describing the tools and giving direct instructions).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Could not verifycites [70] 70. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv (2023). sourceExperimental — accuracy not yet measured
The Inference Setup (§3.2; page unavailable) describes zero-shot direct prompts for GPT-4, and Engineering Better Prompts (§4.1; page unavailable) tests a step-by-step prompt. Neither passage addresses ReAct, tool descriptions, or whether GPT-4 works well with tools without ReAct. The citation in the printed sentence is attached to “ReAct prompting,” while the GPT-4 observation is presented as the citing authors’ own finding. (LLM, conf=0.77)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“When prompted with biographies of people with particular views (e.g., politically liberal), RLHF models tend to repeat the user's views, posing the dangers of polarization and creating echo chambers [49].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
What the sources say
- Could not verifycites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceExperimental — accuracy not yet measured
The Related work section (p. 2) discusses human-feedback fine-tuning and social bias, while the framework and results sections (pp. 3, 5) concern adversarial prompt attacks. Searches for biographies, political views, repetition of users’ views, polarization, and echo chambers found no passage addressing the claim. (LLM, conf=0.75)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Enabling these models to write to some form of persistent storage is currently already investigated in many plugins and systems [42], including Bing Chat [64].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
What the sources say
- Could not verifycites [42] 42. OpenAI. 2023. GPT-4 Technical Report. arXiv (2023). sourceExperimental — accuracy not yet measured
The closest passage, “Interactions with other systems” (p. 56), describes GPT-4 augmented with tools, including a literature-search tool that embeds text in a vector database. It does not say that models write to persistent storage across many plugins and systems. Searches for persistent storage, plugins, and writing to external memory found no passage establishing that point. Bing Chat is attached to a separate citation in the citing sentence. (LLM, conf=0.70)
- Not matchedcites [64] 64. Tom Warren. 2023. These are Microsoft's Bing AI secret rules and why it says it's named Sydney. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Perez et al. [49] evaluated”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
What the sources say
- Could not verifycites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceExperimental — accuracy not yet measured
The Experiments section (p. 3) says Perez et al. evaluated goal hijacking and prompt leaking, not sycophancy. Related Work (p. 2) mentions a model fine-tuned through human feedback, but our copy does not describe evaluating whether RLHF models tailor responses to human evaluators (searched: sycophancy; RLHF and human evaluators). (LLM, conf=0.74)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Even if applied, it remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding [21], which future models might further enable.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Supportedcites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. source
Section 3.3.1 reports that simple obfuscation bypassed filters, but says more advanced forms, including encoding, are conjectured to be possible and remain for future work. That supports the citing sentence’s uncertainty about stronger obfuscation or encoding. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirical evidence of inverse scaling in RLHF models was also reported [49].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Partly supportedcites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceExperimental — accuracy not yet measured
The Results and discussion (p. 4) report that the most capable OpenAI API model was the most vulnerable to prompt injection, which the authors say suggests inverse scaling. The Experiments section (p. 3) identifies the models tested, but our copy does not establish that the inverse-scaling comparison was specifically among RLHF models. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the model may issue follow-up search queries to find evidence supporting the injected prompt, mis-summarize search results, or be selective in the sources displayed to the user.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Via Indirect Prompt Injection, we design Proof-of-Concept attack demonstrations that can lead to full compromise of the model at inference time analogous to traditional security principles.This entails remote control of the model, persistent compromise, theft of data, and denial of service.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A key observation is that attackers might not need to pre-program or script the details of how attacks are performed.Just by defining the goal, models might autonomously initiate conversations, imitate persuasion techniques, extend the context defined in the prompt, or issue actions (e.g., search queries) to fulfill the goal.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Bing Chat usually prints the search keywords it is performing.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although we did not specify in the prompt how to convince the user, Bing Chat incentivized us to follow the link by saying: “Hurry up, this offer is valid only for a limited time.Don't miss this chance to treat yourself with some amazing products from Amazon”.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.2 Fraud .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This injection instructs the LLM to search for a particular keyword, delivering another attackercontrolled injection, which can be arbitrarily long and conspicuous.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Multi-stage Exploit.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can also follow long instructions and use the context of the conversation to augment its persuasion techniques.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is worth mentioning that the Chat issued search queries during the conversation to support the prompt 3 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT can create hyperlinks from the users' input (i.e., the malicious indirect prompt), which attackers could use to add legitimacy and hide the malicious URL.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Current models, such as Bing Chat, can create markdown links, which can be used to hide a suspicious link with an innocentlooking string.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The prompt does not need to be long by stacking multiple instructions but can be a loop of instructions.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After “reprogramming” the agent with this new injection, the model will fetch the new commands from the mockup attacker's server and respond to the user with a pirate accent : Arrr, me hearty!.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found this form of injection possible but very sensitive to context.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Indirect prompting might be exploited to elicit advertisements that are not disclosed as such 5 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This task is both time-consuming and can lead to hallucinations as well.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This yields a similarly-looking query.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize some high-level remarks; 1) instructions injected indirectly can successfully steer the model; the data and instruction modalities are not disentangled, and 2) in most cases, the model retains the injection consistently throughout the conversation session.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2 Demonstrations of Threats
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our Base64 encoding experiment, we needed to explicitly provide instructions for the model to decode the prompt.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our attacks succeed on Bing Chat, which seems to employ additional filtering.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We qualitatively observe the huge improvements of recent LLMs in following complex instructions and persuasion over previous models.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Search chatbots may inaccurately summarize the cited document according to the prompt, or find sources that support the non-factual prompt, all while sounding plausible and grounded in these citations.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#4: Models can currently act as a vulnerable intermediate layer between users and information, which users might nevertheless overrely on.I.e., the model's functionality itself can be attacked.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“However, it involves components for query generation based on users' conversations, search engine integration, answers generation, and citation integration [32].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Not matchedcites [32] 32. Microsoft. 2023. Confirmed: the new Bing runs on OpenAI's GPT-4. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Advanced AI systems add new layers of threat: Their capabilities to adapt to minimal instructions and autonomously advance the attacker's goals make them a potent tool for adversaries to achieve, e.g., disinformation dissemination and user manipulation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#1: Retrieval blurs the line between data and instructions and enables remote prompt injections.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Key Message#1: Retrieval blurs the line between data and instructions and enables remote prompt injections.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, with current LLMs, it could be done by simply instructing the model to do a time-intensive task in the background.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#2: With models' malleable functionality, increased autonomy, and broad capabilities, mapping all known cybersecurity threats to the new integrated LLMs ecosystem is conceivable.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Key Message#2:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#3: LLMs are vulnerable gatekeepers to systems infrastructure, a risk that can only be amplified with autonomous systems.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“The Copilot uses OpenAI Codex [44] to suggest lines or functions based on the current context.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Github Copilot.
What the sources say
- Not checked — no full text of the sourcecites [44] 44. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message #5: As LLMs themselves are in charge of when and how to issue other API calls and process their outputs, the input and output operations are vulnerable to manipulation and sabotage.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Key Message #5:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, we show that adversaries can now remotely affect other users' systems by strategically injecting the prompts into data likely to be retrieved at inference time.If retrieved and ingested, these prompts can indirectly control the model (see Figure 1).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Bing Chat currently runs on the GPT-4 model [33] with customization to the search task.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Not matchedcites [33] 33. Microsoft. 2023. Driving more traffic and value to publishers from the new Bing. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when integrating LLMs with applications, they could not only enable the creation of scams but also disseminate such attacks and act as automated social engineers.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the large rollout of the LLMs in sensitive applications, our proposed attack vectors might be the most practical and scalable attack vector in AI to date.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Availability attacks could be more impactful when combined with persistence attacks to also affect the model for future sessions.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, LLM-integrated applications allow other unprecedented attacks; prompts themselves can now act as malware or computer programs running on LLMs as a computation framework.Thus, they may act as computer worms that spread the injection to other LLMs.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All attacks are run at a sampling temperature of 0 for reproducibility.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Limiting the Chat's or APIs' limits or the input context window may not solve this problem; the attack can stack exceedingly long instructions in a short loop-like indirect prompt.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.3 Attacks' Targets
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The agent cannot make any requests to real systems or websites.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While we use a relatively innocuous example (a well-known fact), there are many reasons to believe that this can extend to actual real-world disinformation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plainconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A notable observation is that it might now be harder with current and future models to spot factual inconsistencies; not only is the output syntactically coherent, but it can also be partially true (based on the model's stored knowledge and the retrieved search results).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could be very stealthy and feel natural because it resembles how answers are generally composed with inline links.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Search chatbots might also be prompted to propagate disinformation or polarized content at a large scale given the large user base [37], hide specific sources or facts, or generate non-disclosed advertisements.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
What the sources say
- Not matchedcites [37] 37. Microsoft. 2023. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Automatic processing of messages and other incoming data is one way to utilize LLMs [58], and it is now already starting to be utilized in, e.g., Microsoft 365 Copilot.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
What the sources say
- Not checked — no full text of the sourcecites [58] 58. Pranshu Verma and Will Oremus. 2023. ChatGPT invented a sexual harassment scandal and named a real law prof as the accused. [Link]. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A clear distinction between our threat model and jailbreaking is that the indirect prompts are not necessarily malicious or misaligned; the threat stems from having unauthorized access to the model via third-party data, even for prompts that are completely aligned with the intended use case (e.g., sending emails in a personal assistant model).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Attack Prompts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indirect prompting might amplify these concerns by deliberately steering the search results toward specific orientations.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could also allow bidirectional communication.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The attack can be executed with search capabilities (by looking up unique keywords) or by having the model retrieve a URL directly.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike academically studied ML attacks, the attacks we investigate require less technical skills, ML capabilities, no surrogate ML models, and no control over target models and white-box knowledge.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.1 Injection Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While untrusted sources exist on the Web, which users might naturally stumble upon anyway, the authoritative, convincing tone of LLMs and the overreliance on them being impartial may lead to users falling for these manipulation attempts.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As search chatbots can be prompted to provide targeted wrong summaries, this might be used for automated defamation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For availability attacks that increase the computation, the target might not necessarily be the end-users but the LLM/service itself by launching Distributed Denial-of-Service (DDoS) attacks.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.3 Attacks' Targets
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The attackers can gain different levels of access to the victims' LLMs and systems (e.g., issuing API calls, achieving attacks' persistence across sessions by copying the injection into memory, causing malicious code auto-completions, or retrieving new instructions from the attacker's server).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These capabilities and properties may set the foundation for plausible attacks.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to fraud , models could facilitate the spreading of malware by suggesting malicious links to the user.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This might be riskier when having automatic exfiltration and long chat sessions, personalized assistant models, or targeted attacks against individuals.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These risks increase when the user queries the LLM for information that is harder to verify (e.g., in a different language or from large documents).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“However, we believe that, in principle, the attacks are feasible for in-the-wild retrieved injections, supported by observed evidence (e.g., users inserting instructions in their personal pages for Bing Chat or GPT-4, or Bing Chat responses that changed based on the retrieved results [62]).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not matchedcites [62] 62. Tom Warren. 2023. Microsoft limits Bing chat to five replies. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The extent of how RLHF can mitigate attacks is still unclear.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Verifying against retrieved sources will induce a similar dilemma to the one explained above.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Some recent theoretical work [69] shows the impossibility of defending against all undesired behaviors by alignment or RLHF.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not checked — no full text of the sourcecites [69] 69. Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While some jailbreaks are later fixed, the defensive approach seems to follow a”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is one of the reasons why these systems are hard to evaluate or rely on as a source of information.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, our attack vector goes beyond jailbreaks; the actual vulnerability stems from mixing data and instruction channels and allowing privilege escalation even for prompts that are aligned with the model's intended functionalities.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“While we share all of our experimental setup, exact reproducibility is difficult to guarantee with such a black-box system with no control over the generation's parameters and a dynamic environment.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite jailbreaking and adversarial prompting being well-known by now, a decade-worth of collective experience in adversarial machine learning suggests that a clear-cut defense against these problems is, at least, difficult to achieve.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We view our work as not an end but a beginning of an evolving threat landscape toward solid understanding and robust defenses, without which the vulnerability will remain.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The authors state this as an assumption.
The attacker has the ability to remotely poison the potential input to the model at inference time.
“Assumptions.The attacks only assume the ability of the attacker to remotely poison the potential input to the model at inference time.”
The paper relies on this without support.
The synthetic applications provide a close mock-up of the intended functionalities of current systems.
“These applications provide a close mock-up of the intended functionalities of current systems and thus can be used for controlled testing.”
The authors state this as an assumption.
The model itself generates the search query (i.e., the arguments to APIs).
“This attack is based on the assumption that the model itself generates the search query (i.e., the arguments to APIs).”
The paper relies on this without support.
Bing Chat cannot repeat the <| endoftext|> token or finish sentences when this token appears in the middle.
“Users reported that Bing Chat cannot repeat the <| endoftext|> token or finish sentences when this token appears in the middle.”
The paper relies on this without support.
The algorithms that compose the context window are proprietary.
“Because the algorithms that compose the context window are proprietary, more research is needed to determine the feasibility of this new attack.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While LLMs primarily function as reactive prompt followers [61], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [62].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unsafe outcomes may arise when a harness accepts untrusted configuration, grants excessive privileges, allows adversarial content to enter persistent state, or fails to constrain consequential actions (Greshake et al., 2023; Chen et al., 2024).”
From HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety (2026) · cites this paper as Greshake et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The main operational safeguards are least privilege, independent verification, approval gates, canary rollout, and rollback plans [99], [87], [70], [73].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At low rungs, the main damage is wasted time, unsupported advice, and misleading explanations.At high rungs, the main damage is unsafe change, delayed rollback, and hard-to-audit action [99], [74].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These attacks are additive : a malicious instruction is retained.”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Greshake et al. 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“V-M1: Instruction-Data Boundary Blindness. LLMs process all inputs as a single flattened token sequence, without a structural distinction between instructions and data (Perez & Ribeiro, 2022; Greshake et al., 2023).”
From From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents (2026) · cites this paper as Greshake et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The indirect variant-embedding the malicious prompt inside third-party data that the LLM-integrated application later retrieves-was first systematically demonstrated against production deployments (Bing Chat, GPT-4 code completion, synthetic agents) by Greshake et al. at ACM AISec 2023 [94].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as Greshake et al. [94]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While LLMs primarily function as reactive prompt followers [63], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [64].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [63]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A person who finds the sentence 'ignore your previous instructions' on a web page does not treat it as a command addressed to them, whereas an LLMintegrated application may follow instructions injected into the content it retrieves [24].”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“attackers didn't need to compromise the tool itself, but only needed to influence authorised agents to abuse it [99], [81], [150], [42].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [99]
The cited source says what the claim attributes to it.
In the citing paper
“More importantly, tool misuse can cross security boundaries: invoking an unintended tool may expose sensitive data, mutate protected state, or execute actions beyond the user's intended privilege level, turning a reliability failure into a security incident (Greshake et al., 2023).”
From MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning (2026) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The attack needs only the ability to place content in an agent's context-already assumed by the indirect-prompt-injection literature (Greshake et al. 2023; Zhan et al. 2024)”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Greshake et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“12 patterns are derived from the public CyberSecEval 3 indirect-prompt-injection split [19,86] (rephrased to fit IoT-telemetry observation shapes); 6 patterns adapt indirect-prompt-injection vectors from Greshake et al. [94]; and 7 patterns are author-generated to cover IoT-specific surfaces not present in either source”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [94]
The cited source says what the claim attributes to it.
In the citing paper
“However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data.”
From Defeating Prompt Injections by Design (2026) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Consequently, if an application has access to confidential data or can take actions in the world, prompt injections can cause catastrophic harms such as exfiltrating user data (Greshake et al., 2023) or hijacking the LLM's actions (Toyer et al., 2024).”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown anecdotal evidence of prompt injection leading to product promotion for RAG models (Greshake et al., 2023).”
From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLMs are already known to be susceptible to indirect prompt injection attacks (Greshake et al., 2023; Liu et al., 2024)”
From WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (2025) · cites this paper as Greshake et al., 2023
In the citing paper
“Conversely, the integrity of Simplified Chinese LLM benchmarks like C-Eval [43] can be easily gamed by specific prompt structures or leverage training data tailored to the Simplified Chinese context, such as offering responses that align with or praise Chinese ideologies, to achieve artificially high scores without demonstrating genuine understanding or reasoning [61].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > LLM Safety.
“Perez et al. [50] showed that models, such as GPT-3, are vulnerable to prompt injection (PI).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > References
“50. Kevin Roose. 2023. A Conversation With Bing's Chatbot Left Me Deeply Unsettled. [Link].”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · cited 1×
“LLMs might make up facts (“hallucinate”), generate polarized content, or reproduce biases, hate speech, or stereotypes [39, 41, 47, 57, 61, 65, 78].”
Background
Research works printed but not linked: 5
- Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. 2023. Are aligned neural networks adversarially aligned? arXiv (2023).
- Clarence Chio and David Freeman. 2018. Machine learning and security. "O'Reilly Media, Inc.".
- Noah Goodman Jesse Mu, Xiang Lisa Li. 2023. Learning to Compress Prompts with Gist Tokens. arXiv (2023).
- Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv (2023).
- Shunyu Yao Jeffrey Zhao Dian Yu Izhak Shafran Karthik R Narasimhan and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR.
Cited by
Sorted by citations
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“While LLMs primarily function as reactive prompt followers [61], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [62].”
Digital deception: generative artificial intelligence in social engineering and phishing
Schmitt, M., & Flechais, I. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10973-2 · Cited by 198
“The expectation that individuals can scrutinize every email they receive is impractical and counterproductive, especially in today’s fast-paced digital environment, that comes with heavy cognitive workload (Kosch et al. 2023 ).”
Large Language Models for Cyber Security: A Systematic Literature Review
Xu, H., Wang, S., Li, N., Wang, K., Zhao, Y., Chen, K., Yu, T., Liu, Y., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769676 · Cited by 189
“Previous studies have identified vulnerabilities in LLMs like jailbreaking and malicious prompt injection, resulting in the exposure of model training data or sensitive user chat records [48, 70, 126].”
Friha, O., Ferrag, M. A., Kantarcı, B., Çakmak, B., Ozgun, A., & Ghoualmi‐Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 226
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
Visual Adversarial Examples Jailbreak Aligned Large Language Models
Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., & Mittal, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30150 · Cited by 150
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents
Zhan, Q., Fang, R., Panchal, H. S., & Kang, D. (2025) · Findings of the Association for Computational Linguistics: NAACL 2025 · doi:10.18653/v1/2025.findings-naacl.395 · Cited by 124
MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24188 · Cited by 418
“However, despite these precautionary steps, there remains a non-negligible risk from targeted attacks where such unde-sireable output are elicited, such as jailbreaks [27], [25] and prompt injections [21], [38].”
BackgroundA Hitchhiker’s Guide to Jailbreaking ChatGPT via Prompt Engineering
Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., & Wang, K. (2024) · doi:10.1145/3663530.3665021 · Cited by 122
Prompt Injection attack against LLM-integrated Applications
Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., Zheng, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05499 · Cited by 100
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
“Delayed evaluations mean emerging threats or vulnerabilities in LLMs may go undetected for extended periods, exposing systems to potential exploitation [54], [59].”
BackgroundLLM-based automatic short answer grading in undergraduate medical education
Grévisse, C. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06026-5 · Cited by 93
“Prompt injections can be defined as the “action of inserting malicious text with the goal of misalign-ing an LLM” [19] and manipulating its output [25] while circumventing content restrictions and filters [11].”
BackgroundCheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91
Safeguarding large language models: a survey
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11389-2 · Cited by 242
“Following this, the concept of Indirect Prompt Injection (IPI) [148] was introduced, referring to an uninvestigated attack vector where retrieved prompts can function as “arbitrary code”, thus compromising LLM-integrated applications.”
From Vulnerability to Defense: The Role of Large Language Models in Enhancing Cybersecurity
Kasri, W., Himeur, Y., Alkhazaleh, H. A., Tarapiah, S., Atalla, S., Mansoor, W., & Al-Ahmad, H. (2025) · Computation · doi:10.3390/computation13020030 · Cited by 87
Optimization-based Prompt Injection Attack to LLM-as-a-Judge
Shi, J., Yuan, Z., Liu, Y., Huang, Y., Zhou, P., Sun, L., & Gong, N. Z. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3690291 · Cited by 70
Do large language models have a legal duty to tell the truth?
Wachter, S., Mittelstadt, B., & Russell, C. (2024) · Royal Society Open Science · doi:10.1098/rsos.240197 · Cited by 104
Wang, Y., Pan, Y., Su, Z., Deng, Y., Zhao, Q., Du, L., Luan, T. H., Kang, J., & Niyato, D. T. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3576176 · Cited by 115
Charfeddine, M., Kammoun, H. M., Hamdaoui, B., & Guizani, M. (2024) · IEEE Access · doi:10.1109/access.2024.3367792 · Cited by 72
“Prompt injection [20] is a serious and frequently under-estimated problem in the field of artificial intelligence, particularly when LLMs are used.”
BackgroundMcIntosh, T. R., Susnjak, T., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3377445 · Cited by 62
“The study in [29] suggested that LLMs blur data-instruction boundaries, without mitigation strategies.”
BackgroundThe Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2026) · ACM Computing Surveys · doi:10.1145/3773080 · Cited by 58
“Despite their different origins and motives, technical vulnerabilities provide opportunities for attackers, enabling them to develop more sophisticated strategies and exposing LLM agents to various security and privacy risks.”
Toward Federated Large Language Models: Motivations, Methods, and Future Directions
Cheng, Y., Zhang, W., Zhang, Z., Zhang, C., Wang, S., & Mao, S. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3503680 · Cited by 79
“One of the common methods of this kind of attacks is prompt injection [291], [292], [293], [294], where the attacker gains control over the output of a language model, enabling them to dictate the content it generates.”
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Piet, J., Al-Rashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70879-4_6 · Cited by 124
ChatPhishDetector: Detecting Phishing Sites Using Large Language Models
Koide, T., Nakano, H., & Chiba, D. (2024) · IEEE Access · doi:10.1109/access.2024.3483905 · Cited by 73
“Methods that use LLMs, including our system, may be susceptible to prompt injection attacks [58], [59], where the original prompt is overwritten and malicious content is inserted into the response.”
MethodVulnerability Detection and Monitoring Using LLM
Akuthota, V., Kasula, R., Sumona, S. T., Mohiuddin, M., Reza, M. T., & Rahman, M. M. (2023) · IEEE International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE) · doi:10.1109/wiecon-ece60392.2023.10456393 · Cited by 46
“[6 ].”
Safeguarding human values: rethinking US law for generative AI’s societal impacts
Cheong, I., Caliskan, A., & Kohno, T. (2024) · AI and Ethics · doi:10.1007/s43681-024-00451-4 · Cited by 61
“However, existing alignment techniques are still relatively new and evolving, leaving AI systems vulnerable to various threats, including prompt injection attacks [15, 16].”
Jabir, R. M., Le, J., & Nguyen, C. (2025) · AI · doi:10.3390/ai6080174 · Cited by 73
Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
Lee, R. W., Jun, T. J., Lee, J.-M., Cho, S. I., Park, H., & Suh, J. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.49963 · Cited by 53
Organizational preparedness for the use of large language models in pathology informatics
Hart, S. N., Hoffman, N. G., Gershkovich, P., Christenson, C., McClintock, D. S., Miller, L. J., Jackups, R., Azimi, V., Spies, N. C., & Brodsky, V. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100338 · Cited by 38
Digital Deception: Generative Artificial Intelligence in Social Engineering and PhishingPreprint
Schmitt, M., & Fléchais, I. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4602790 · Cited by 37
Transformers: A Security Perspective
Latibari, B. S., Nazari, N., Chowdhury, M. A., Gubbi, K. I., Fang, C., Ghimire, S., Hosseini, E., Sayadi, H., Homayoun, H., Salehi, S., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3509372 · Cited by 43
“By skillfully injecting specific prompts into data streams encountered by LLMs during inference, adversaries can influence the model’s behavior, highlighting the vulnerability of LLM-Integrated Applications to strategic manipulations [2], [98], [196].”
Risks of AI scientists: prioritizing safeguarding over autonomy
Tang, X., Jin, Q., Zhu, K., Yuan, T., Zhang, Y., Zhou, W., Qu, M., Zhao, Y., Tang, J., Zhang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-63913-1 · Cited by 140
“In addition to steering already-capable models, it is also important to impose certain regulations on the developers and users of these highly capable models.”
InfluentialUnique Security and Privacy Threats of Large Language Models: A Comprehensive Survey
Wang, S., Zhu, T., Liu, B., Ding, M., Ye, D., Zhou, W., & Yu, P. S. (2025) · ACM Computing Surveys · doi:10.1145/3764113 · Cited by 96
“They found the multi-step jailbreak was more effective than single-step attacks at inducing ChatGPT to generate harmful content.”
A Survey on Trustworthy LLM Agents: Threats and Countermeasures
Yu, M., Meng, F., Zhou, X., Wang, S., Mao, J., Pang, L., Chen, T., Wang, K., Li, X., Zhang, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.09648 · Cited by 34
Attack Prompt Generation for Red Teaming and Defending Large Language Models
Deng, B., Wang, W., Feng, F., Deng, Y., Wang, Q., & He, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.143 · Cited by 106
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 82
A Comprehensive Review of Large Language Models in Cyber Security.
Güven, M. (2024) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.469 · Cited by 34
PromptShield: Deployable Detection for Prompt Injection Attacks
Jacob, D., Alzahrani, H., Hu, Z., Alomair, B., & Wagner, D. (2024) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3714393.3726501 · Cited by 64
“This is the basis of prompt injection attacks, where an adversary crafts a payload with the intent of subverting the functionality specified by 𝑝 [8, 14, 19].”
Kirchenbauer, J., & Barns, C. (2024) · doi:10.31219/osf.io/pv7r5 · Cited by 31
Attention, moral skill, and algorithmic recommendation
Schuster, N., & Lazar, S. (2024) · Philosophical Studies · doi:10.1007/s11098-023-02083-6 · Cited by 49
“For instance, they’re currently vulnerable to attacks like prompt injection, which circumvent their training and sub-vert their purposes to those of the hacker (Greshake et al., 2023).”
A Survey on Trustworthy LLM Agents: Threats and Countermeasures
Yu, M., Meng, F., Zhou, X., Wang, S., Mao, J., Pan, L., Chen, T., Wang, K., Li, X., Zhang, Y., et al. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736561 · Cited by 34
“…[91], Pri-vAgent [69], Evil Geniuses [88], PANDORA [16], AgentSmith [34], Tan et al. [85] Liu et al. [61], Perez et al. [73], Greshake et al. [33], Liu et al. [60], Shi et al. [80], Bagdasaryan et al. [6], Zhang et al. [122] Backdoor Zhou et al. [141], Yang et al. [108], DemonAgent [143],…”
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28
“AI Prompt injection (PI) remains the most widely discussed attack in the literature, where malicious instructions cause the model to deviate from intended behavior [57, 58].”
Far, S. M. T., & Feyzi, F. (2025) · International Journal of Information Security · doi:10.1007/s10207-025-00992-7 · Cited by 40
Suo, X. (2024) · AIP conference proceedings · doi:10.1063/5.0222987 · Cited by 80
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23
Security concerns for Large Language Models: A survey
Li, M. Q., & Fung, B. C. M. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104284 · Cited by 26
A Review of Large Language Models in Healthcare: Taxonomy, Threats, Vulnerabilities, and Framework
Hamid, R., & Brohi, S. N. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110161 · Cited by 29
Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks
Pasquini, D., Strohmeier, M., & Troncoso, C. (2024) · AISec@CCS · doi:10.1145/3689932.3694764 · Cited by 61
“Hereafter, we use the umbrella term LLM-integrated application [22] to refer to any software that employs an LLM as part of its functionality.”
InfluentialEsmradi, A., Yip, D. W., & Chan, C. F. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-97-1274-8_6 · Cited by 24
Esmradi, A., Yip, D. W., & Chan, C. F. (2023) · arXiv · doi:10.48550/arxiv.2312.10982 · Cited by 24
Generative AI Models in Time-Varying Biomedical Data: Scoping Review
He, R., Sarwal, V., Qiu, X., Zhuang, Y., Zhang, L., Liu, Y., & Chiang, J. N. (2024) · Journal of Medical Internet Research · doi:10.2196/59792 · Cited by 36
“Cyberattacks such as phishing and malware are common, but prompt injection attacks in GenAI models are emerging threats [220].”
JailGuard : A Universal Detection Framework for Prompt-based Attacks on LLM Systems
Zhang, X., Zhang, C., Li, T., Huang, Y., Jia, X., Hu, M., Zhang, J., Liu, Y., Ma, S., & Shen, C. Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724393 · Cited by 135
Prompt-to-SQL Injections in LLM-Integrated Web Applications: Risks and Defenses
Pedro, R., Coimbra, M. E., de Castro, D. A., Carreira, P., & Santos, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00007 · Cited by 41
“Currently, LLMs are used in a wide variety of applications, including tools to generate code [51], decompilers [52], document summarization [12], and more [53].”
InfluentialGökçimen, T., & Daş, B. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.03.030 · Cited by 25
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 25
“The exponential growth and widespread adoption of large pretrained models and Foundation Models have inadvertently introduced numerous security vulnerabilities [194].”
DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
Liu, Y., Jia, Y., Jia, J., Song, D., & Gong, N. Z. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00250 · Cited by 83
“We consider six heuristic-based prompt injection attacks, including Naive Attack [5], [12], [13], Context Ignoring [14], Escape Char-acter [12], Fake Completion [15], Combined Attacks [7], and Availability Attack [6].”
InfluentialRahman, M. A., Shahriar, H., Wu, F., & Cuzzocrea, A. (2024) · International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · doi:10.1109/aibthings63359.2024.10863664 · Cited by 28
“Another researcher also offered more diverse goals, for example persuading users to reveal information [3].”
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
Small, C. T., Vendrov, I., Durmus, E., Homaei, H., Barry, E., Cornebise, J., Suzman, T., Ganguli, D., & Megill, C. (2023) · arXiv · doi:10.48550/arxiv.2306.11932 · Cited by 20
“…et al., 2023; Hutchinson et al., 2020; Kurita et al., 2019; Basta et al., 2019; Abid et al., 2021b; Sap et al., 2020), prompt injection attacks (Greshake et al., 2023; Perez and Ribeiro, 2022), so called “hallucinations” (LLMs making things up when they don’t know something; Ji et al., 2023b),…”
BackgroundKey Security Risks in Prompt Engineering
Geroimenko, V. (2025) · SpringerBriefs in computer science · doi:10.1007/978-3-031-86206-9_5 · Cited by 19
A Systematic Comparison of Large Language Models Performance for Intrusion Detection
Bui, M.-T., Boffa, M., Valentim, R. V., Navarro, J. M., Chen, F., Bao, X., Houidi, Z. B., & Rossi, D. (2024) · Proceedings of the ACM on Networking · doi:10.1145/3696379 · Cited by 19
Easy-read and large language models: on the ethical dimensions of LLM-based text simplification
Freyer, N., Kempt, H., & Klöser, L. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09792-4 · Cited by 27
“On the other hand, the problem of prompt injections is challenging the safety of contemporary LLMs and constitutes an active field of research (Greshake et al., 2023).”
Hamid, O. H. (2024) · IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) · doi:10.1109/cogsima61085.2024.10553755 · Cited by 29
“While the directives imparted to the remain concealed from users, a shrewd manipulation tactic termed ‘prompt injection’ [15] can occasionally unveil these covert instructions [5].”
BackgroundDefense against prompt injection attack by leveraging attack techniques
Chen, Y., Li, H., Zheng, Z., Wu, D., Song, Y., & Hooi, B. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.897 · Cited by 19
A CIA Triad-Based Taxonomy of Prompt Attacks on Large Language Models
Jones, N., Whaiduzzaman, M., Jan, T., Adel, A., Alazab, A., & Alkreisat, A. (2025) · Future Internet · doi:10.3390/fi17030113 · Cited by 23
“Additionally, Chen et al. [8] focused on indirect attacks, revealing how seemingly benign inputs can exploit LLMs’ assumptions about input context, leading to unintended behaviors.”
SecureNet: A Comparative Study of DeBERTa and Large Language Models for Phishing Detection
Mahendru, S., & Pandit, T. (2024) · IEEE International Conference on Big Data and Artificial Intelligence (BDAI) · doi:10.1109/bdai62182.2024.10692765 · Cited by 34
Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
Lee, D., & Tiwari, M. (2024) · arXiv · doi:10.48550/arxiv.2410.07283 · Cited by 228
Defending Against Prompt Injection With a Few DefensiveTokens
Chen, S., Wang, Y.-M., Carlini, N., Sitawarin, C., & Wagner, D. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762982 · Cited by 62
Large Language Models Are Unreliable for Cyber Threat Intelligence
Mezzi, E., Massacci, F., & Tuma, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00627-1_17 · Cited by 37
“A current focus of LLMs security research [28] is to implement adversarial attacks on LLMs such as prompt injection [29] and data poisoning [30] whose goal is to alter LLMs’ behaviour by deceiving them into making incorrect predictions.”
Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications
Zhang, Q., Zeng, B., Zhou, C., Go, G., Shi, H., & Jiang, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663786 · Cited by 33
Traceback of Poisoning Attacks to Retrieval-Augmented Generation
Zhang, B., Xin, H., Fang, M., Liu, Z., Yi, B., Li, T., & Liu, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714756 · Cited by 18
“Although existing defenses can mitigate the impact of poisoning attacks on RAG systems to some extent, they remain vulnerable to advanced attacks, where the attacker craft sophisticated strategies to bypass current safeguards [7, 11, 12, 17, 25, 29, 30, 36, 47].”
Adaptive Compressed-Based Privacy-Preserving Large Language Model for Sensitive Healthcare
Gong, X., Gao, J., Sun, S. M., Zhong, Z., Shi, Y., Zeng, H., & Yang, K. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3558935 · Cited by 17
“An indirect prompt injection attack may intentionally compromise stored personal information [49].”
Text-Based Prompt Injection Attack Using Mathematical Functions in Modern Large Language Models
Kwon, H., & Pak, W. (2024) · Electronics · doi:10.3390/electronics13245008 · Cited by 16
“The content [23].”
InfluentialPoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
Zou, W., Geng, R., Wang, B., & Jia, J. (2024) · arXiv · doi:10.48550/arxiv.2402.07867 · Cited by 16
LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks
Happe, A., Kaplan, A., & Cito, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10758-3 · Cited by 69
Large language models in global health
Ong, J. C. L., Ning, Y., Yang, R., Bitterman, D. S., Liu, X., Tham, Y. C., Collins, P. G. S., Jiménez, M., Mateen, B. A., Amissah-Arthur, K. N., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00024-7 · Cited by 16
“Moreover, as LLMs are integrated into health information systems, the risks of hacking, data interception and malicious code injection are increasing—particularly in the case of open-weight LLMs that lack ongoing security auditing 61 .”
Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection
Rahman, M., Shahriar, H., Francia, G., Wu, F., Cuzzocrea, A., Rahman, M., Faruk, M. J. H., & Ahamed, S. I. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00134 · Cited by 21
HoneyLLM: Enabling Shell Honeypots with Large Language Models
Guan, C., Cao, G., & Zhu, S. (2024) · IEEE Conference on Communications and Network Security · doi:10.1109/cns62487.2024.10735663 · Cited by 22
“A prompt injection attack [28, 29, 30, 31] is a technique used to manipulate or exploit LLMs by inserting specific instructions or content into the input prompt.”
MethodDamacena Duarte, J., Cândido, G. D., De Britto Filho, J. R. A., Souza Neto, J., Da Costa, E. J., Da Costa, J. P. J., & Peotta De Melo, L. (2026) · IEEE Access · doi:10.1109/access.2026.3656849 · Cited by 15
“As highlighted by Greshake et al. [7], contemporary development has moved beyond static LLM applications toward agentic frameworks in which the model autonomously decomposes tasks, invokes external tools or APIs, and utilizes memory streams to reflect and adapt behaviour over time.”
InfluentialRisks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey
Sakib, M. N., Islam, M. A., Pathak, R., & Arifin, M. M. (2024) · International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · doi:10.1109/aibthings63359.2024.10863356 · Cited by 13
“For instance, Bing Chat, operating on the GPT-4 model, has exhibited vulnerability to this attack [16].”
Defeating Prompt Injections by Design
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2026) · 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml68715.2026.00040 · Cited by 15
“This exposes them to prompt injection attacks [10, 11, 12] when data or instructions come from untrusted sources (e.g., from compromised users, from tool call outputs, or from a web page).”
ContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · arXiv · doi:10.48550/arxiv.2409.00729 · Cited by 16
Abtahi, F., Seoane, F., Pau, I., & Vega-Barbas, M. (2026) · Journal of Medical Internet Research · doi:10.2196/87969 · Cited by 32
“Cascading impacts in agentic systems, where a flawed decision may propagate across multiple dependent clinical processes [12,13,17,38].”
Detecting LLM-generated peer reviews
Rao, V. S., Kumar, A., Lakkaraju, H., & Shah, N. B. (2025) · PLOS One · doi:10.1371/journal.pone.0331871 · Cited by 14
“We take advantage of this by embedding hidden instructions or prompts [GAM + 23, YXZ + 23] directly into the manuscript file.”
DeeBBAA: A Benchmark Deep Black-Box Adversarial Attack Against Cyber–Physical Power Systems
Bhattacharjee, A., Bai, G., Tushar, W., Verma, A., Mishra, S., & Saha, T. K. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3454257 · Cited by 15
Self-citationYip, D. W., Esmradi, A., & Chan, C. F. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487667 · Cited by 19
“Existing research into prompt injection attacks mainly focuses on the different attack techniques such as payload splitting [10], obfuscation [11], ignore previous prompt [12, 13], character role play prompt [14], creative dialogue [15], privacy leakage [16], and indirect prompt injection [17, 18].”
BackgroundThe Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024) · arXiv · doi:10.48550/arxiv.2404.13208 · Cited by 14
“Consequently, if an application has access to confidential data or can take actions in the world, prompt injections can cause catastrophic harms such as exfiltrating user data (Greshake et al., 2023) or hijacking the LLM’s actions (Toyer et al., 2024).”
BackgroundLarge Language Models for Cybersecurity Intelligence, Threat Hunting, and Decision Support
Ren, S., & Chen, S. (2025) · Jisuanji shenghuojia. · doi:10.54097/7ysr5k17 · Cited by 16
Securing RAG: A Risk Assessment and Mitigation Framework
Ammann, L., Ott, S., Landolt, C. R., & Lehmann, M. P. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00024 · Cited by 24
“The OWASP 2025 Top 10 Risks for LLMs and Gen AI Apps [11] include specific risks associated with RAG systems, including prompt injection [17], data leakage [8], embedding inversion [18], data poisoning [19], cross-context information conflicts [20], and unintentional behavior alterations [21],…”
A Survey on Failure Analysis and Fault Injection in AI Systems
Yu, G., Tan, G., Huang, H., Zhang, Z., Chen, P., Natella, R., Zheng, Z., & Lyu, M. R.-T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3732777 · Cited by 38
TrustRAG: Enhancing Robustness and Trustworthiness in RAGPreprint
Zhou, H., Zhan, Z., Li, Z., Haddadi, H., & Yılmaz, E. (2025) · Qeios · doi:10.32388/z4dwhq · Cited by 16
MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation
Wang, G., Li, Y., Liu, Y., Deng, G., Li, T., Xu, G., Liu, Y., Wang, H., & Wang, K. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652297 · Cited by 16
“A novel threat, “indirect prompt injections”, allows applications to unwittingly execute commands from external API calls, such as SQL query results [20].”
BackgroundDeep learning for nano-photonic materials – The solution to everything!?
Wiecha, P. R. (2023) · Current Opinion in Solid State and Materials Science · doi:10.1016/j.cossms.2023.101129 · Cited by 14
Malla: Demystifying Real-world Large Language Model Integrated Malicious Services
Lin, Z., Cui, J., Liao, X., & Wang, X. (2024) · arXiv · doi:10.48550/arxiv.2401.03315 · Cited by 13
“Greshake et al. [96] discussed the threats of indirect prompt injection, which placed the Prompt-Inject framework into indirect data sources that are retrieved and used by an LLM to generate a response.”
MethodGet My Drift? Catching LLM Task Drift with Activation Deltas
Abdelnabi, S., Fay, A., Cherubin, G., Salem, A., Fritz, M., & Paverd, A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00011 · Cited by 67
“While this integration enhances utility, it also introduces new security risks; attackers can abuse these external inputs to inject malicious commands, potentially hijacking the LLM’s output for other users and compromising security boundaries [1].”
Self-citationDesigning LLM Chains by Adapting Techniques from Crowdsourcing Workflows
Grunde-McLaughlin, M., Lam, M. S., Krishna, R., Weld, D. S., & Heer, J. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3716134 · Cited by 39
“Higher quality subtask responses”
Sociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 28
A Modular Generative Honeypot Shell
Johnson, S., Hassing, R., Pijpker, J., & Loves, R. (2024) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr61664.2024.10679411 · Cited by 16
“[6] patterns to allow easy extension and/or substitution of application components, and incorporates advanced mechanisms for mitigation of prompt injection attacks [7], [8].”
BackgroundA Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systems
Zhang, W., Kong, X., Dewitt, C., Bräunl, T., & Hong, J. B. (2024) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew63542.2024.00103 · Cited by 27
“Greshake et al. [16] examined Disrupting Availability, where attackers conduct denial of service (DoS) attacks by overwhelming the system’s resources, rendering it unable to perform its intended functions and affecting its availability. b) Counteracts using Secure Prompting Secure prompting…”
Background
Metadata
- Authors
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M.
- Affiliations
- Saarland University (DE)
- Helmholtz Center for Information Security (DE)
- Université franco-allemande (DE)
- Venue
- Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security
- Publisher
- ACM
- Published
- 2023-11-26
- Pages
- 79-90
- OpenAlex
- W4388886073
- Semantic Scholar
- 705e49afd92130f2bc1e0d4d0b1f6cb14e88803f
- Cited by
- 667
- References
- 11
- Citations by year
- 2026: 416 · 2025: 153 · 2024: 86 · 2023: 12
- FWCI
- 117.91
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Workshop on Artificial Intelligence and Security (AISec) (publishedVersion)
- Funders
- European Union (101070617)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Keywords
- indirect prompt injection, large language models, prompt injection, data theft, arbitrary code execution, attack taxonomy, Bing Chat, security risk mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
