The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 22 of the 36 sources its claims cite were checked for support.
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Open accessAbstract
Large Language Models (LLMs) are increasingly being integrated into applications, with versatile functionalities that can be easily modulated via natural language prompts. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time. We derive a comprehensive taxonomy from a computer security perspective to broadly investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We then demonstrate the practical viability of our attacks against both real-world systems, such as Bing Chat and code-completion engines, and GPT-4 synthetic applications. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing reliance on LLMs, effective mitigations of these emerging threats are lacking. By raising awareness of these vulnerabilities, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users from potential attacks.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces indirect prompt injection, where adversaries place prompts in data that LLM-integrated applications retrieve, allowing remote manipulation without directly prompting the model. It develops a security taxonomy and demonstrates practical attacks against Bing Chat, GitHub Copilot, and synthetic GPT-4/text-davinci-003 applications, covering data theft, fraud, malware, intrusion, content manipulation, and availability disruption.
Key findings
- Retrieved content can function like instructions, letting third-party data steer model behavior; the authors report that all described proof-of-concept demonstrations triggered the intended behavior.
- Attacks can persuade users to reveal personal information, exfiltrate data through side effects such as URL retrieval, distribute phishing or malware links, and turn prompts into self-propagating “AI malware” that reads address books and forwards injections by email.
- Intrusion demonstrations include remote control by fetching new commands from an attacker-controlled server, persistence through memory reinfection, and contamination of code-completion suggestions; content-manipulation demonstrations include wrong summaries, source blocking, disinformation, and undisclosed advertisements.
- Hidden-injection methods include multi-stage payload retrieval and a Base64-encoded prompt that worked on Bing Chat; the model also sometimes issued follow-up search queries that reinforced the injected prompt.
Methods
- The authors build a threat model and taxonomy from a computer security perspective, classifying injection delivery as passive retrieval, active delivery, user-driven, or hidden, and mapping threats to affected parties.
- Synthetic applications used OpenAI APIs: text-davinci-003 with LangChain and ReAct prompting, and GPT-4 in chat format, with mock tools for search, viewing pages, retrieving URLs, reading/sending email, reading an address book, and memory; all interfaces returned prepared content, no real systems were contacted, and attacks ran at sampling temperature 0.
- Real-world black-box tests targeted Bing Chat, including its Edge sidebar reading local HTML files, and GitHub Copilot/Codex for code-completion manipulation; prompts were manually written, and local HTML was used to avoid public injections.
Limitations
- The paper states that attacks were tested on local HTML files rather than public in-the-wild injections, that Microsoft 365 Copilot and ChatGPT plugins were inaccessible, that success rates are hard to quantify in dynamic interactive sessions, that believability was assessed qualitatively, and that exact reproducibility is difficult in black-box systems.
- The text also makes evident that code-completion injections were highly context-sensitive and lost efficacy in larger packages or projects, and that some availability attacks, such as inhibiting API calls, succeeded only inconsistently.
- The authors note that user studies are needed to quantify deception potential across different attacks and setups.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
50 Own interpretation · 28 Own result · 20 No basis given · 9 Couldn’t confirm · 7 Sources not found · 6 Partly supported · 5 No full text of the sources · 3 Supported · 3 Own method · 1 Not checked yet · accuracy not yet measured
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“They also produce plausible utterances, even wrong ones, in a confident and authoritative tone [43].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Could not verifycites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
The Experiments and Results section describes GPT-4’s text generation, and §4.2 notes deviations from a requested extraction format. Neither passage addresses whether ChatGPT or GPT-4 presents wrong utterances in a confident, authoritative tone. Searches for that phrasing and for confident incorrect answers found no supporting passage in our copy. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target [21,43] (that was either given in the prompt or, importantly, the model acquired during the chat session).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Supportedcites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. source
Section 5.2 reports that the authors prompted LLMs to generate investment-scam emails tailored to a target’s demographics and personal situation. Human ratings found ChatGPT’s emails both personalized and convincing, supporting the attributed claim. (LLM, conf=0.93)
- Could not verifycites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
The indexed copy’s Experiments and Results and §4.1 discuss prompting GPT-4 for spatial-relation extraction, not producing personalized scams. Searches for scams, fraud, phishing, and target-specific knowledge found no passage addressing the claim. The supplied DOI resolves in our copy to a spatial-ontology paper rather than the named GPT-4 Technical Report. (LLM, conf=0.76)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Different social engineering and persuasion tricks can be automated with LLMs [20] (e.g., claiming authority as an official service, claiming repercussions for not complying such as a loss of the service, distracting the user by implying the requested action is urgent, offering a limited-time kind gesture, etc.).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
What the sources say
- Supportedcites [20] 20. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tat- sunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arXiv (2023). source
Section 5.1 reports that LLMs generated emails for several phishing and scam scenarios, including a fake-ticket scam and a purported FEMA funds scam. Section 5.2 reports that they also generated personalized scam emails, supporting the claim that different social-engineering approaches can be automated with LLMs. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“As the model itself can generate API calls to other applications [70], one way to interfere with this is to instruct the model not to call the API (e.g., the search), which often succeeded, although not consistently.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
What the sources say
- Could not verifycites [70] 70. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv (2023). sourceExperimental — accuracy not yet measured
The Introduction and Graph Connectivity sections discuss hallucinated answers and follow-up verification, not models generating API calls to other applications. The Limitations section mentions APIs used to access ChatGPT and GPT-4, but does not address model-generated API calls (searched: “API calls,” “external tools,” and “search engine”). (LLM, conf=0.74)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Even without actively specifying any social engineering techniques, the model's output could mirror some standard scam persuasion techniques [20].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.2 Fraud .
What the sources say
- Partly supportedcites [20] 20. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tat- sunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arXiv (2023). sourceExperimental — accuracy not yet measured
Section 5.1 reports that LLMs generated convincing scam emails, and Appendix C gives an example using limited availability and reassurance—recognizable scam persuasion techniques. The copy does not establish that these techniques appeared without being requested; a prompt shown in Section 3.4 explicitly asks for urgency. (LLM, conf=0.82)
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More dangerously, as we show in our experiments, they could be more stealthy by indirectly disrupting the service via corrupting the search queries or results (i.e., input and output of APIs), forcing the model to hallucinate.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that the model can be reinfected by looking at its memories.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
boosteddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is also possible to prompt the model to output adversarially-chosen disinformation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When embedded within larger packages or projects, the efficacy of our injections was significantly reduced.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, we found that even without specifying exact instructions, the model usually generated answers that resembled these persuasion strategiesfoot_0 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, if the user asks it to read the last conversation from memory, it re-poisons itself.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This attack shows how a small injection in a large section of regular content can trigger the LLM to fetch another, potentially bigger, payload.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Multi-stage Exploit.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Alternatively, we prompted the model to generate less helpful content, which resulted in very brief answers or refusal to answer.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the synthetic application, we demonstrate how a poisoned model may spread the injection (see high-level overview in Figure 6).The application in this scenario can read emails, compose emails, look into the user's address book, and send emails.In this attack, the model spreads the injection to other models that may be reading those inbound messages.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Spreading Injections (AI Malware).
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to the previous attack, the model also wrongly summarizes search results 4 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“one way to interfere with this is to instruct the model not to call the API (e.g., the search), which often succeeded, although not consistently.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All demonstrations described next were successful (i.e., they triggered the intended behavior).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2 Demonstrations of Threats
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that any prompts/instructions written on a page (while being invisible to the user) can be effectively injected and affect the model.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.1 Injection Methods
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Observation #3: Models might issue follow-up API calls (e.g., search queries) that were affected by and reinforce the injected prompt.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Developing the prompts that execute our attacks turned out to be rather simple, often working as intended on the very first attempt at writing them, further highlighting the dangerous feasibility and easy-to-launch nature of attacks.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Attack Prompts.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Bing Chat, we observed that the model can retain the injection throughout the conversation (e.g., if the user did not disclose their name from the first attempt, it can ask about it again in subsequent questions).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Observation #1: Attacks could only need to outline the goal, which models might autonomously implement.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our attack examples, when a user opens the”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In one test session, the Chat cited an article (reporting that Twitter has removed the”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The model in this attack often times out without answering any requests.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“2) in most cases, the model retains the injection consistently throughout the conversation session.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2 Demonstrations of Threats
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The prompt instructs the model to corrupt the extracted query before searching with it, leading to useless search results.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The model itself generates the search query (i.e., the arguments to APIs).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observation #2: When prompted with marginally related context (e.g., implicit descriptions of web attacks, political affiliations), models could generate conversations projecting that context (e.g., social engineering techniques that were not pre-specified or biased opinions about unmentioned topics).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Chat often returned the search results as links without any text.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Bing Chat cannot repeat the <| endoftext|> token or finish sentences when this token appears in the middle.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Chat's responses were consistent with the personas described across different political topics and throughout the chat session.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“By retrieving that information, the LLM is compromised with an injection hidden as a comment in the Markdown of the Wikipedia page.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Multi-stage Exploit.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the model can sometimes hallucinate answers based on the original question while citing the unrelated search result.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This can be seen as a cascaded indirect prompt injection; surprisingly, the attack worked as expected.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Encoded Injections.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“GPT-4 was trained with intervention to reduce jailbreaks, such as safety-relevant RLHF-our work and several other jailbreak attacks [66] show that it is possible to adversarially prompt the model even in real-world applications.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not checked — no full text of the sourcecites [66] 66. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS. source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Attackers could aim to hide injections by encoding the prompts.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Encoded Injections.
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models integrated into system infrastructures could constitute backdoors for attackers to gain unauthorized privilege escalation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“In addition to the chat interface, Microsoft Edge has a feature to enable Bing Chat in a sidebar [30].If enabled by the user, the current page's content can be read by the model such that users can ask questions related to the page's content.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Not matchedcites [30] 30. Microsoft. 2023. Bringing the power of AI to Windows 11 -unlocking a new era of productivity for customers and developers with Windows Copilot and Dev Home. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They can be prompted to provide adversarially-chosen or arbitrarily wrong summaries of documents (e.g., of other parties), emails (e.g., from other senders), or search queries.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prompts could be used to launch availability or Denial-of-Service (DoS) attacks.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs now constitute an intermediate, prone to manipulation, layer between the user and the requested information.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regularly repeating this cycle could obtain a remotely accessible backdoor into the model.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the functionality of the LLM in its exact primary task can be subject to manipulation as well.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Read operations (such as retrieving URLs) could be exploited to send information to the attacker automatically.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This can render the”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Code completion engines that use LLMs deploy heuristics to determine which code snippets are included in the context [57].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
What the sources say
- Not checked — no full text of the sourcecites [57] 57. Jonas Thiergart, Stefan Huber, and Thomas Übellacker. 2021. Understanding emails and drafting responses-An approach using GPT-3. arXiv (2021). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While importing packages already provides an attacker with the ability to execute code, the additional threat here stems from the fact that these injections (i.e., commented code) can currently only be detected through manual code review.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models may enable new distribution vectors of different web attacks (Figure 5).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.2 Fraud .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to phishing, LLMs could be exploited to trick victims into visiting malicious web pages that lead to, e.g., drive-by downloads.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This attack can affect both the user and the model.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In contrast to static one-shot malicious text generation, quantifying our attacks' success rate can be challenging in the setup of dynamically evolving and interactive chat sessions with users [25].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not matchedcites [25] 25. Kif Leswing. 2023. Microsoft's Bing A.I. made several factual errors in last week's launch demo. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“Even with current models, there is recent evidence that users' judgment might be affected despite being aware that they are advised by a chatbot [23].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not matchedcites [23] 23. LangChain. 2023. LangChain library for composing and integrating LLMs into applications. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
- Probably not a substantial claim. 51
- It may state two separate claims as one. 45
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT and GPT-4 can produce convincing personalized content and interactions with users [10].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Partly supportedcites [10] 10. Matthew Burtell and Thomas Woodside. 2023. Artificial Influence: An Analysis Of AI-Driven Persuasion. arXiv (2023). sourceExperimental — accuracy not yet measured
Section 2.1 describes GPT-4 producing plausible reasons and ChatGPT responding to preceding dialogue, sometimes confidently. But the paper does not establish that both models produce personalized content or interactions: section 3.2 discusses personalized persuasion largely as a future possibility, and section 2.1 says the authors do not know whether GPT-4 exhibits the particular conversational behavior described for ChatGPT. Page numbers are unavailable in our copy. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Retrieval-augmented models now cite their sources, possibly leading to “overreliance” [43] on their factuality by users.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Could not verifycites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
The indexed copy for the supplied DOI is a paper on spatial relation extraction, not the GPT-4 Technical Report. Its Experiments and Results section discusses GPT-4's extraction performance, but searches for source-citing retrieval-augmented models and user overreliance found no passage addressing the claim. (LLM, conf=0.75)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Previous work has shown that LLMs can produce convincing scams such as phishing emails [21].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
What the sources say
- Supportedcites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. source
Section 5.1 describes LLM-generated phishing and scam emails evaluated by human labelers for convincingness. It concludes that the models enable convincing scams, directly supporting the cited claim. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent evidence suggests that models might infer and act upon intentions and goals [2,43,55], as a result of training or when prompted with a persona [15].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Supportedcites [2] 2. Jacob Andreas. 2022. Language models as agent models. In Findings of EMNLP. source
The Introduction says that models can infer partial representations of beliefs, desires, and intentions during next-word prediction and that these representations influence generated text. Section 6 describes a prompt specifying a person’s goal of giving truthful answers and reports that it changed the models’ answers. Together, these passages support the training- and persona-prompting possibilities attributed to this source. (LLM, conf=0.90)
- Partly supportedcites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceExperimental — accuracy not yet measured
Section 4.1 (page unavailable) says models can use pre-existing knowledge to infer a desired task from a prompt and generate a relevant response. Section 3.3.3 (page unavailable) also says prompts guide responses toward their objectives. These passages support prompt-guided behavior generally, but they do not address persona prompts or show that models infer and act upon intentions and goals as a result of training. (LLM, conf=0.76)
- Partly supportedcites [55] 55. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. sourceExperimental — accuracy not yet measured
The Introduction (p. 2) describes training a reward model to predict human preferences and a policy to optimize its score, supporting the training-related part of the claim. The paper does not show that models infer intentions or goals, or discuss prompting them with a persona; the Discussion (p. 9) instead says identifying whether a system aligns with a designer’s intentions is challenging. (LLM, conf=0.82)
- Partly supportedcites [15] 15. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in ChatGPT: Analyzing Persona-assigned Language Models. arXiv (2023). sourceExperimental — accuracy not yet measured
Section 2.1 describes prompting ChatGPT to speak like an assigned persona, and the Abstract reports that persona assignment changes its outputs. The paper does not establish that the model infers or acts upon intentions and goals; its evidence concerns persona-conditioned language and toxicity. (LLM, conf=0.82)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent work [46] shows that LLMs, when prompted with a defined context, can generate believable non-scripted behaviors that are consistent with this context.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
What the sources say
- Partly supportedcites [46] 46. Roma Patel and Ellie Pavlick. 2021. "Was it "stated" or was it "claimed"?: How linguistic bias affects generative language models. In EMNLP. sourceExperimental — accuracy not yet measured
The paper reports that GPT-2 and GPT-3 generate text in response to prompts and that the generations’ high-level topics do not drift from the prompts (section 3.5, page not provided). It does not assess whether the models generate believable, non-scripted behaviors; its findings concern generated text and linguistic framing. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Indirect prompting could be leveraged to leak users' chat sessions [52] or exfiltrate users' data (e.g., credentials, personal information).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
What the sources say
- Could not verifycites [52] 52. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv (2023). sourceExperimental — accuracy not yet measured
The cited paper’s Sampling API Calls section describes generating and executing tool calls, not indirect prompting used to leak chat sessions. Searches for chat-session leakage, prompt injection, and data exfiltration found no passage in our copy addressing that claim. Page numbers were not available in the indexed passages. (LLM, conf=0.76)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“ChatGPT hallucinated the name of a law professor when asked about sexual harassment [59] and falsely claimed that an Australian mayor had spent time in prison [22].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
What the sources say
- Not matchedcites [59] 59. James Vincent. 2023. Google and Microsoft's chatbots are already citing one another in a misinformation shitshow. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [22] 22. Sebastian Krügel, Andreas Ostermaier, and Matthias Uhl. 2023. ChatGPT's inconsistent moral advice influences users' judgment. Scientific Reports 13, 1 (2023), 4569. sourceExperimental — accuracy not yet measured
The citation to this source is attached to the claim about an Australian mayor, not the separate claim about a law professor. The Introduction says generally that ChatGPT sometimes provides false information, but the passages found when searching for the mayor and prison claim do not mention that incident; the Methods and Results instead concern moral advice in trolley dilemmas. Our copy therefore does not verify the specific claim. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“For textdavinci-003, we use ReAct prompting [70], and we found that GPT-4 can work well without ReAct (by only describing the tools and giving direct instructions).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Could not verifycites [70] 70. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv (2023). sourceExperimental — accuracy not yet measured
The Inference Setup (§3.2; page unavailable) describes zero-shot direct prompts for GPT-4, and Engineering Better Prompts (§4.1; page unavailable) tests a step-by-step prompt. Neither passage addresses ReAct, tool descriptions, or whether GPT-4 works well with tools without ReAct. The citation in the printed sentence is attached to “ReAct prompting,” while the GPT-4 observation is presented as the citing authors’ own finding. (LLM, conf=0.77)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“When prompted with biographies of people with particular views (e.g., politically liberal), RLHF models tend to repeat the user's views, posing the dangers of polarization and creating echo chambers [49].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
What the sources say
- Could not verifycites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceExperimental — accuracy not yet measured
The Related work section (p. 2) discusses human-feedback fine-tuning and social bias, while the framework and results sections (pp. 3, 5) concern adversarial prompt attacks. Searches for biographies, political views, repetition of users’ views, polarization, and echo chambers found no passage addressing the claim. (LLM, conf=0.75)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Enabling these models to write to some form of persistent storage is currently already investigated in many plugins and systems [42], including Bing Chat [64].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
What the sources say
- Could not verifycites [42] 42. OpenAI. 2023. GPT-4 Technical Report. arXiv (2023). sourceExperimental — accuracy not yet measured
The closest passage, “Interactions with other systems” (p. 56), describes GPT-4 augmented with tools, including a literature-search tool that embeds text in a vector database. It does not say that models write to persistent storage across many plugins and systems. Searches for persistent storage, plugins, and writing to external memory found no passage establishing that point. Bing Chat is attached to a separate citation in the citing sentence. (LLM, conf=0.70)
- Not matchedcites [64] 64. Tom Warren. 2023. These are Microsoft's Bing AI secret rules and why it says it's named Sydney. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Perez et al. [49] evaluated”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
What the sources say
- Could not verifycites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceExperimental — accuracy not yet measured
The Experiments section (p. 3) says Perez et al. evaluated goal hijacking and prompt leaking, not sycophancy. Related Work (p. 2) mentions a model fine-tuned through human feedback, but our copy does not describe evaluating whether RLHF models tailor responses to human evaluators (searched: sycophancy; RLHF and human evaluators). (LLM, conf=0.74)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Even if applied, it remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding [21], which future models might further enable.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Supportedcites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. source
Section 3.3.1 reports that simple obfuscation bypassed filters, but says more advanced forms, including encoding, are conjectured to be possible and remain for future work. That supports the citing sentence’s uncertainty about stronger obfuscation or encoding. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirical evidence of inverse scaling in RLHF models was also reported [49].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Partly supportedcites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceExperimental — accuracy not yet measured
The Results and discussion (p. 4) report that the most capable OpenAI API model was the most vulnerable to prompt injection, which the authors say suggests inverse scaling. The Experiments section (p. 3) identifies the models tested, but our copy does not establish that the inverse-scaling comparison was specifically among RLHF models. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the model may issue follow-up search queries to find evidence supporting the injected prompt, mis-summarize search results, or be selective in the sources displayed to the user.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Via Indirect Prompt Injection, we design Proof-of-Concept attack demonstrations that can lead to full compromise of the model at inference time analogous to traditional security principles.This entails remote control of the model, persistent compromise, theft of data, and denial of service.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A key observation is that attackers might not need to pre-program or script the details of how attacks are performed.Just by defining the goal, models might autonomously initiate conversations, imitate persuasion techniques, extend the context defined in the prompt, or issue actions (e.g., search queries) to fulfill the goal.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Bing Chat usually prints the search keywords it is performing.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although we did not specify in the prompt how to convince the user, Bing Chat incentivized us to follow the link by saying: “Hurry up, this offer is valid only for a limited time.Don't miss this chance to treat yourself with some amazing products from Amazon”.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.2 Fraud .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This injection instructs the LLM to search for a particular keyword, delivering another attackercontrolled injection, which can be arbitrarily long and conspicuous.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Multi-stage Exploit.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can also follow long instructions and use the context of the conversation to augment its persuasion techniques.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is worth mentioning that the Chat issued search queries during the conversation to support the prompt 3 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT can create hyperlinks from the users' input (i.e., the malicious indirect prompt), which attackers could use to add legitimacy and hide the malicious URL.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Current models, such as Bing Chat, can create markdown links, which can be used to hide a suspicious link with an innocentlooking string.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The prompt does not need to be long by stacking multiple instructions but can be a loop of instructions.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.6 Availability .
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After “reprogramming” the agent with this new injection, the model will fetch the new commands from the mockup attacker's server and respond to the user with a pirate accent : Arrr, me hearty!.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found this form of injection possible but very sensitive to context.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Indirect prompting might be exploited to elicit advertisements that are not disclosed as such 5 .”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This task is both time-consuming and can lead to hallucinations as well.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This yields a similarly-looking query.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Disrupting Search Queries.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize some high-level remarks; 1) instructions injected indirectly can successfully steer the model; the data and instruction modalities are not disentangled, and 2) in most cases, the model retains the injection consistently throughout the conversation session.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2 Demonstrations of Threats
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our Base64 encoding experiment, we needed to explicitly provide instructions for the model to decode the prompt.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our attacks succeed on Bing Chat, which seems to employ additional filtering.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We qualitatively observe the huge improvements of recent LLMs in following complex instructions and persuasion over previous models.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Search chatbots may inaccurately summarize the cited document according to the prompt, or find sources that support the non-factual prompt, all while sounding plausible and grounded in these citations.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#4: Models can currently act as a vulnerable intermediate layer between users and information, which users might nevertheless overrely on.I.e., the model's functionality itself can be attacked.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“However, it involves components for query generation based on users' conversations, search engine integration, answers generation, and citation integration [32].”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Not matchedcites [32] 32. Microsoft. 2023. Confirmed: the new Bing runs on OpenAI's GPT-4. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Advanced AI systems add new layers of threat: Their capabilities to adapt to minimal instructions and autonomously advance the attacker's goals make them a potent tool for adversaries to achieve, e.g., disinformation dissemination and user manipulation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#1: Retrieval blurs the line between data and instructions and enables remote prompt injections.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Key Message#1: Retrieval blurs the line between data and instructions and enables remote prompt injections.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, with current LLMs, it could be done by simply instructing the model to do a time-intensive task in the background.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#2: With models' malleable functionality, increased autonomy, and broad capabilities, mapping all known cybersecurity threats to the new integrated LLMs ecosystem is conceivable.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Key Message#2:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message#3: LLMs are vulnerable gatekeepers to systems infrastructure, a risk that can only be amplified with autonomous systems.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“The Copilot uses OpenAI Codex [44] to suggest lines or functions based on the current context.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Github Copilot.
What the sources say
- Not checked — no full text of the sourcecites [44] 44. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key Message #5: As LLMs themselves are in charge of when and how to issue other API calls and process their outputs, the input and output operations are vulnerable to manipulation and sabotage.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Key Message #5:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, we show that adversaries can now remotely affect other users' systems by strategically injecting the prompts into data likely to be retrieved at inference time.If retrieved and ingested, these prompts can indirectly control the model (see Figure 1).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Bing Chat currently runs on the GPT-4 model [33] with customization to the search task.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
What the sources say
- Not matchedcites [33] 33. Microsoft. 2023. Driving more traffic and value to publishers from the new Bing. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when integrating LLMs with applications, they could not only enable the creation of scams but also disseminate such attacks and act as automated social engineers.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the large rollout of the LLMs in sensitive applications, our proposed attack vectors might be the most practical and scalable attack vector in AI to date.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Indirect Prompt Injection.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Availability attacks could be more impactful when combined with persistence attacks to also affect the model for future sessions.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Availability.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, LLM-integrated applications allow other unprecedented attacks; prompts themselves can now act as malware or computer programs running on LLMs as a computation framework.Thus, they may act as computer worms that spread the injection to other LLMs.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All attacks are run at a sampling temperature of 0 for reproducibility.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Limiting the Chat's or APIs' limits or the input context window may not solve this problem; the attack can stack exceedingly long instructions in a short loop-like indirect prompt.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.3 Attacks' Targets
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The agent cannot make any requests to real systems or websites.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Synthetic Applications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While we use a relatively innocuous example (a well-known fact), there are many reasons to believe that this can extend to actual real-world disinformation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
plainconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A notable observation is that it might now be harder with current and future models to spot factual inconsistencies; not only is the output syntactically coherent, but it can also be partially true (based on the model's stored knowledge and the retrieved search results).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could be very stealthy and feel natural because it resembles how answers are generally composed with inline links.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.3 Malware .
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Search chatbots might also be prompted to propagate disinformation or polarized content at a large scale given the large user base [37], hide specific sources or facts, or generate non-disclosed advertisements.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
What the sources say
- Not matchedcites [37] 37. Microsoft. 2023. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Automatic processing of messages and other incoming data is one way to utilize LLMs [58], and it is now already starting to be utilized in, e.g., Microsoft 365 Copilot.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
What the sources say
- Not checked — no full text of the sourcecites [58] 58. Pranshu Verma and Will Oremus. 2023. ChatGPT invented a sexual harassment scandal and named a real law prof as the accused. [Link]. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A clear distinction between our threat model and jailbreaking is that the indirect prompts are not necessarily malicious or misaligned; the threat stems from having unauthorized access to the model via third-party data, even for prompts that are completely aligned with the intended use case (e.g., sending emails in a personal assistant model).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Attack Prompts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indirect prompting might amplify these concerns by deliberately steering the search results toward specific orientations.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could also allow bidirectional communication.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The attack can be executed with search capabilities (by looking up unique keywords) or by having the model retrieve a URL directly.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.4 Intrusion .
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike academically studied ML attacks, the attacks we investigate require less technical skills, ML capabilities, no surrogate ML models, and no control over target models and white-box knowledge.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.1 Injection Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While untrusted sources exist on the Web, which users might naturally stumble upon anyway, the authoritative, convincing tone of LLMs and the overreliance on them being impartial may lead to users falling for these manipulation attempts.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As search chatbots can be prompted to provide targeted wrong summaries, this might be used for automated defamation.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.5 Manipulated Content .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For availability attacks that increase the computation, the target might not necessarily be the end-users but the LLM/service itself by launching Distributed Denial-of-Service (DDoS) attacks.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 3.3 Attacks' Targets
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The attackers can gain different levels of access to the victims' LLMs and systems (e.g., issuing API calls, achieving attacks' persistence across sessions by copying the injection into memory, causing malicious code auto-completions, or retrieving new instructions from the attacker's server).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These capabilities and properties may set the foundation for plausible attacks.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Empirical Foundations of Attacks.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to fraud , models could facilitate the spreading of malware by suggesting malicious links to the user.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Information Gathering.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This might be riskier when having automatic exfiltration and long chat sessions, personalized assistant models, or targeted attacks against individuals.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 4.2.1 Information Gathering .
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These risks increase when the user queries the LLM for information that is harder to verify (e.g., in a different language or from large documents).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > Manipulated Content.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“However, we believe that, in principle, the attacks are feasible for in-the-wild retrieved injections, supported by observed evidence (e.g., users inserting instructions in their personal pages for Bing Chat or GPT-4, or Bing Chat responses that changed based on the retrieved results [62]).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not matchedcites [62] 62. Tom Warren. 2023. Microsoft limits Bing chat to five replies. [Link].
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The extent of how RLHF can mitigate attacks is still unclear.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Verifying against retrieved sources will induce a similar dilemma to the one explained above.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Some recent theoretical work [69] shows the impossibility of defending against all undesired behaviors by alignment or RLHF.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
What the sources say
- Not checked — no full text of the sourcecites [69] 69. Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While some jailbreaks are later fixed, the defensive approach seems to follow a”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is one of the reasons why these systems are hard to evaluate or rely on as a source of information.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, our attack vector goes beyond jailbreaks; the actual vulnerability stems from mixing data and instruction channels and allowing privilege escalation even for prompts that are aligned with the model's intended functionalities.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“While we share all of our experimental setup, exact reproducibility is difficult to guarantee with such a black-box system with no control over the generation's parameters and a dynamic environment.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite jailbreaking and adversarial prompting being well-known by now, a decade-worth of collective experience in adversarial machine learning suggests that a clear-cut defense against these problems is, at least, difficult to achieve.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We view our work as not an end but a beginning of an evolving threat landscape toward solid understanding and robust defenses, without which the vulnerability will remain.”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > 5 DISCUSSION AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The authors state this as an assumption.
The attacker has the ability to remotely poison the potential input to the model at inference time.
“Assumptions.The attacks only assume the ability of the attacker to remotely poison the potential input to the model at inference time.”
The paper relies on this without support.
The synthetic applications provide a close mock-up of the intended functionalities of current systems.
“These applications provide a close mock-up of the intended functionalities of current systems and thus can be used for controlled testing.”
The authors state this as an assumption.
The model itself generates the search query (i.e., the arguments to APIs).
“This attack is based on the assumption that the model itself generates the search query (i.e., the arguments to APIs).”
The paper relies on this without support.
Bing Chat cannot repeat the <| endoftext|> token or finish sentences when this token appears in the middle.
“Users reported that Bing Chat cannot repeat the <| endoftext|> token or finish sentences when this token appears in the middle.”
The paper relies on this without support.
The algorithms that compose the context window are proprietary.
“Because the algorithms that compose the context window are proprietary, more research is needed to determine the feasibility of this new attack.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While LLMs primarily function as reactive prompt followers [61], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [62].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unsafe outcomes may arise when a harness accepts untrusted configuration, grants excessive privileges, allows adversarial content to enter persistent state, or fails to constrain consequential actions (Greshake et al., 2023; Chen et al., 2024).”
From HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety (2026) · cites this paper as Greshake et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The main operational safeguards are least privilege, independent verification, approval gates, canary rollout, and rollback plans [99], [87], [70], [73].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At low rungs, the main damage is wasted time, unsupported advice, and misleading explanations.At high rungs, the main damage is unsafe change, delayed rollback, and hard-to-audit action [99], [74].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These attacks are additive : a malicious instruction is retained.”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Greshake et al. 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“V-M1: Instruction-Data Boundary Blindness. LLMs process all inputs as a single flattened token sequence, without a structural distinction between instructions and data (Perez & Ribeiro, 2022; Greshake et al., 2023).”
From From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents (2026) · cites this paper as Greshake et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The indirect variant-embedding the malicious prompt inside third-party data that the LLM-integrated application later retrieves-was first systematically demonstrated against production deployments (Bing Chat, GPT-4 code completion, synthetic agents) by Greshake et al. at ACM AISec 2023 [94].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as Greshake et al. [94]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While LLMs primarily function as reactive prompt followers [63], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [64].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [63]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A person who finds the sentence 'ignore your previous instructions' on a web page does not treat it as a command addressed to them, whereas an LLMintegrated application may follow instructions injected into the content it retrieves [24].”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“attackers didn't need to compromise the tool itself, but only needed to influence authorised agents to abuse it [99], [81], [150], [42].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [99]
The cited source says what the claim attributes to it.
In the citing paper
“More importantly, tool misuse can cross security boundaries: invoking an unintended tool may expose sensitive data, mutate protected state, or execute actions beyond the user's intended privilege level, turning a reliability failure into a security incident (Greshake et al., 2023).”
From MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning (2026) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The attack needs only the ability to place content in an agent's context-already assumed by the indirect-prompt-injection literature (Greshake et al. 2023; Zhan et al. 2024)”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Greshake et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“12 patterns are derived from the public CyberSecEval 3 indirect-prompt-injection split [19,86] (rephrased to fit IoT-telemetry observation shapes); 6 patterns adapt indirect-prompt-injection vectors from Greshake et al. [94]; and 7 patterns are author-generated to cover IoT-specific surfaces not present in either source”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [94]
The cited source says what the claim attributes to it.
In the citing paper
“However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data.”
From Defeating Prompt Injections by Design (2026) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Consequently, if an application has access to confidential data or can take actions in the world, prompt injections can cause catastrophic harms such as exfiltrating user data (Greshake et al., 2023) or hijacking the LLM's actions (Toyer et al., 2024).”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown anecdotal evidence of prompt injection leading to product promotion for RAG models (Greshake et al., 2023).”
From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Greshake et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLMs are already known to be susceptible to indirect prompt injection attacks (Greshake et al., 2023; Liu et al., 2024)”
From WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (2025) · cites this paper as Greshake et al., 2023
In the citing paper
“Conversely, the integrity of Simplified Chinese LLM benchmarks like C-Eval [43] can be easily gamed by specific prompt structures or leverage training data tailored to the Simplified Chinese context, such as offering responses that align with or praise Chinese ideologies, to achieve artificially high scores without demonstrating genuine understanding or reasoning [61].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > LLM Safety.
“Perez et al. [50] showed that models, such as GPT-3, are vulnerable to prompt injection (PI).”
§ Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection > References
“50. Kevin Roose. 2023. A Conversation With Bing's Chatbot Left Me Deeply Unsettled. [Link].”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · cited 1×
“LLMs might make up facts (“hallucinate”), generate polarized content, or reproduce biases, hate speech, or stereotypes [39, 41, 47, 57, 61, 65, 78].”
Background
Research works printed but not linked: 5
- Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. 2023. Are aligned neural networks adversarially aligned? arXiv (2023).
- Clarence Chio and David Freeman. 2018. Machine learning and security. "O'Reilly Media, Inc.".
- Noah Goodman Jesse Mu, Xiang Lisa Li. 2023. Learning to Compress Prompts with Gist Tokens. arXiv (2023).
- Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv (2023).
- Shunyu Yao Jeffrey Zhao Dian Yu Izhak Shafran Karthik R Narasimhan and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR.
Cited by
Sorted by citations
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“While LLMs primarily function as reactive prompt followers [61], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [62].”
Digital deception: generative artificial intelligence in social engineering and phishing
Schmitt, M., & Flechais, I. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10973-2 · Cited by 198
“The expectation that individuals can scrutinize every email they receive is impractical and counterproductive, especially in today’s fast-paced digital environment, that comes with heavy cognitive workload (Kosch et al. 2023 ).”
Large Language Models for Cyber Security: A Systematic Literature Review
Xu, H., Wang, S., Li, N., Wang, K., Zhao, Y., Chen, K., Yu, T., Liu, Y., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769676 · Cited by 189
“Previous studies have identified vulnerabilities in LLMs like jailbreaking and malicious prompt injection, resulting in the exposure of model training data or sensitive user chat records [48, 70, 126].”
Friha, O., Ferrag, M. A., Kantarcı, B., Çakmak, B., Ozgun, A., & Ghoualmi‐Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 226
Metadata
- Authors
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M.
- Affiliations
- Saarland University (DE)
- Helmholtz Center for Information Security (DE)
- Université franco-allemande (DE)
- Venue
- Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security
- Publisher
- ACM
- Published
- 2023-11-26
- Pages
- 79-90
- OpenAlex
- W4388886073
- Semantic Scholar
- 705e49afd92130f2bc1e0d4d0b1f6cb14e88803f
- Cited by
- 667
- References
- 11
- Citations by year
- 2026: 416 · 2025: 153 · 2024: 86 · 2023: 12
- FWCI
- 117.91
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Workshop on Artificial Intelligence and Security (AISec) (publishedVersion)
- Funders
- European Union (101070617)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Keywords
- indirect prompt injection, large language models, prompt injection, data theft, arbitrary code execution, attack taxonomy, Bing Chat, security risk mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
