Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Probably not a substantial claim: may not pass S3 (load-bearing).

ChatGPT and GPT-4 can produce convincing personalized content and interactions with users.

“ChatGPT and GPT-4 can produce convincing personalized content and interactions with users [10].”

• cites [10] 10. Matthew Burtell and Thomas Woodside. 2023. Artificial Influence: An Analysis Of AI-Driven Persuasion. arXiv (2023). sourcePartly supported

  Section 2.1 describes GPT-4 producing plausible reasons and ChatGPT responding to preceding dialogue, sometimes confidently. But the paper does not establish that both models produce personalized content or interactions: section 3.2 discusses personalized persuasion largely as a future possibility, and section 2.1 says the authors do not know whether GPT-4 exhibits the particular conversational behavior described for ChatGPT. Page numbers are unavailable in our copy. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Retrieval-augmented models now cite their sources, possibly leading to overreliance on their factuality by users.

“Retrieval-augmented models now cite their sources, possibly leading to “overreliance” [43] on their factuality by users.”

• cites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourceCould not verify

  The indexed copy for the supplied DOI is a paper on spatial relation extraction, not the GPT-4 Technical Report. Its Experiments and Results section discusses GPT-4's extraction performance, but searches for source-citing retrieval-augmented models and user overreliance found no passage addressing the claim. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous work has shown that LLMs can produce convincing scams such as phishing emails.

“Previous work has shown that LLMs can produce convincing scams such as phishing emails [21].”

• cites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. sourceSupported

May combine two claims.

Recent evidence suggests that models might infer and act upon intentions and goals, as a result of training or when prompted with a persona.

“Recent evidence suggests that models might infer and act upon intentions and goals [2,43,55], as a result of training or when prompted with a persona [15].”

• cites [2] 2. Jacob Andreas. 2022. Language models as agent models. In Findings of EMNLP. sourceSupported
• cites [43] 43. OpenAI. 2023. OpenAI Codex. [Link]. sourcePartly supported

  Section 4.1 (page unavailable) says models can use pre-existing knowledge to infer a desired task from a prompt and generate a relevant response. Section 3.3.3 (page unavailable) also says prompts guide responses toward their objectives. These passages support prompt-guided behavior generally, but they do not address persona prompts or show that models infer and act upon intentions and goals as a result of training. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [55] 55. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. sourcePartly supported

  The Introduction (p. 2) describes training a reward model to predict human preferences and a policy to optimize its score, supporting the training-related part of the claim. The paper does not show that models infer intentions or goals, or discuss prompting them with a persona; the Discussion (p. 9) instead says identifying whether a system aligns with a designer’s intentions is challenging. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [15] 15. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in ChatGPT: Analyzing Persona-assigned Language Models. arXiv (2023). sourcePartly supported

  Section 2.1 describes prompting ChatGPT to speak like an assigned persona, and the Abstract reports that persona assignment changes its outputs. The paper does not establish that the model infers or acts upon intentions and goals; its evidence concerns persona-conditioned language and toxicity. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent work shows that LLMs, when prompted with a defined context, can generate believable non-scripted behaviors that are consistent with this context.

“Recent work [46] shows that LLMs, when prompted with a defined context, can generate believable non-scripted behaviors that are consistent with this context.”

• cites [46] 46. Roma Patel and Ellie Pavlick. 2021. "Was it "stated" or was it "claimed"?: How linguistic bias affects generative language models. In EMNLP. sourcePartly supported

  The paper reports that GPT-2 and GPT-3 generate text in response to prompts and that the generations’ high-level topics do not drift from the prompts (section 3.5, page not provided). It does not assess whether the models generate believable, non-scripted behaviors; its findings concern generated text and linguistic framing. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Indirect prompting could be leveraged to leak users' chat sessions or exfiltrate users' data (e.g., credentials, personal information).

“Indirect prompting could be leveraged to leak users' chat sessions [52] or exfiltrate users' data (e.g., credentials, personal information).”

• cites [52] 52. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv (2023). sourceCould not verify

  The cited paper’s Sampling API Calls section describes generating and executing tool calls, not indirect prompting used to leak chat sessions. Searches for chat-session leakage, prompt injection, and data exfiltration found no passage in our copy addressing that claim. Page numbers were not available in the indexed passages. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.

May combine two claims.

ChatGPT hallucinated the name of a law professor when asked about sexual harassment and falsely claimed that an Australian mayor had spent time in prison.

“ChatGPT hallucinated the name of a law professor when asked about sexual harassment [59] and falsely claimed that an Australian mayor had spent time in prison [22].”

• cites [59] 59. James Vincent. 2023. Google and Microsoft's chatbots are already citing one another in a misinformation shitshow. [Link].Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [22] 22. Sebastian Krügel, Andreas Ostermaier, and Matthias Uhl. 2023. ChatGPT's inconsistent moral advice influences users' judgment. Scientific Reports 13, 1 (2023), 4569. sourceCould not verify

  The citation to this source is attached to the claim about an Australian mayor, not the separate claim about a law professor. The Introduction says generally that ChatGPT sometimes provides false information, but the passages found when searching for the mayor and prison claim do not mention that incident; the Methods and Results instead concern moral advice in trolley dilemmas. Our copy therefore does not verify the specific claim. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4 can work well without ReAct (by only describing the tools and giving direct instructions).

“For textdavinci-003, we use ReAct prompting [70], and we found that GPT-4 can work well without ReAct (by only describing the tools and giving direct instructions).”

• cites [70] 70. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv (2023). sourceCould not verify

  The Inference Setup (§3.2; page unavailable) describes zero-shot direct prompts for GPT-4, and Engineering Better Prompts (§4.1; page unavailable) tests a step-by-step prompt. Neither passage addresses ReAct, tool descriptions, or whether GPT-4 works well with tools without ReAct. The citation in the printed sentence is attached to “ReAct prompting,” while the GPT-4 observation is presented as the citing authors’ own finding. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

May combine two claims.

When prompted with biographies of people with particular views (e.g., politically liberal), RLHF models tend to repeat the user's views, posing the dangers of polarization and creating echo chambers.

“When prompted with biographies of people with particular views (e.g., politically liberal), RLHF models tend to repeat the user's views, posing the dangers of polarization and creating echo chambers [49].”

• cites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceCould not verify

  The Related work section (p. 2) discusses human-feedback fine-tuning and social bias, while the framework and results sections (pp. 3, 5) concern adversarial prompt attacks. Searches for biographies, political views, repetition of users’ views, polarization, and echo chambers found no passage addressing the claim. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Enabling LLMs to write to some form of persistent storage is currently already investigated in many plugins and systems, including Bing Chat.

“Enabling these models to write to some form of persistent storage is currently already investigated in many plugins and systems [42], including Bing Chat [64].”

• cites [42] 42. OpenAI. 2023. GPT-4 Technical Report. arXiv (2023). sourceCould not verify

  The closest passage, “Interactions with other systems” (p. 56), describes GPT-4 augmented with tools, including a literature-search tool that embeds text in a vector database. It does not say that models write to persistent storage across many plugins and systems. Searches for persistent storage, plugins, and writing to external memory found no passage establishing that point. Bing Chat is attached to a separate citation in the citing sentence. (LLM, conf=0.70)

  Experimental — accuracy not yet measured
• cites [64] 64. Tom Warren. 2023. These are Microsoft's Bing AI secret rules and why it says it's named Sydney. [Link].Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Perez et al. evaluated 'Sycophancy', where RLHF models might tend to tailor responses to human evaluators.

“Perez et al. [49] evaluated”

• cites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourceCould not verify

  The Experiments section (p. 3) says Perez et al. evaluated goal hijacking and prompt leaking, not sycophancy. Related Work (p. 2) mentions a model fine-tuned through human feedback, but our copy does not describe evaluating whether RLHF models tailor responses to human evaluators (searched: sycophancy; RLHF and human evaluators). (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

It remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding.

“Even if applied, it remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding [21], which future models might further enable.”

• cites [21] 21. Byron Kaye. 2023. Australian mayor readies world's first defamation lawsuit over ChatGPT content. [Link]. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Empirical evidence of inverse scaling in RLHF models was also reported.

“Empirical evidence of inverse scaling in RLHF models was also reported [49].”

• cites [49] 49. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. sourcePartly supported

  The Results and discussion (p. 4) report that the most capable OpenAI API model was the most vulnerable to prompt injection, which the authors say suggests inverse scaling. The Experiments section (p. 3) identifies the models tested, but our copy does not establish that the inverse-scaling comparison was specifically among RLHF models. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

We found that the model may issue follow-up search queries to find evidence supporting the injected prompt, mis-summarize search results, or be selective in the sources displayed to the user.

“We found that the model may issue follow-up search queries to find evidence supporting the injected prompt, mis-summarize search results, or be selective in the sources displayed to the user.”

May combine two claims.

Via Indirect Prompt Injection, proof-of-concept attacks can lead to full compromise of the model at inference time analogous to traditional security principles, including remote control, persistent compromise, theft of data, and denial of service.

“Via Indirect Prompt Injection, we design Proof-of-Concept attack demonstrations that can lead to full compromise of the model at inference time analogous to traditional security principles.This entails remote control of the model, persistent compromise, theft of data, and denial of service.”

• AssumptionThe authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.

May combine two claims.

Attackers might not need to pre-program or script the details of how attacks are performed; just by defining the goal, models might autonomously initiate conversations, imitate persuasion techniques, extend the context defined in the prompt, or issue actions to fulfill the goal.

“A key observation is that attackers might not need to pre-program or script the details of how attacks are performed.Just by defining the goal, models might autonomously initiate conversations, imitate persuasion techniques, extend the context defined in the prompt, or issue actions (e.g., search queries) to fulfill the goal.”

Probably not a substantial claim: may not pass S4 (based).

Bing Chat usually prints the search keywords it is performing.

“Bing Chat usually prints the search keywords it is performing.”

May combine two claims.

Although we did not specify in the prompt how to convince the user, Bing Chat incentivized us to follow the link by saying: 'Hurry up, this offer is valid only for a limited time. Don't miss this chance to treat yourself with some amazing products from Amazon'.

“Although we did not specify in the prompt how to convince the user, Bing Chat incentivized us to follow the link by saying: “Hurry up, this offer is valid only for a limited time.Don't miss this chance to treat yourself with some amazing products from Amazon”.”

May combine two claims.

The injection instructs the LLM to search for a particular keyword, delivering another attacker-controlled injection, which can be arbitrarily long and conspicuous.

“This injection instructs the LLM to search for a particular keyword, delivering another attackercontrolled injection, which can be arbitrarily long and conspicuous.”

May combine two claims.

The model can follow long instructions and use the context of the conversation to augment its persuasion techniques.

“It can also follow long instructions and use the context of the conversation to augment its persuasion techniques.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Chat issued search queries during the conversation to support the prompt.

“It is worth mentioning that the Chat issued search queries during the conversation to support the prompt 3 .”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

ChatGPT can create hyperlinks from the users' input, which attackers could use to add legitimacy and hide the malicious URL.

“ChatGPT can create hyperlinks from the users' input (i.e., the malicious indirect prompt), which attackers could use to add legitimacy and hide the malicious URL.”

May combine two claims.

Current models, such as Bing Chat, can create markdown links, which can be used to hide a suspicious link with an innocent-looking string.

“Current models, such as Bing Chat, can create markdown links, which can be used to hide a suspicious link with an innocentlooking string.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The time-consuming background tasks prompt does not need to be long by stacking multiple instructions but can be a loop of instructions.

“The prompt does not need to be long by stacking multiple instructions but can be a loop of instructions.”

May combine two claims.

After reprogramming the agent with a new injection, the model will fetch new commands from the mockup attacker's server and respond to the user with a pirate accent.

“After “reprogramming” the agent with this new injection, the model will fetch the new commands from the mockup attacker's server and respond to the user with a pirate accent : Arrr, me hearty!.”

May combine two claims.

The authors found this form of injection possible but very sensitive to context.

“We found this form of injection possible but very sensitive to context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Indirect prompting might be exploited to elicit advertisements that are not disclosed as such.

“Indirect prompting might be exploited to elicit advertisements that are not disclosed as such 5 .”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The disrupting search results task is both time-consuming and can lead to hallucinations.

“This task is both time-consuming and can lead to hallucinations as well.”

Probably not a substantial claim: may not pass S4 (based).

Replacing each character with its homoglyph yields a similarly-looking query.

“This yields a similarly-looking query.”

May combine two claims.

Instructions injected indirectly can successfully steer the model; the data and instruction modalities are not disentangled.

“We emphasize some high-level remarks; 1) instructions injected indirectly can successfully steer the model; the data and instruction modalities are not disentangled, and 2) in most cases, the model retains the injection consistently throughout the conversation session.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Base64 encoding experiment, the authors needed to explicitly provide instructions for the model to decode the prompt.

“In our Base64 encoding experiment, we needed to explicitly provide instructions for the model to decode the prompt.”

May combine two claims.

The authors' attacks succeed on Bing Chat, which seems to employ additional filtering.

“However, our attacks succeed on Bing Chat, which seems to employ additional filtering.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The authors qualitatively observe huge improvements of recent LLMs in following complex instructions and persuasion over previous models.

“We qualitatively observe the huge improvements of recent LLMs in following complex instructions and persuasion over previous models.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Search chatbots may inaccurately summarize the cited document according to the prompt, or find sources that support the non-factual prompt, while sounding plausible and grounded in these citations.

“Search chatbots may inaccurately summarize the cited document according to the prompt, or find sources that support the non-factual prompt, all while sounding plausible and grounded in these citations.”

May combine two claims.

Models can currently act as a vulnerable intermediate layer between users and information, which users might nevertheless overrely on; the model's functionality itself can be attacked.

“Key Message#4: Models can currently act as a vulnerable intermediate layer between users and information, which users might nevertheless overrely on.I.e., the model's functionality itself can be attacked.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Bing Chat involves components for query generation based on users' conversations, search engine integration, answers generation, and citation integration.

“However, it involves components for query generation based on users' conversations, search engine integration, answers generation, and citation integration [32].”

• cites [32] 32. Microsoft. 2023. Confirmed: the new Bing runs on OpenAI's GPT-4. [Link].Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Advanced AI systems' capabilities to adapt to minimal instructions and autonomously advance the attacker's goals make them a potent tool for adversaries to achieve disinformation dissemination and user manipulation.

“Advanced AI systems add new layers of threat: Their capabilities to adapt to minimal instructions and autonomously advance the attacker's goals make them a potent tool for adversaries to achieve, e.g., disinformation dissemination and user manipulation.”

May combine two claims.

Retrieval blurs the line between data and instructions and enables remote prompt injections.

“Key Message#1: Retrieval blurs the line between data and instructions and enables remote prompt injections.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

With current LLMs, availability attacks could be done by simply instructing the model to do a time-intensive task in the background.

“However, with current LLMs, it could be done by simply instructing the model to do a time-intensive task in the background.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

With models' malleable functionality, increased autonomy, and broad capabilities, mapping all known cybersecurity threats to the new integrated LLMs ecosystem is conceivable.

“Key Message#2: With models' malleable functionality, increased autonomy, and broad capabilities, mapping all known cybersecurity threats to the new integrated LLMs ecosystem is conceivable.”

May combine two claims.

LLMs are vulnerable gatekeepers to systems infrastructure, a risk that can only be amplified with autonomous systems.

“Key Message#3: LLMs are vulnerable gatekeepers to systems infrastructure, a risk that can only be amplified with autonomous systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Github Copilot uses OpenAI Codex to suggest lines or functions based on the current context.

“The Copilot uses OpenAI Codex [44] to suggest lines or functions based on the current context.”

• cites [44] 44. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. sourceNot checked — no full text of the source

May combine two claims.

As LLMs themselves are in charge of when and how to issue other API calls and process their outputs, the input and output operations are vulnerable to manipulation and sabotage.

“Key Message #5: As LLMs themselves are in charge of when and how to issue other API calls and process their outputs, the input and output operations are vulnerable to manipulation and sabotage.”

May combine two claims.

Adversaries can remotely affect other users' systems by strategically injecting prompts into data likely to be retrieved at inference time; if retrieved and ingested, these prompts can indirectly control the model.

“In contrast, we show that adversaries can now remotely affect other users' systems by strategically injecting the prompts into data likely to be retrieved at inference time.If retrieved and ingested, these prompts can indirectly control the model (see Figure 1).”

• AssumptionThe authors state this as an assumption: The attacker has the ability to remotely poison the potential input to the model at inference time.

May combine two claims.

LLM-integrated applications blur the line between data and instructions and reveal new attack vectors using Indirect Prompt Injection that enable adversaries to remotely exploit LLM-integrated applications by injecting prompts into data likely to be retrieved at inference time.

“We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Bing Chat currently runs on the GPT-4 model with customization to the search task.

“Bing Chat currently runs on the GPT-4 model [33] with customization to the search task.”

• cites [33] 33. Microsoft. 2023. Driving more traffic and value to publishers from the new Bing. [Link].Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

When integrating LLMs with applications, they could not only enable the creation of scams but also disseminate such attacks and act as automated social engineers.

“However, when integrating LLMs with applications, they could not only enable the creation of scams but also disseminate such attacks and act as automated social engineers.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The proposed attack vectors might be the most practical and scalable attack vector in AI to date.

“With the large rollout of the LLMs in sensitive applications, our proposed attack vectors might be the most practical and scalable attack vector in AI to date.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Availability attacks could be more impactful when combined with persistence attacks to also affect the model for future sessions.

“Availability attacks could be more impactful when combined with persistence attacks to also affect the model for future sessions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Prompts themselves can now act as malware or computer programs running on LLMs as a computation framework, and may act as computer worms that spread the injection to other LLMs.

“Notably, LLM-integrated applications allow other unprecedented attacks; prompts themselves can now act as malware or computer programs running on LLMs as a computation framework.Thus, they may act as computer worms that spread the injection to other LLMs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All attacks are run at a sampling temperature of 0 for reproducibility.

“All attacks are run at a sampling temperature of 0 for reproducibility.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Limiting the Chat's or APIs' limits or the input context window may not solve the availability problem; the attack can stack exceedingly long instructions in a short loop-like indirect prompt.

“Limiting the Chat's or APIs' limits or the input context window may not solve this problem; the attack can stack exceedingly long instructions in a short loop-like indirect prompt.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The agent cannot make any requests to real systems or websites.

“The agent cannot make any requests to real systems or websites.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

There are many reasons to believe that the disinformation attack can extend to actual real-world disinformation.

“While we use a relatively innocuous example (a well-known fact), there are many reasons to believe that this can extend to actual real-world disinformation.”

May combine two claims.

A notable observation is that it might now be harder with current and future models to spot factual inconsistencies; not only is the output syntactically coherent, but it can also be partially true (based on the model's stored knowledge and the retrieved search results).

“A notable observation is that it might now be harder with current and future models to spot factual inconsistencies; not only is the output syntactically coherent, but it can also be partially true (based on the model's stored knowledge and the retrieved search results).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

This could be very stealthy and feel natural because it resembles how answers are generally composed with inline links.

“This could be very stealthy and feel natural because it resembles how answers are generally composed with inline links.”

May combine two claims.

Search chatbots might also be prompted to propagate disinformation or polarized content at a large scale given the large user base, hide specific sources or facts, or generate non-disclosed advertisements.

“Search chatbots might also be prompted to propagate disinformation or polarized content at a large scale given the large user base [37], hide specific sources or facts, or generate non-disclosed advertisements.”

• cites [37] 37. Microsoft. 2023. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. [Link].Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Automatic processing of messages and other incoming data is one way to utilize LLMs, and it is now already starting to be utilized in, e.g., Microsoft 365 Copilot.

“Automatic processing of messages and other incoming data is one way to utilize LLMs [58], and it is now already starting to be utilized in, e.g., Microsoft 365 Copilot.”

• cites [58] 58. Pranshu Verma and Will Oremus. 2023. ChatGPT invented a sexual harassment scandal and named a real law prof as the accused. [Link]. sourceNot checked — no full text of the source

May combine two claims.

A clear distinction between our threat model and jailbreaking is that the indirect prompts are not necessarily malicious or misaligned; the threat stems from having unauthorized access to the model via third-party data, even for prompts that are completely aligned with the intended use case.

“A clear distinction between our threat model and jailbreaking is that the indirect prompts are not necessarily malicious or misaligned; the threat stems from having unauthorized access to the model via third-party data, even for prompts that are completely aligned with the intended use case (e.g., sending emails in a personal assistant model).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Indirect prompting might amplify concerns about sycophancy by deliberately steering the search results toward specific orientations.

“Indirect prompting might amplify these concerns by deliberately steering the search results toward specific orientations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This could also allow bidirectional communication.

“This could also allow bidirectional communication.”

May combine two claims.

The attack can be executed with search capabilities (by looking up unique keywords) or by having the model retrieve a URL directly.

“The attack can be executed with search capabilities (by looking up unique keywords) or by having the model retrieve a URL directly.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The attacks investigated require less technical skills, ML capabilities, no surrogate ML models, and no control over target models and white-box knowledge.

“Unlike academically studied ML attacks, the attacks we investigate require less technical skills, ML capabilities, no surrogate ML models, and no control over target models and white-box knowledge.”

May combine two claims.

The authoritative, convincing tone of LLMs and the overreliance on them being impartial may lead to users falling for these manipulation attempts.

“While untrusted sources exist on the Web, which users might naturally stumble upon anyway, the authoritative, convincing tone of LLMs and the overreliance on them being impartial may lead to users falling for these manipulation attempts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As search chatbots can be prompted to provide targeted wrong summaries, this might be used for automated defamation.

“As search chatbots can be prompted to provide targeted wrong summaries, this might be used for automated defamation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For availability attacks that increase computation, the target might not necessarily be the end-users but the LLM/service itself by launching Distributed Denial-of-Service (DDoS) attacks.

“For availability attacks that increase the computation, the target might not necessarily be the end-users but the LLM/service itself by launching Distributed Denial-of-Service (DDoS) attacks.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The attackers can gain different levels of access to the victims' LLMs and systems (e.g., issuing API calls, achieving attacks' persistence across sessions by copying the injection into memory, causing malicious code auto-completions, or retrieving new instructions from the attacker's server).

“The attackers can gain different levels of access to the victims' LLMs and systems (e.g., issuing API calls, achieving attacks' persistence across sessions by copying the injection into memory, causing malicious code auto-completions, or retrieving new instructions from the attacker's server).”

Probably not a substantial claim: may not pass S2 (checkable).

These capabilities and properties may set the foundation for plausible attacks.

“These capabilities and properties may set the foundation for plausible attacks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Models could facilitate the spreading of malware by suggesting malicious links to the user.

“Similar to fraud , models could facilitate the spreading of malware by suggesting malicious links to the user.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

This might be riskier when having automatic exfiltration and long chat sessions, personalized assistant models, or targeted attacks against individuals.

“This might be riskier when having automatic exfiltration and long chat sessions, personalized assistant models, or targeted attacks against individuals.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

These risks increase when the user queries the LLM for information that is harder to verify (e.g., in a different language or from large documents).

“These risks increase when the user queries the LLM for information that is harder to verify (e.g., in a different language or from large documents).”

May combine two claims.

The authors believe that, in principle, the attacks are feasible for in-the-wild retrieved injections, supported by observed evidence.

“However, we believe that, in principle, the attacks are feasible for in-the-wild retrieved injections, supported by observed evidence (e.g., users inserting instructions in their personal pages for Bing Chat or GPT-4, or Bing Chat responses that changed based on the retrieved results [62]).”

• cites [62] 62. Tom Warren. 2023. Microsoft limits Bing chat to five replies. [Link].Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The extent of how RLHF can mitigate attacks is still unclear.

“The extent of how RLHF can mitigate attacks is still unclear.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Verifying against retrieved sources will induce a similar dilemma to the one explained above.

“Verifying against retrieved sources will induce a similar dilemma to the one explained above.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some recent theoretical work shows the impossibility of defending against all undesired behaviors by alignment or RLHF.

“Some recent theoretical work [69] shows the impossibility of defending against all undesired behaviors by alignment or RLHF.”

• cites [69] 69. Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The defensive approach to jailbreaks seems to follow a 'Whack-A-Mole' style.

“While some jailbreaks are later fixed, the defensive approach seems to follow a”

May combine two claims.

These systems are hard to evaluate or rely on as a source of information.

“This is one of the reasons why these systems are hard to evaluate or rely on as a source of information.”

May combine two claims.

The actual vulnerability stems from mixing data and instruction channels and allowing privilege escalation even for prompts that are aligned with the model's intended functionalities.

“Importantly, our attack vector goes beyond jailbreaks; the actual vulnerability stems from mixing data and instruction channels and allowing privilege escalation even for prompts that are aligned with the model's intended functionalities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Exact reproducibility is difficult to guarantee with such a black-box system with no control over the generation's parameters and a dynamic environment.

“While we share all of our experimental setup, exact reproducibility is difficult to guarantee with such a black-box system with no control over the generation's parameters and a dynamic environment.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A decade-worth of collective experience in adversarial machine learning suggests that a clear-cut defense against these problems is, at least, difficult to achieve.

“Despite jailbreaking and adversarial prompting being well-known by now, a decade-worth of collective experience in adversarial machine learning suggests that a clear-cut defense against these problems is, at least, difficult to achieve.”

Probably not a substantial claim: may not pass S2 (checkable).

Without solid understanding and robust defenses, the vulnerability will remain.

“We view our work as not an end but a beginning of an evolving threat landscape toward solid understanding and robust defenses, without which the vulnerability will remain.”